While performing a brXM v15 upgrade in a staging environment we have run into an issue where webfiles do not seem to be properly bootstrapped. I’ve checked to make sure that the repo.bootstrap system property is set to true, as it normally is for our upgrades, as well as other things but it seems that we get 404 HTTP responses for any webfiles related call. After setting the log level for org.hippoecm.hst.core.container.WebFileValve from WARN to DEBUG we noticed the following log entry while retrying the upgrade by restarting tomcat.
16:49:29 INFO No '/hst-allowlist.txt' configured in web files for 'site'. All web files will be allowed. In the next PATCH version (HST 3.1.1) all web files will be blocked if there is no '/hst-allowlist.txt' configured in web files.
After digging into this by looking at the webfiles through the CMS console I realized that indeed hst-allowlist.txt was missing and hst-whitelist.txt still remained. I believe this explains why the site is not serving up webfiles based on the the explaining above which should have probably been better off logged at the WARN level than INFO in my opinion. This upgrade process has worked fine in lower environments where we have done the original development deployments through very similar process so I’m not sure what is happening during this rehearsal for a production release.
The question remaining is why the hst-allowlist.txt file and perhaps other webfiles are not being bootstrapped? I don’t see any other related errors or relevant warnings in catalina.out so other than setting all log levels to debug and digging through the firehose of logs to see if I can find anything is there any specific things I should look for that may help to troubleshoot bootstrapping or upgrade processes in more detail or even better yet explain the underlying cause for the behavior and error mentioned above?
A detail that I’ve found as well that could be related is that the following error is logged when tomcat starts up as well.
08.12.2023 09:42:41 WARN main [ProtectedRemoveManager.<init>:46] protectedhandlers-config is missing -> DIFF processing can fail for the Remove operation if the content toremove is protected!
This error appears to be originating from the ProtectedRemoveManager constructor when a config is not provided. The only thing that I can find related to this is some mention about a possible missing /WEB-INF/protectedHandlersConfig.xml file which looks to be something specific about JCR of which I can find nothing about in the Bloomreach documentation or our project.
One other note is that we also had to adjust the repository-directory property in context.xml in order to work around an issue where the project seemed to be picking up this value and wrongly creating the repository in the parent directory of where we normally place things. After changing this value, the repository directory is being created in the proper place but we still don’t have an explanation of why the behavior was different, perhaps something related to JDK8 vs. JDK11 or a difference in behavior between brXM v14 / v15? This may or may not be related but I figured I’d mention it if there could be some underlying explanation which may have influenced both the current issue and the repository directory issue which we believe at this point that we have effectively resolved.
Hello @jpierson ,
this was probably an oversight during your upgrade from v14 to v15.
On this documentation page it is explicitly mentioned to rename the file from hst-whitelist to hst-allowlist that is part of your project.
To my knowledge, no changes related to context.xml or the repository-directory have been done as part of the upgrade.
Kind regards
Thank you @Lef_Karamoulas for your suggestion. The file was renamed accordingly in our project and I’ve confirmed that in the release artifact for the CMS that it contains the webfiles archive with the proper file with the updated name. Still after startup, and presumably bootstrapping, it looks like the whitelist is what shows up in webfiles and no allowlist. My best guess is that something is failing to bootstrap the webfiles in general but I don’t see any obvious indicator in the logs to point at that being the case or pointing out any particular cause.
We just got through deleting the repository directory and restarting after setting the log level to DEBUG across the board and so far I don’t see anything helpful to troubleshoot the issue but I was able to confirm that I see the same error as before the cms.war deployment log event indicates that it finished.
08.12.2023 11:52:51 WARN main [ProtectedRemoveManager.:46] protectedhandlers-config is missing → DIFF processing can fail for the Remove operation if the content toremove is protected!
08-Dec-2023 11:52:51.159 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/vendor/APACHE/TOMCAT/tomcat_base/webapps/cms.war] has finished in [301,981] ms
I’m reviewing the following documentation to see if I can find any clue about troubleshooting production boostrapping. So far I’m unable to find anything very useful from the logs alone.
Docs:
After some troubleshooting today we found a few extra things:
- The related configuration within .yaml files did seem to get bootstrapped. We verified this by looking at the change in the
nextFireTime property from security.yaml which changed since the previous successful deployment.
- Manually changing
hst-whitelist.txt to hst-allowlist.txt under webfiles in the CMS console allowed the files to load in the site and no longer respond with 404 errors.
- Verified that the
protectedhandlers-config is missing warning does appear to occur in logs for some of the environments where we haven’t had the the mentioned webfiles bootstrapping issue. This indicates that this issue may be unrelated to this log warning.
- We looked at some of the the plugins used in our root project for compatibility with v15 just to get clues but nothing seems out of place which would explain the difference in behavior in the specific staging environment where we’ve run into this issue.
Our current working theory is that the webfiles bundle did not update since the only change in the deployment was a file rename. The file rename alone doesn’t seem to be enough to change the webfiles digest ID used to determine whether there are any changes necessary to bootstrap and thus if it the only change then the webfiles are not updated. Unfortunately for brXM 15, the file rename is the only change necessary so unless any other changes are made to the content of any of the files included in the webfile bundle, it may be a subtle issue and not well understood behavior that the files are not loaded and thus will not be served up in these scenarios due to the hst-allowlist.txt requirement.
If the webfiles bootstraping behavior described above turns out to be the root cause, I would say that having this fixed in a future patch release of v15 would be a good idea to help others avoid running into this issue or at least a note added to the v15 upgrade documentation to provide steps as a workaround.
The above finding is correct. We have updated the relevant upgrade documentation page with a note.