Unable To Merge Isolated (But Non-Conflicting) Repository Changes Made In Source And Target Environments During Redeployment In BR Cloud

#1

Hello,

// TL;DR; START

In a project setup consisting of two environments, local and QA, where local is my developer workstation and QA is the deployment of my project on the BR Cloud, I was hoping that:

A subsequent deployment to QA following small, non-conflicting English caption repository changes in each environment would result in both changes merged properly when bootstrapping kicks in during redeployment in QA.

I observe that “repo.bootstrap” is set to “true” under CMS > Admin > System Properties in the BR Cloud QA environment, thus suggesting bootstrapping should occur. However, this experiment fails - preserving the original caption change in QA, but not that from local. Furthermore, I see an error in the CMS log file which may be related to this issue (discussed further below).

Anyone know why this doesn’t work? Longer explanation follows on exactly what I did. Thank you very much!

// TL;DR; END

Here are the steps I followed to attempt this experiment:

Part I: Setup Of Both Environments:

  • I have a simple BR project on my local machine with some custom captions (affecting yaml/storage) and some customized FTL templates. It appears to run fine with no obvious errors in the log file.
  • I run “mvn clean verify” followed by “mvn -P dist” in my local machine project root folder. I observe the newly created distributable in the target/ folder.
  • In the BR Cloud, I create a new environment - say “QA” - and I (1) upload the distributable, and (2) deploy it for the first time to this environment. It seems to run fine as well. There are no obvious errors in the log files.

Part II: Applying Isolated Changes To Both Environments:

  • In QA, CMS > Content > administration > labels > Global (essentials.global), I change a caption, then publish. (I literally only add three dots, “…” to the end of an existing caption value.)
  • In QA, on the main site application, I verify the rendering of “…” on some existing page that utilises the affected caption.
  • In my local, CMS > Content > administration > labels > Global (essentials.global), I add two new captions. I observe that this has resulted in modifications to yaml and storage/, as expected.
  • In my local, I modify an FTL file for an existing page, which outputs these two new captions.
  • In my local, on the main site application, I verify the proper rendering of these two new captions. Nothing appears amiss with the log console output.

Part III: Updating QA With A Redeployment From A New Distributable

  • Once more in local, I run “mvn clean verify” followed by “mvn -P dist”. I observe the newly created distributable in the target/ folder.
  • In the BR Cloud, for the QA environment, I upload the distributable and issue a “Redeploy”. (An error occurs behind the scenes in a CMS log file, apparently related to this redeployment - but I will discuss that error further below.)

Part IV: Observing The QA Environment

  • I test the main site application in QA. I make the following three observations: (1) The former “…” modification to an existing caption (as per Part II) is present in QA, just as it was previously. (2) The new FTL mark-up changes from my local were successfully deployed and are viewable in QA. However, (3) The two new captions introduced by the new FTL changes do not render in QA as they do in my local (instead, they show missing caption strings like, “???my.caption.key???”).
  • Realizing that my experiment has failed, I go to QA CMS > Content > administration > labels > Global and attempt to locate the two new captions. I then discover they are not there. (Gasp! Insert frightening music.)

The above initially suggested to me that the bootstrapping in QA (based on the yaml file changes that occurred in my local to add the two new captions) did not take place during the redeploy. Yet I had confirmed that repo.bootstrap=true in QA, as mentioned earlier. I subsequently downloaded the log files (cmslogs*.tar.gz) and checked them.

I noticed that there appear to be two sets of log files - cms1 and cms2.

Question 1: Does anyone know why there are two sets of log files from the BR Cloud? I presume that the reason for it is because the deployment is done in a two-node environment, is that correct? (Let me know if I’m wrong. I believe this has nothing to do with author-versus-delivery nodes, as I think I need to deploy those separately, but if I’m mistaken I would very much appreciate someone kindly informing me.)

In the first set of log files (under cms1), I observe an obvious error in cms.log:

05.04.2019 21:07:22 [localhost-startStop-1] ERROR [org.onehippo.cm.engine.ConfigurationServiceImpl.serviceRegistered():968] Failed to add hcm site: myproject for context path: /site
javax.jcr.RepositoryException: java.lang.RuntimeException: Problem loading stream for resource in baseline
	at org.onehippo.cm.engine.ConfigurationServiceImpl.applyConfig(ConfigurationServiceImpl.java:797) ~[hippo-repository-engine-13.0.0.jar:13.0.0]
...
...
...
Caused by: javax.jcr.InvalidItemStateException: Item does not exist anymore: bccd99f5-bf92-4680-bd59-f0a87af925c5
	at org.apache.jackrabbit.core.ItemImpl.itemSanityCheck(ItemImpl.java:116) ~[jackrabbit-core-2.16.2-h3.jar:2.16.2-h3]
	at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:90) ~[jackrabbit-core-2.16.2-h3.jar:2.16.2-h3]
	at org.apache.jackrabbit.core.NodeImpl.getNode(NodeImpl.java:2161) ~[jackrabbit-core-2.16.2-h3.jar:2.16.2-h3]

My suspicions at this point is that there was an attempt to invoke bootstrapping (I note configuration-related methods in the stack trace, such as “applyConfig” among others). Furthermore, perhaps this error resulted in the interruption of the bootstrapping process and that might explain why the two new captions from the yaml files never made it into QA. But I’m not entirely certain.

Realizing that the QA environment was expecting an object with (UUID, I presume) bccd99f5-bf92-4680-bd59-f0a87af925c5 to exist, I decided to search for it:

  • In the QA /cms/repository application, I clicked on the UUID tab, pasted the above identifier, clicked “Search”, but found no results.
  • Repeating the same search in my local environment, just to be sure, I found no results there either.

Question 2: Is my expectation of successfully merging two isolated repository changes (one local, the other on the target environment - changes non-conflicting) following the redeployment of a new distributable flawed?

I am inclined to believe that this bootstrapping and merging should work, as BR documentation states, “If you are upgrading your project or deploying a new version this means you need to explicitly enable bootstrap processing in order to make sure the repository is updated with the latest packaged configurations. This done by setting the system property repo.bootstrap=true.” This is found on the following page:

A parting thought: If this error message is indeed causing a bootstrapping issue, that might suggest my QA environment is now broken - until I do something to fix the issue. Being a BR newbie developer, I’m not sure how to go about doing that… (Gasp again! Insert more frightening music.)

Firstly, huge thanks for reading all of this. If anyone would be so kind as to answer either question or provide any kind of direction, I would be very grateful.

Thank you so much,
Nick

P.S. I’m speculating from some further online searching that perhaps my QA environment in the BR Cloud needs to have search indexes “checked and fixed”, as discussed in the following document, but I’m not sure how to do that in the BR Cloud - and I could be very wrong about this.

#2

hi @nick
I think your way is already correct in general just there is issue at some of the point. So I suggest you need to check.

  1. Which yaml file updated at your local, make sure those yaml file is existed and updated in your distribution file .tar
  2. Make sure boostrap = true when you deploy QA
  3. Check the log if the bootstrap said to be done sucessfully
  4. Check the folder /hcm:hcm/hcm:baseline which store information about last bootstrap, eg… which file have been updated during the bootstrap.

I just able to know that :slight_smile:

#3

Hello, @ThoLe .

I want to thank you very much for taking the time to answer me with your very helpful tips.

Drilling into the embedded WAR file of the distributable, I’m able to locate a [myprojectname]-repository-data-application-0.1.0-SNAPSHOT.jar file, within which I can clearly see the new captions. They are not getting imported - whereas ftl (source) file changes are easily brought in. (Obviously, there’s no merging that needs to be done with the latter.)

Consequently, I end up with an “updated” web app that clearly shows the new features I added (i.e. FTL/Java file changes), but new yaml file changes - notably captions - appear to be missing.

I tried looking under the “hcm:hcm” folder as you recommended in the repository and can clearly see the revised time under the only child folder, “hcm:baseline”, which reflects the time of redeployment. And that is good, but I’m unable to glean much of any further info. From that node, I was able to dig down into a **/hcm-content/content/documents/administration/labels.yaml node under this tree, export it as YAML, open it up, but there were just a few settings in there and no captions… I’m not really sure what to make of this.

I’ll keep investigating and I’ll let you know what I come up with.

Again, I want to thank you very kindly for your excellent pointers, ThoLe! I really appreciate you reaching out to help.

Nick

#4

Hello, @ThoLe - and to anyone who reads this entry,

The underlying reason for the errors reported here appears to be a BR defect. Please see the following ticket entry for further information:

https://issues.onehippo.com/browse/CMS-11831

The short of it:

Redeploying (and I would say also during initial deployment in some cases) of BR prior to (as yet unreleased) version 13.2 on the BR Cloud using a multi-node cluster causes bootstrapping errors. Please avoid this for the time being.

I presume, in the interim, the solution would therefore be to switch to a single-node cluster until this issue gets resolved and a given project is upgraded to v13.2.

Thank you,
Nick

#5

Hi @nick
It really take time to deal with such issue. Great you find it.