Hi,
We have now migrated our hippo cms to kubernetes. It is now running on 5 pods, which serves the shop frontend and the author instance is running on 1 pod. The problem is now, that when cms content is changed (for example Image a is changed to image b), some pods still serves the old image A and some serve the new image b. When I delete the local repository on a pod, which deliver the wrong cms content, this pod deliver the correct cms-content, after the local repository is recreated. So I think the synchronisation of the different locals repositories of the pods fails.
When I save some cms-Content, I get the following Warning;
site-carhartt-webshop-0 shop 18.03.2019 12:18:42 WARN ClusterNode-site-carhartt-webshop-0.site-carhartt-webshop.shop.svc.cluster.local [ServicingSearchIndex.aggregateDescendants:751] Unable to add index fields for child states of 82559a8f-c3c3-4e99-8598-572e1d69ce5c because an item could not be found. Probably because it was removed again. (full stack trace on debug level)
Can you give me a hint, how I could solve this issue?
best regards
Andi
It sounds like your pods are either not up to date on synchronizing or the index is corrupted somehow. There is no way to “force” a synchronization.
You can check the latest revision number directly in the db like this:
SELECT * FROM REPOSITORY_GLOBAL_REVISION;
And then you can check what version your pod is on:
SELECT * FROM REPOSITORY_LOCAL_REVISIONS;
I would expect your pod to lag behind. If it isn’t then I would say it is a corrupt index. Why this happens would need to be investigated. Note that at this time we do not officially support kubernetes so I can’t promise any solution. That being said, we do have some experience with it and this is not a case I am familiar with.
If the pod is merely lagging behind, waiting should solve it. If it isn’t lagging behind then you will need to at least restart the pod for an index consistency check.
Hi,
The revisions seems to be correct on DB-level:
mysql> SELECT * FROM REPOSITORY_GLOBAL_REVISION;
±------------+
| REVISION_ID |
±------------+
| 3812106 |
±------------+
1 row in set (0.03 sec)
mysql> SELECT * FROM REPOSITORY_LOCAL_REVISIONS;
±-------------------------------------------------------------------------±------------+
| JOURNAL_ID | REVISION_ID |
±-------------------------------------------------------------------------±------------+
|
| author-carhartt-webshop-0.author-carhartt-webshop.shop.svc.cluster.local | 3812106 |
| site-carhartt-webshop-0.site-carhartt-webshop.shop.svc.cluster.local | 3812106 |
| site-carhartt-webshop-1.site-carhartt-webshop.shop.svc.cluster.local | 3812106 |
| site-carhartt-webshop-2.site-carhartt-webshop.shop.svc.cluster.local | 3812106 |
| site-carhartt-webshop-3.site-carhartt-webshop.shop.svc.cluster.local | 3812106 |
| site-carhartt-webshop-4.site-carhartt-webshop.shop.svc.cluster.local | 3812106 |
±-------------------------------------------------------------------------±------------+
22 rows in set (0.03 sec)
I will do the consistency check now
Hi,
I have restarted one pod for doing the consistency check, and after restarting I got this error:
`site-carhartt-webshop-0 shop 18.03.2019 14:06:50 ERROR ClusterNode-site-carhartt-webshop-0.site-carhartt-webshop.shop.svc.cluster.local [ClusterNode.run:309] Unexpected exception while syncing of journal: null
site-carhartt-webshop-0 shop java.lang.NullPointerException
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.journal.AbstractJournal.doSync(AbstractJournal.java:249)
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.journal.DatabaseJournal.doSync(DatabaseJournal.java:443)
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.journal.AbstractJournal.internalSync(AbstractJournal.java:222)
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:190)
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.cluster.ClusterNode.internalSync(ClusterNode.java:340)
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:356)
site-carhartt-webshop-0 shop at org.apache.jackrabbit.core.cluster.ClusterNode.run(ClusterNode.java:303)
site-carhartt-webshop-0 shop at java.lang.Thread.run(Thread.java:748)
site-carhartt-webshop-0 shop 18.03.2019 14:06:53 ERROR pool-9-thread-1 [DbResetExpiredLocksJanitor.run:54] Error while trying to reset locks`
How can I handle that error?
best regards
Andi