Hi there,
A website that we host was taken down yesterday, with the only indication in the logs being a search index problem just before the site died. We restored the DB to an earlier version and scaled the server to bring the site back up, but would like to know more information about how this may have happened. We would also like to know if there is something more we can do to prevent it from happening again.
The exception that occurred in the logs was:
25.02.2021 03:00:04 INFO ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@3f91d09a], elements=103364, usedmemorykb=262077, maxmemorykb=262144, access=62479174, miss=261914
25.02.2021 11:35:36 INFO pool-7-thread-1 [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@3f91d09a], elements=103392, usedmemorykb=262142, maxmemorykb=262144, access=62479428, miss=262090
25.02.2021 11:39:18 INFO ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@3f91d09a], elements=103385, usedmemorykb=262116, maxmemorykb=262144, access=62479555, miss=262109
25.02.2021 11:47:34 WARN ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [SearchIndex.updateNodes:729] Exception while creating document for node: 46f20fbc-8210-4b3d-bc30-823589bd14b4: javax.jcr.ItemNotFoundException: org.apache.jackrabbit.core.state.NoSuchItemStateException: 46f20fbc-8210-4b3d-bc30-823589bd14b4/{http://www.onehippo.org/jcr/hippostd/nt/2.0}content
25.02.2021 11:47:34 INFO ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@3f91d09a], elements=103391, usedmemorykb=262139, maxmemorykb=262144, access=62479682, miss=262137
25.02.2021 11:47:39 INFO ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@4a24b7ce], elements=43845, usedmemorykb=59600, maxmemorykb=65536, access=975741, miss=44066
25.02.2021 11:50:04 WARN ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [SearchIndex.updateNodes:729] Exception while creating document for node: 5722c838-fbe6-4284-bebd-9f2f01f25026: javax.jcr.RepositoryException: Missing child node entry for node with id: 5722c838-fbe6-4284-bebd-9f2f01f25026
25.02.2021 11:50:04 INFO ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@3f91d09a], elements=103387, usedmemorykb=262143, maxmemorykb=262144, access=62480063, miss=262196
25.02.2021 11:51:55 WARN ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [SearchIndex.updateNodes:729] Exception while creating document for node: 9c55b168-e9fc-4818-90a6-8adaa6b8538e: javax.jcr.ItemNotFoundException: org.apache.jackrabbit.core.state.NoSuchItemStateException: 9c55b168-e9fc-4818-90a6-8adaa6b8538e/{http://www.onehippo.org/jcr/hippostd/nt/2.0}content
25.02.2021 11:51:55 INFO ClusterNode-nhbcwebsite-7dd946cb4f-mpplp [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@3f91d09a], elements=103373, usedmemorykb=262142, maxmemorykb=262144, access=62480571, miss=262285
Any info would be appreciated,
Best regards,