Bloomreach Docker startups - taking longer sometimes

1

Hi there, sometimes when our bloomreach container starts up, it initialises the index ok , starts deploying the web application then we seem to see loads of these messages…

INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=224281, usedmemorykb=54460, maxmemorykb=262144, access=265938, miss=246670

Does anyone know what its doing?

The problem is it fails its readniess checks and the pod gets restarted… vicious circle of death…

We are deploying into kubernetes and on the deployment yml we have liveness and readiness checks on the /site/ping url to check the CMS has started.

Full logs below…

Extracting lucene index export…
Archive: /lucerneindexfolder/liveindex/liveindex.zip
inflating: /brxm/project/target/storage/workspaces/default/index/_17/segments_1
inflating: /brxm/project/target/storage/workspaces/default/index/_17/segments_4
inflating: /brxm/project/target/storage/workspaces/default/index/_17/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_17/_2.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_16/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_16/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_16/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_16/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_15/segments_1
inflating: /brxm/project/target/storage/workspaces/default/index/_15/_6.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_15/segments_8
inflating: /brxm/project/target/storage/workspaces/default/index/_15/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_7/segments_1
inflating: /brxm/project/target/storage/workspaces/default/index/_7/segments_9
inflating: /brxm/project/target/storage/workspaces/default/index/_7/_7.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_7/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_5/segments_1
inflating: /brxm/project/target/storage/workspaces/default/index/_5/_5.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_5/segments_7
inflating: /brxm/project/target/storage/workspaces/default/index/_5/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_14/segments_1
inflating: /brxm/project/target/storage/workspaces/default/index/_14/_9.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_14/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_14/segments_b
inflating: /brxm/project/target/storage/workspaces/default/index/_13/segments_1
inflating: /brxm/project/target/storage/workspaces/default/index/_13/_9.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_13/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_13/segments_b
inflating: /brxm/project/target/storage/workspaces/default/index/_12/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_12/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_12/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_12/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_11/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_11/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_11/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_11/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_10/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_10/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_10/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_10/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_9/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_9/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_9/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_9/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_8/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_8/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_8/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_8/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_4/segments_2
inflating: /brxm/project/target/storage/workspaces/default/index/_4/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_4/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_4/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_6/segments_4
inflating: /brxm/project/target/storage/workspaces/default/index/_6/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_6/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_6/_0_2.del
inflating: /brxm/project/target/storage/workspaces/default/index/_6/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_2/segments_kb
inflating: /brxm/project/target/storage/workspaces/default/index/_2/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_2/_0_k9.del
inflating: /brxm/project/target/storage/workspaces/default/index/_2/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_2/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_3/segments_3
inflating: /brxm/project/target/storage/workspaces/default/index/_3/_0_1.del
inflating: /brxm/project/target/storage/workspaces/default/index/_3/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_3/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_3/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_1/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_1/segments_q7
inflating: /brxm/project/target/storage/workspaces/default/index/_1/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_1/_0_q5.del
inflating: /brxm/project/target/storage/workspaces/default/index/_1/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_0/_0_dw.del
inflating: /brxm/project/target/storage/workspaces/default/index/_0/cache.inSegmentParents
inflating: /brxm/project/target/storage/workspaces/default/index/_0/_0.cfs
inflating: /brxm/project/target/storage/workspaces/default/index/_0/segments.gen
inflating: /brxm/project/target/storage/workspaces/default/index/_0/segments_dy
inflating: /brxm/project/target/storage/workspaces/default/index/indexRevision.properties
21-Mar-2024 07:06:09.003 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.35] using APR version [1.7.0].
21-Mar-2024 07:06:09.006 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true].
21-Mar-2024 07:06:09.006 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
21-Mar-2024 07:06:09.009 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1n 15 Mar 2022]
21-Mar-2024 07:06:09.351 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“http-nio-8080”]
21-Mar-2024 07:06:09.381 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“ajp-nio-127.0.0.1-8009”]
21-Mar-2024 07:06:09.382 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [618] milliseconds
21-Mar-2024 07:06:09.418 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
21-Mar-2024 07:06:09.419 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.65]
21-Mar-2024 07:06:09.434 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/cms]
21.03.2024 07:06:41 INFO main [SearchIndex.doInit:659] Index initialized: /brxm/project/target/storage/workspaces/default/index Version: 3
21.03.2024 07:07:39 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=29460, usedmemorykb=3766, maxmemorykb=65536, access=29464, miss=29460
21.03.2024 07:07:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=31937, usedmemorykb=13429, maxmemorykb=262144, access=39370, miss=33131
21.03.2024 07:08:39 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=61972, usedmemorykb=7830, maxmemorykb=65536, access=61976, miss=61972
21.03.2024 07:08:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=65296, usedmemorykb=21758, maxmemorykb=262144, access=76708, miss=68144
21.03.2024 07:09:39 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=96643, usedmemorykb=12164, maxmemorykb=65536, access=96647, miss=96643
21.03.2024 07:09:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=99898, usedmemorykb=28746, maxmemorykb=262144, access=114808, miss=105144
21.03.2024 07:10:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=129663, usedmemorykb=16292, maxmemorykb=65536, access=129667, miss=129663
21.03.2024 07:10:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=132193, usedmemorykb=34851, maxmemorykb=262144, access=151257, miss=140556
21.03.2024 07:11:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=162429, usedmemorykb=20388, maxmemorykb=65536, access=162433, miss=162429
21.03.2024 07:11:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=163509, usedmemorykb=40509, maxmemorykb=262144, access=187579, miss=175797
21.03.2024 07:12:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=194179, usedmemorykb=24356, maxmemorykb=65536, access=194183, miss=194179
21.03.2024 07:12:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=194038, usedmemorykb=49407, maxmemorykb=262144, access=229235, miss=211370
21.03.2024 07:13:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=226945, usedmemorykb=28452, maxmemorykb=65536, access=226949, miss=226945
21.03.2024 07:13:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=224281, usedmemorykb=54460, maxmemorykb=262144, access=265938, miss=246670
21.03.2024 07:14:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=260346, usedmemorykb=32627, maxmemorykb=65536, access=260350, miss=260346
21.03.2024 07:14:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=254474, usedmemorykb=59323, maxmemorykb=262144, access=302641, miss=282406
21.03.2024 07:15:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=versionBundleCache[ConcurrentCache@377f495d], elements=293874, usedmemorykb=36818, maxmemorykb=65536, access=293878, miss=293874
21.03.2024 07:15:40 INFO main [AbstractBundlePersistenceManager.logCacheStats:876] cachename=defaultBundleCache[ConcurrentCache@185215ea], elements=284652, usedmemorykb=64190, maxmemorykb=262144, access=339471, miss=318451

Any thoughts?
Thanks
J

2

Hi @Mort365 ,

Can you please check the documentation Run and Develop with Docker - Bloomreach Experience Manager (PaaS/Self-Hosted) - The Fast and Flexible Headless CMS and check the configurations are installed properly or not?

3

Those messages are from the bundle cache and they seem to show a high miss rate. What is the size of your bundle cache and java heap?

4

Hi all, this is the default brxm image.

Settings for below

Default JVM heap size variables

ENV JAVA_MINHEAP 512m

ENV JAVA_MAXHEAP 1024m

Default tomcat http max threads variable

ENV TOMCAT_MAXTHREADS 250

Default repository settings

ENV REPO_PATH ${docker.brxm.project.path}/target/storage

ENV SITE_INDEX_PATH ${docker.brxm.project.path}/target/storage/workspaces/default/index

ENV REPO_CONFIG “”

ENV REPO_BOOTSTRAP false

ENV REPO_AUTOEXPORT_ALLOWED false

ENV REPO_WORKSPACE_BUNDLE_CACHE 256

ENV REPO_VERSIONING_BUNDLE_CACHE 64

5

current container resources are set to 3 cpu and 4gb ram

6

We have been running in kubernetes for nearly 4 years… every now and again this happens though…

Config is all default.

J

7

Some things to consider:

  1. The max heap is quite low considering the bundle caches take up 320Mb. It depends on your app really but maybe you need to bump up the caches and max heap.
  2. Are you taking a frequent index backup? If not then it might be indexing the delta between the backup and the repository and that takes time.
  3. Are you clearing the database from old revision IDs? See Repository Maintenance - Bloomreach Experience Manager (PaaS/Self-Hosted) - The Fast and Flexible Headless CMS and On-Premise Kubernetes Setup - Bloomreach Experience Manager (PaaS/Self-Hosted) - The Fast and Flexible Headless CMS
8

1: okay cool, what do you suggest?
2: yes we are every 4 hours.
3: no…??

9

Well, I’m not really sure why this happens. You can try to increase the bundle cache size, and also increase the heap by the same amount. But on startup you might expect the cache to be empty, so I won’t guarantee this is a solution. I would start by doubling the bundle cache. If that doesn’t help, I don’t think you need to go higher.

10

if we were to delete all the data in this table ’ ```
REPOSITORY_JOURNAL


What would happen?
11

Please see Repository Maintenance - Bloomreach Experience Manager (PaaS/Self-Hosted) - The Fast and Flexible Headless CMS

cleaning the journal is a good thing, but some care must be taken.

12

Hi all, thanks for your responses. WE found that the jounral was indeed massive… so we have cleaned it all up.

1 Like