Errors related to targeting/relevance in Elastic Search

I’ve noticed similar errors to this in some of the Bloomreach Experience Manager over time but recently these began happening very consistently about every 3-4 seconds. I can’t find much details on troubleshooting specific Elastic Search errors other than connection issues in the Relevance Module Troubleshooting documentation but although I’m also seeing blank data in the Trend tab the cause doesn’t seem to be a java.net.ConnectException: Connection refused error. Any pointers on where to look or what to do next to resolve this would be appreciated. This instance is using on HippoCMS/brXM v12.6.10.

18.06.2021 12:20:51 ERROR pool-22-thread-1 [TargetingJob.run:98] Job /targeting:targeting/targeting:dataflow/modelTrainer failed
com.onehippo.cms7.targeting.storage.RuntimeDataStoreException: Failed to create targeting:visits
        at com.onehippo.cms7.targeting.DataStoreKeeper.creationSeriouslyFailed(DataStoreKeeper.java:217) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreKeeper.tryToCreateStore(DataStoreKeeper.java:151) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreKeeper.lambda$get$0(DataStoreKeeper.java:86) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: com.onehippo.cms7.targeting.storage.DataStoreCreationException: Failed to upload mapping
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.updateMapping(AbstractElasticStore.java:213) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.initialize(AbstractElasticStore.java:117) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.<init>(AbstractElasticStore.java:105) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticVisitStore.<init>(ElasticVisitStore.java:162) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticStoreFactory.createVisitStore(ElasticStoreFactory.java:139) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.profiling.ProfilingDataStoreFactory.createVisitStore(ProfilingDataStoreFactory.java:54) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreServiceImpl.createVisitStore(DataStoreServiceImpl.java:356) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreKeeper.tryToCreateStore(DataStoreKeeper.java:143) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        ... 2 more
Caused by: java.io.IOException: {"root_cause":[{"type":"remote_transport_exception","reason":"[HOSTNAME123][10.100.192.57:9302][indices:admin/mapping/put]"}],"type":"illegal_argument_exception","reason":"object mapping [requests] can't be changed from non-nested to nested"}
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.validateResultIsSuccessful(AbstractElasticStore.java:219) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.updateMapping(AbstractElasticStore.java:206) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.initialize(AbstractElasticStore.java:117) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.<init>(AbstractElasticStore.java:105) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticVisitStore.<init>(ElasticVisitStore.java:162) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticStoreFactory.createVisitStore(ElasticStoreFactory.java:139) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.profiling.ProfilingDataStoreFactory.createVisitStore(ProfilingDataStoreFactory.java:54) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreServiceImpl.createVisitStore(DataStoreServiceImpl.java:356) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreKeeper.tryToCreateStore(DataStoreKeeper.java:143) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        ... 2 more
...skipping...
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticStoreFactory.createVisitStore(ElasticStoreFactory.java:139) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.profiling.ProfilingDataStoreFactory.createVisitStore(ProfilingDataStoreFactory.java:54) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreServiceImpl.createVisitStore(DataStoreServiceImpl.java:356) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreKeeper.tryToCreateStore(DataStoreKeeper.java:143) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        ... 2 more
Caused by: java.io.IOException: {"root_cause":[{"type":"remote_transport_exception","reason":"[HOSTNAME123][10.100.192.57:9302][indices:admin/mapping/put]"}],"type":"illegal_argument_exception","reason":"object mapping [requests] can't be changed from non-nested to nested"}
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.validateResultIsSuccessful(AbstractElasticStore.java:219) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.updateMapping(AbstractElasticStore.java:206) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.initialize(AbstractElasticStore.java:117) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.AbstractElasticStore.<init>(AbstractElasticStore.java:105) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticVisitStore.<init>(ElasticVisitStore.java:162) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.elastic5.ElasticStoreFactory.createVisitStore(ElasticStoreFactory.java:139) ~[hippo-addon-targeting-state-elastic5-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.storage.profiling.ProfilingDataStoreFactory.createVisitStore(ProfilingDataStoreFactory.java:54) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreServiceImpl.createVisitStore(DataStoreServiceImpl.java:356) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        at com.onehippo.cms7.targeting.DataStoreKeeper.tryToCreateStore(DataStoreKeeper.java:143) ~[hippo-addon-targeting-engine-5.6.10-1.jar:5.6.10-1]
        ... 2 more

Recently this issue had led us into a condition where there was an open file leak which caused our site to become unavailable. The only way of alleviating the issue was to reboot the server at a frequency of at least once per 1 hour and 50 minutes or as we had found later creating a new blank index and pointing to that.

Does anybody have a clue why the _mapping in some cases gets into a state where updateMapping fails due to the above error? I can see clearly by looking at the json returned from _mapping that the following message is correct in that we have a nested structure for requests whereas other environments where we do not see this error we have what looks like a very different schema.

object mapping [requests] can’t be changed from non-nested to nested

Although we are still investigating this we are becoming fairly certain that the issue described above was likely caused by the creation of a new Elastic Search index with the same name while Bloomreach Experience Manager was running.

Hi

I can offer no direct insight on your issue but I highly suggest you open a ticket through your customer account

We have an open ticket already @jasper.floor, thanks you for the suggestion.