Repository issue after removing entry from .cnd file

Experience Manager (PaaS/OnPrem)
1

Hello,

I found out after deleting an entry from a .cnd file that its not good to do that. Happened when trying to revert some commits from git and the current situation is that I have a node that still references the deleted entry.

28.11.2023 11:55:48 [main] INFO  [org.apache.jackrabbit.core.persistence.bundle.ConsistencyCheckerImpl.info():389] No 'lost+found' node specified: orphans cannot be fixed
28.11.2023 11:55:49 [main] INFO  [org.apache.jackrabbit.core.query.lucene.SearchIndex.doInit():659] Index initialized: /opt/cms/tomcat/../repository/workspaces/default/index Version: 3
28.11.2023 11:56:00 [main] ERROR [org.onehippo.cm.engine.ConfigurationServiceImpl.applyConfig():959] Failed to apply config
javax.jcr.nodetype.NoSuchNodeTypeException: {http://www.onehippo.org/gmshippo/nt/1.0}SubsidiaryLocationsConfig
	at org.apache.jackrabbit.core.nodetype.NodeTypeRegistry.getEffectiveNodeType(NodeTypeRegistry.java:1043) ~[jackrabbit-core-2.21.6-h2.jar:2.21.6-h2]
	at org.apache.jackrabbit.core.nodetype.NodeTypeRegistry.getEffectiveNodeType(NodeTypeRegistry.java:478) ~[jackrabbit-core-2.21.6-h2.jar:2.21.6-h2]
	at org.apache.jackrabbit.core.nodetype.NodeTypeManagerImpl.getNodeType(NodeTypeManagerImpl.java:186) ~[jackrabbit-core-2.21.6-h2.jar:2.21.6-h2]
	at org.apache.jackrabbit.core.NodeImpl.addNode(NodeImpl.java:1255) ~[jackrabbit-core-2.21.6-h2.jar:14.7.3]
	at org.apache.jackrabbit.core.session.AddNodeOperation.perform(AddNodeOperation.java:111) ~[jackrabbit-core-2.21.6-h2.jar:2.21.6-h2]
	at org.apache.jackrabbit.core.session.AddNodeOperation.perform(AddNodeOperation.java:37) ~[jackrabbit-core-2.21.6-h2.jar:2.21.6-h2]
	at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216) ~[jackrabbit-core-2.21.6-h2.jar:2.21.6-h2]
	at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91) ~[jackrabbit-core-2.21.6-h2.jar:14.7.3]
	at org.apache.jackrabbit.core.NodeImpl.addNodeWithUuid(NodeImpl.java:1818) ~[jackrabbit-core-2.21.6-h2.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.addNode(ConfigurationConfigService.java:784) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteChildNodesDelta(ConfigurationConfigService.java:695) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteNodeDelta(ConfigurationConfigService.java:451) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteChildNodesDelta(ConfigurationConfigService.java:705) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteNodeDelta(ConfigurationConfigService.java:451) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteChildNodesDelta(ConfigurationConfigService.java:705) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteNodeDelta(ConfigurationConfigService.java:451) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteChildNodesDelta(ConfigurationConfigService.java:705) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteNodeDelta(ConfigurationConfigService.java:451) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteChildNodesDelta(ConfigurationConfigService.java:705) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteNodeDelta(ConfigurationConfigService.java:451) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationConfigService.computeAndWriteDelta(ConfigurationConfigService.java:322) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationServiceImpl.applyConfig(ConfigurationServiceImpl.java:947) [hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationServiceImpl.init(ConfigurationServiceImpl.java:436) [hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationServiceImpl.start(ConfigurationServiceImpl.java:206) [hippo-repository-engine-14.7.3.jar:14.7.3]
	at com.onehippo.repository.HippoEnterpriseRepository.initializeConfiguration(HippoEnterpriseRepository.java:180) [hippo-enterprise-repository-engine-14.7.3.jar:14.7.3]
	at org.hippoecm.repository.LocalHippoRepository.initialize(LocalHippoRepository.java:292) [hippo-repository-engine-14.7.3.jar:14.7.3]
	at com.onehippo.repository.HippoEnterpriseRepository.create(HippoEnterpriseRepository.java:65) [hippo-enterprise-repository-engine-14.7.3.jar:14.7.3]
	at com.onehippo.repository.HippoEnterpriseRepository.create(HippoEnterpriseRepository.java:55) [hippo-enterprise-repository-engine-14.7.3.jar:14.7.3]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_382]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_382]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
	at org.hippoecm.repository.HippoRepositoryFactory.getHippoRepository(HippoRepositoryFactory.java:147) [hippo-repository-connector-14.7.3.jar:14.7.3]
	at org.hippoecm.repository.RepositoryServlet.init(RepositoryServlet.java:207) [hippo-repository-servlets-14.7.3.jar:14.7.3]
	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1123) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1078) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:971) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4868) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5184) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1850) [catalina.jar:9.0.17]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_382]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_382]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.17]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [?:1.8.0_382]
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1577) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:929) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) [catalina.jar:9.0.17]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_382]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.17]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [?:1.8.0_382]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:928) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.Catalina.start(Catalina.java:634) [catalina.jar:9.0.17]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_382]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_382]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350) [bootstrap.jar:9.0.17]
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492) [bootstrap.jar:9.0.17]
28.11.2023 11:56:04 [main] ERROR [org.hippoecm.repository.RepositoryServlet.init():249] Error while setting up JCR repository: 
javax.jcr.RepositoryException: unchecked exception: java.lang.IllegalStateException: Pending changes at this moment not allowed
	at org.hippoecm.repository.HippoRepositoryFactory.getHippoRepository(HippoRepositoryFactory.java:161) ~[hippo-repository-connector-14.7.3.jar:14.7.3]
	at org.hippoecm.repository.RepositoryServlet.init(RepositoryServlet.java:207) [hippo-repository-servlets-14.7.3.jar:14.7.3]
	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1123) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1078) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:971) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4868) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5184) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1850) [catalina.jar:9.0.17]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_382]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_382]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.17]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [?:1.8.0_382]
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1577) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:929) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) [catalina.jar:9.0.17]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) [catalina.jar:9.0.17]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_382]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.17]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [?:1.8.0_382]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:928) [catalina.jar:9.0.17]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.17]
	at org.apache.catalina.startup.Catalina.start(Catalina.java:634) [catalina.jar:9.0.17]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_382]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_382]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350) [bootstrap.jar:9.0.17]
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492) [bootstrap.jar:9.0.17]
Caused by: java.lang.IllegalStateException: Pending changes at this moment not allowed
	at org.onehippo.cm.engine.ConfigurationServiceImpl.applyPostMigrators(ConfigurationServiceImpl.java:577) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationServiceImpl.init(ConfigurationServiceImpl.java:492) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at org.onehippo.cm.engine.ConfigurationServiceImpl.start(ConfigurationServiceImpl.java:206) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at com.onehippo.repository.HippoEnterpriseRepository.initializeConfiguration(HippoEnterpriseRepository.java:180) ~[hippo-enterprise-repository-engine-14.7.3.jar:14.7.3]
	at org.hippoecm.repository.LocalHippoRepository.initialize(LocalHippoRepository.java:292) ~[hippo-repository-engine-14.7.3.jar:14.7.3]
	at com.onehippo.repository.HippoEnterpriseRepository.create(HippoEnterpriseRepository.java:65) ~[hippo-enterprise-repository-engine-14.7.3.jar:14.7.3]
	at com.onehippo.repository.HippoEnterpriseRepository.create(HippoEnterpriseRepository.java:55) ~[hippo-enterprise-repository-engine-14.7.3.jar:14.7.3]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_382]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_382]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
	at org.hippoecm.repository.HippoRepositoryFactory.getHippoRepository(HippoRepositoryFactory.java:147) ~[hippo-repository-connector-14.7.3.jar:14.7.3]
	... 45 more
28.11.2023 11:58:18 [http-nio-8080-exec-1] ERROR [org.hippoecm.hst.container.HstFilter.doFilter():47] The HST Container Services are not initialized yet.

Bloomreach version: 14.7.3

Fixing attemtps up to this point:

  • Re-adding the entry to the .cnd file, with the same property name
  • Deleting the ‘repository’ folder to force reindexing the content
  • Using hippo-addon-checker - couldn’t get it working
  • Setting ‘consistencyCheck’ and ‘consistencyFix’ to true in conf/repository.xml

I am currently trying to get the repository back up and running

Please help, Thank you!

2

Hope this isn’t production. Breaking changes in the cnd are not allowed. Basically removing anything (and a rename is a deletion and addition). I would unrevert the reverted commits first. If that works, you can remove all the artifacts you don’t want (code, configuration, content, …) buy never remove anything from the cnd. If the last is desired there are ways of doing it, but in general it isn’t considered worth the effort.

Not quite sure where you are getting stuck in the repo at the moment, but if you aren’t in production you should restore from a previous back up. Then you can go ahead and delete anything from the cnd that hasn’t made it to the repository yet. If your changes have reached production, or any system you don’t want to restore from backup, then don’t change the cnd with removals.

3

Thank you for your response!

Luckily its Dev. But I am unable restore the repository from a back up since we don’t have back-ups for dev (downgraded machine) and it isn’t managed by myself. What I am currently trying to accomplish is a reset/override of the repository with the development data such that the CMS will start up.

4

@gf23 I think you can try to sync back prod to dev env.
Just sync database and run CMS with a clean index folder, it would create a new clean index when restarting.
If you want to sync more faster, you can use an index backup also.

5

To resolve this issue, you need to identify and update any source files that reference the removed entry. This may involve removing references to the entry, replacing them with alternative references, or modifying the code to accommodate the removal of the entry.