Failed to initialize workspace 'default' in Jackrabbit / Guvnor Repository in BRMS 5.3

Issue

• In a two node setup of a production cluster, users start getting the following exception after a restart.

2015-09-08 08:08:26,410 ERROR [org.apache.jackrabbit.core.persistence.pool.BundleDbPersistenceManager] (main) failed to read bundle: dae3d122-3e13-46b8-aa1f-d79d02d18691: java.sql.SQLException: Failed to parse bundle dae3d122-3e13-46b8-aa1f-d79d02d18691
2015-09-08 08:08:26,411 ERROR [org.apache.jackrabbit.core.query.OnWorkspaceInconsistency] (main) Node /jcr:system/jcr:versionStorage/15/84/bf/1584bf47-10ba-49f2-820c-7f2113d462f7/1.0 (2a29eb10-774a-4b56-8e39-7842043013ff) has missing child 'jcr:frozenNode' (dae3d122-3e13-46b8-aa1f-d79d02d18691)
2015-09-08 08:08:26,411 ERROR [org.apache.jackrabbit.core.RepositoryImpl] (main) Failed to initialize workspace 'default'
javax.jcr.RepositoryException: Unable to initialize query handler: org.apache.jackrabbit.core.query.lucene.SearchIndex@3f2f5317
    at org.apache.jackrabbit.core.config.RepositoryConfigurationParser$1.getQueryHandler(RepositoryConfigurationParser.java:637)
    at org.apache.jackrabbit.core.config.RepositoryConfig.getQueryHandler(RepositoryConfig.java:1032)
    at org.apache.jackrabbit.core.SearchManager.<init>(SearchManager.java:174)
    at org.apache.jackrabbit.core.RepositoryImpl.getSystemSearchManager(RepositoryImpl.java:630)
    at org.apache.jackrabbit.core.RepositoryImpl.access$100(RepositoryImpl.java:124)
    at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.getSearchManager(RepositoryImpl.java:1860)
    at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.doPostInitialize(RepositoryImpl.java:2095)
    at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.initialize(RepositoryImpl.java:2000)
    at org.apache.jackrabbit.core.RepositoryImpl.initStartupWorkspaces(RepositoryImpl.java:533)
    at org.apache.jackrabbit.core.RepositoryImpl.<init>(RepositoryImpl.java:342)
    at org.apache.jackrabbit.core.RepositoryImpl.create(RepositoryImpl.java:605)
    at org.apache.jackrabbit.core.TransientRepository$2.getRepository(TransientRepository.java:232)
    at org.apache.jackrabbit.core.TransientRepository.startRepository(TransientRepository.java:280)
    at org.apache.jackrabbit.core.TransientRepository.login(TransientRepository.java:376)
    at org.apache.jackrabbit.commons.AbstractRepository.login(AbstractRepository.java:123)
    at org.drools.repository.JCRRepositoryConfigurator.login(JCRRepositoryConfigurator.java:95)
    at org.drools.repository.RulesRepositoryConfigurator.login(RulesRepositoryConfigurator.java:85)
    at org.drools.guvnor.server.repository.RepositoryStartupService.newSession(RepositoryStartupService.java:219)
    at org.drools.guvnor.server.repository.RepositoryStartupService.create(RepositoryStartupService.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
    at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
    at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
    at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
    at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
    at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:185)
    at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:103)
    at org.drools.guvnor.server.repository.RepositoryStartupService_$$_javassist_seam_2.create(RepositoryStartupService_$$_javassist_seam_2.java)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
    at org.jboss.seam.util.Reflections.invokeAndWrap(Reflections.java:144)
    at org.jboss.seam.Component.callComponentMethod(Component.java:2275)
    at org.jboss.seam.Component.callCreateMethod(Component.java:2198)
    at org.jboss.seam.Component.newInstance(Component.java:2158)
    at org.jboss.seam.contexts.Contexts.startup(Contexts.java:305)
    at org.jboss.seam.contexts.Contexts.startup(Contexts.java:279)
    at org.jboss.seam.contexts.ServletLifecycle.endInitialization(ServletLifecycle.java:142)
    at org.jboss.seam.init.Initialization.init(Initialization.java:744)
    at org.jboss.seam.servlet.SeamListener.contextInitialized(SeamListener.java:36)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3910)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4389)
    at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeployInternal(TomcatDeployment.java:313)
    at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeploy(TomcatDeployment.java:145)
    at org.jboss.web.deployers.AbstractWarDeployment.start(AbstractWarDeployment.java:461)
    at org.jboss.web.deployers.WebModule.startModule(WebModule.java:122)
    at org.jboss.web.deployers.WebModule.start(WebModule.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
    at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
    at $Proxy38.start(Unknown Source)
    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
    at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
    at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
    at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
    at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
    at org.jboss.system.ServiceController.start(ServiceController.java:460)
    at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
    at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
    at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)
    at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)
    at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)
    at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
    at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
    at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
    at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)
    at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
    at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
    at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
    at org.jboss.Main.boot(Main.java:229)
    at org.jboss.Main$1.call(Main.java:561)
    at org.jboss.Main$1.call(Main.java:557)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error indexing workspace
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createInitialIndex(MultiIndex.java:402)
    at org.apache.jackrabbit.core.query.lucene.SearchIndex.doInit(SearchIndex.java:525)
    at org.apache.jackrabbit.core.query.AbstractQueryHandler.init(AbstractQueryHandler.java:78)
    at org.apache.jackrabbit.core.config.RepositoryConfigurationParser$1.getQueryHandler(RepositoryConfigurationParser.java:634)
    ... 119 more
Caused by: org.apache.jackrabbit.core.state.NoSuchItemStateException: dae3d122-3e13-46b8-aa1f-d79d02d18691
    at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:278)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1215)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1221)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1221)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1221)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1221)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1221)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createIndex(MultiIndex.java:1221)
    at org.apache.jackrabbit.core.query.lucene.MultiIndex.createInitialIndex(MultiIndex.java:395)
    ... 122 more

• This ERROR is shown in the log file of in one node only. The other node works without this exception. Event though users tried to clean the local search index of each Jackrabbit working directory (the directory where the repository.xml is located), it does not solve the issue. Afther this exception, the affected host becomes unusable. How to fix this problem?