Fabric container loosing Fabric configuration after a restart

Solution Verified - Updated -

Environment

  • JBoss Fuse 6.1
  • Fabric

Issue

  • After restarting a container which was previously added to fabric and added to the ensemble, it seems to have no knowledge of the fabric.
  • Zookeeper URL not updated correctly when creating ensemble
    I've created three containers. I've then created an ensemble by creating a fabric on the first (fabric:create) then adding in the other two (fabric:ensemble-add). This all works fine, and the ensemble appears correctly. However, if I stop (bin/stop) one of the containers then restart (bin/start) it, it doesn't appear back in the fabric. I see the following error in the logs:
Unable to start blueprint container for bundle org.apache.karaf.features.management due to unresolved dependencies [(objectClass=org.apache.karaf.features.FeaturesService)]

Resolution

This issue is being tracked by FABRIC-1078

Root Cause

The exact problem is this:

  • fabric:ensemble-add/remove creates new profiles and adds them to selected containers and updates ZK nodes responsible for storing new ZK URL
  • listeners attached to Git data store (which keeps profiles data) and ZK nodes (which store ZK URL and ensemble members) are fired
  • when root container is finally assigned new profile (created for the purpose of new ensemble), FabricConfigAdminBridge updates config admin PIDs, among others: io.fabric8.zookeeper which stores zookeeper.url property
  • before config.update(props) there's invocation of config.setBundleLocation(null) which ... causes the old configuration to be stored silently (org.apache.felix.cm.impl.ConfigurationBase#storeSilently()), but using different thread, which may be activated after config.update() with new values.

Diagnostic Steps

This issue can be verified by performing the following steps:

Setup fabric:

JBossFuse:karaf@fusepro> fabric:container-list
[id]                           [version] [connected] [profiles]                                         [provision status]
fusepro*                       1.0       true        fabric, fabric-ensemble-0000-1, jboss-fuse-full    success
oxygen                         1.0       true        fabric                                             success
oxygen-2                       1.0       true        fabric                                             success
JBossFuse:karaf@fusepro>

Add the remote root containers to the fabric:

JBossFuse:karaf@fusepro> fabric:ensemble-add oxygen oxygen-2
This will change of the zookeeper connection string.
Are you sure want to proceed(yes/no):yes
Updated Zookeeper connection string: 192.168.1.17:2182,10.211.55.6:2181,10.211.55.6:2182
JBossFuse:karaf@fusepro>

Verify the profiles have been updated:

JBossFuse:karaf@fusepro> fabric:container-list
[id]                           [version] [connected] [profiles]                                         [provision status]
fusepro*                       1.0       true        fabric, jboss-fuse-full, fabric-ensemble-0001-1    success
oxygen                         1.0       true        fabric, fabric-ensemble-0001-2                     success
oxygen-2                       1.0       true        fabric, fabric-ensemble-0001-3                     success
JBossFuse:karaf@fusepro>

Verify the zookeeper.url is properly set:

JBossFuse:karaf@fusepro> config:proplist -p io.fabric8.zookeeper
   service.pid = io.fabric8.zookeeper
   zookeeper.password = ZKENC=YWRtaW4=
   zookeeper.url = 192.168.1.17:2182,10.211.55.6:2181,10.211.55.6:2182
   fabric.zookeeper.pid = io.fabric8.zookeeper
JBossFuse:karaf@fusepro>

Terminate the container with the exit command:

JBossFuse:karaf@fusepro> exit
jasons-mbp:jboss-fuse-6.1.0.redhat-379 jsherman$

Restart the container:

jasons-mbp:jboss-fuse-6.1.0.redhat-379 jsherman$ ./bin/fuse

Check the zookeeper.url property:

JBossFuse:karaf@fusepro> config:proplist -p io.fabric8.zookeeper
   service.pid = io.fabric8.zookeeper
   zookeeper.password = ZKENC=YWRtaW4=
   zookeeper.url = 192.168.1.17:2181
   fabric.zookeeper.pid = io.fabric8.zookeeper
JBossFuse:karaf@fusepro>

Fabric commands no longer working:

JBossFuse:karaf@fusepro> fabric:container-list
Command not found: fabric:container-list
JBossFuse:karaf@fusepro>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments