9.2. Administering a Fabric Ensemble

Recommendations for an ensemble in production

To assure high availability of the Fabric registry in a production environment, it is recommended that you observe the following guidelines for a Fabric ensemble:
  • Deploy a minimum of five Fabric servers in production (if one server is taken down for maintenance, one other server can fail, and the Fabric registry will still be available).
  • Fabric servers should be deployed on separate host machines.
  • Each Fabric server should only have a Fabric registry agent deployed inside it. No other profiles should be deployed in it.
  • The size of the ensemble should be fixed at the outset, and not changed later (if you subsequently add or remove containers from the ensemble, the ZooKeeper IP ports would be re-assigned).

Expanding the ensemble

Once you have an initial ensemble, consisting of one Fabric server, you can expand the ensemble by invoking the fabric:ensemble-add command. To expand the ensemble, perform the following steps:
  1. Create some new managed containers in the current fabric, which you can then add to the ensemble. Use the default profile for these new containers. For a production environment, it is recommended that you create at least four new managed containers (must be an even number), each running on their own host.
  2. While logged on to a container in the fabric, use the fabric:ensemble-add command to add the managed containers to the ensemble. For example, given the four managed containers, container1, container2, container3, and container4, you would enter the following command: 
    > fabric:ensemble-add container1 container2 container3 container4
    Note
    You must specify an even number of containers to the fabric:ensemble-add command.
  3. To check that the ensemble has been successfully created, invoke the fabric:container-list command.
Important
Do not attempt to expand (or shrink) a Fabric ensemble in a production environment. When you add containers to (or remove containers from) an ensemble, the ZooKeeper IP ports are all re-assigned, which typically causes the containers in the fabric to lose connectivity with the ensemble.

Changing an ensemble password

Each ensemble has a password. When you create an ensemble you can specify the --zookeeper-password option to define the ensemble password. If you do not then Zookeeper creates an ensemble password for you. You might want to change the ensemble password when a test ensemble becomes a production ensemble. To display the current ensemble password, execute the following command: 
> fabric:ensemble-password
To change the ensemble password, execute the fabric:ensemble-password command and specify the new password. For example, to change the password to tiger, enter the following command: 
> fabric:ensemble-password tiger
Zookeeper responds with a message to wait while it updates the password on each node. A node must be running for it to be updated with the new password. Zookeeper also instructs you to enter the following command when the prompt reappears. This command saves the updated password: 
> fabric:ensemble-password --commit

Taking a Fabric server down for maintenance

If you need to perform any maintenance on the host where a Fabric server is running, you can do this while maintaining availability of the Fabric registry, so long as a quorum (more than half) of the Fabric servers are still running. To stop a Fabric server, simply invoke the fabric:container-stop command, specifying the name of the Fabric server.
In general, it is recommended to have at least five Fabric servers in the ensemble. Three is not an adequate number. For example, with three servers in the ensemble consider what happens when you take a Fabric server down for maintenance. The two remaining Fabric servers form a quorum, but there is now no tolerance for failure. If one of the remaining Fabric servers fails, the whole fabric fails. In order to maintain high availability during maintenance, it is therefore essential to have at least five Fabric servers in the ensemble.