Chapter 18. Enabling SSL for JMX Connectors

Abstract

Under Fabric, to enable SSL for JMX connectors:
  1. Obtain or create a certificate for each server host that will be used in the fabric cluster.
  2. Import the certificates with chain into a Java keystore (JKS).
  3. Select a standard location for keystore and related artifacts.
  4. Create a blueprint configuration for loading the keystore.
  5. Set keystore environment variables on Fuse containers.
  6. Start Fuse and create Fabric cluster.
  7. Configure the cluster for JMX.
  8. Upgrade Fuse containers.
  9. Test the configuration.
The procedure described here assumes that you can use the same filenames and the same file locations on all hosts in the fabric cluster. If your configuration does not allow this, you can use environment variables to customize the configuration on individual hosts. The recommendation is to work through the steps described here and then see Fabric Profiles for information about flexibly configuriing particular hosts.

Obtain or create certificates

Obtain or create a certificate for each server host to be used in the fabric cluster.
The process of enabling SSL for JMX connectors is easier if the certificates are signed by a certificate authority (CA). Steps for creating a certificate authority are outside the scope of this documentation. An example of the process for creating a root and intermediate certificate authority and signing server certificates is in the JBoss Fuse Security Guide appendix.

Import certificates with chain

Import the certificates with chain into a Java keystore (JKS). This is a generic procedure documented elsewhere. While this step overlaps with the process for creating certificates, the details are here because you must customize some of the final steps for creating the certificates for this example.
If you have PEM-encoded (Privacy Enhanced Mail) certificates then you must include the chain with the root CA, intermediate CA and server certificate in the chain. To do this, follow these steps:
  1. Concatenate the PEM-encoded certificates into a single chain, for example: 
    $ cat intermediate/certs/serverx.example.com.cert.pem intermediate/certs/ca-chain.cert.pem > intermediate/certs/serverx.example.com.cert.export.pem
  2. Convert the chain file into a PKCS12 keystore, for example: 
    $ openssl pkcs12 -export -inkey intermediate/private/serverx.example.com.key.pem -in intermediate/certs/serverx.example.com.cert.export.pem -out serverx.chain.pkcs12
  3. Import the keystore and CA certificates into a Java keystore, for example: 
    $ keytool -importkeystore -trustcacerts -srckeystore serverx.chain.pkcs12 -srcstoretype PKCS12 -destkeystore server-chain.jks
$ keytool -import -file intermediate/certs/intermediate.cert.pem -alias intermediate -trustcacerts -keystore server-chain.jks -storepass changepass
$ keytool -import -file certs/ca.cert.pem -alias root -trustcacerts -keystore server-chain.jks -storepass changepass
  4. Ensure that the alias for the server key/certificate is the same for each keystore. This avoids needing a separate configuration for each host, for example: 
    keytool -changealias -alias "1" -destalias "server" -keypass changepass -keystore server-chain.jks -storepass changepass
    In the previous example, 1 is the default alias of the server key originally imported into the keystore.
The result of these steps includes:
  • A keystore for each host in the cluster.
  • Each keystore has the same final filename. After you create and move them, you can rename all of them as something like server-chain.jks.
  • Each keystore contains the server key/certificate for the host on which it is deployed.
  • Each keystore uses the same alias for the server certificate, for example, server or jboss. This makes the configuration portable across hosts.
  • The CA certificates are also in the keystore and are marked as trusted so the keystore can also function as a trust store.
You might be able to configure keystores with other types of certificates, such as self-signed certificates or a single certificate with multiple server names. Doing this is outside the scope of this documentation.

Select standard location for keystore

Select a standardized location for the keystore file and related artifacts. For example, on each server, /opt/fuse/keystore could contain the keystore file for that server.

Create blueprint configuration for loading keystore

Create a blueprint configuration for loading the keystore.
You use the blueprint configuration later when you assign it to a profile. On each server, the blueprint configuration file should have the same name and contents. For example, the /opt/fuse/keystore directory might contain the jmx.keystore.xml file with this content: 
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
     xmlns:jaas="http://karaf.apache.org/xmlns/jaas/v1.0.0"
     default-activation="lazy">

     <jaas:keystore name="server_keystore"
          rank="1"
          path="file:///opt/fuse/keystore/server-chain.jks"
          keystorePassword="changepass"
          keyPasswords="server=changepass" />
</blueprint>

Set keystore environment variables on Fuse containers

Set environment variables on each Fuse container to specify that the keystore file is the trust store for the container. For example, you can use the EXTRA_JAVA_OPTS parameter in $KARAF_HOME/bin/setenv: 
export EXTRA_JAVA_OPTS="-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass"
This ensures that the JMX client trusts the certificates passed by other hosts because of the trusted CAs.

Start Fuse and create Fabric cluster

Start your Fuse instance and create a Fabric cluster, for example: 
> fabric:create --wait-for-provisioning --zookeeper-password admin --global-resolver manualip --resolver manualip --manual-ip node11.redhat.com --bind-address node11.redhat.com
> container-create-ssh --resolver manualip --manual-ip node12.redhat.com --bind-address node12.redhat.com --zookeeper-password admin --user rlee --private-key /home/rlee/.ssh/id_rsa --host node12.redhat.com --profile fabric --profile jboss-fuse-full --jvm-opts '-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass' --path /opt/fuse/containers node12
> container-create-ssh --resolver manualip --manual-ip node13.redhat.com --bind-address node13.redhat.com --zookeeper-password admin --user rlee --private-key /home/rlee/.ssh/id_rsa --host node13.redhat.com --profile fabric --profile jboss-fuse-full --jvm-opts '-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass' --path /opt/fuse/containers node13
> fabric:ensemble-add node12 node13
If you already have a working ensemble then you must add the JVM parameters in the setenv or setenv.bat scripts for each ensemble container and then restart each container one by one. For example, the setenv script in the bin directory of the node13 container should be something like this: 
export JAVA_OPTS="-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass"
If you have existing child containers then you need to update their JVM properties to include the trust store and the trust store password. You can do this by using the Hawtio console or by editing the instance.properties file in the parent container instances directory, for example: 
item.1.name = apps11
item.1.root = false
item.1.loc = /opt/fuse/jboss-fuse-6.3.0.redhat-187/instances/apps11
item.1.pid = 13219
item.1.opts = -server -Dcom.sun.management.jmxremote -Dorg.jboss.gravia.repository.storage.dir=data/repository -Dzookeeper.url=\"node11.redhat.com:2182,node12.redhat.com:2182,node13.redhat.com:2182\" -Dzookeeper.password.encode=\"true\" -Dzookeeper.password=\"admin\" -Xmx768m -XX:MaxPermSize=256m -Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -Dbind.address=node11.redhat.com -Dlocal.resolver=manualip -Dmanualip=node11.redhat.com -Dio.fabric8.datastore.gitRemotePollInterval=60000 -Dio.fabric8.datastore.importDir=fabric -Dio.fabric8.datastore.felix.fileinstall.filename=file:/opt/fuse/jboss-fuse-6.3.0.redhat-187/etc/io.fabric8.datastore.cfg -Dio.fabric8.datastore.service.pid=io.fabric8.datastore -Dio.fabric8.datastore.component.name=io.fabric8.datastore

Configure the cluster for JMX

The previous steps put the required infrastructure in place. This includes certificates and blueprint configurations in common locations and updated JVM arguments to scripts. To configure the cluster for JMX over SSL/TLS:
  1. Create a new version for the changes. This lets you roll back or roll forward as needed. For example: 
    > fabric:version-create 1.1
  2. Modify the profile. You might want to use the default profile for configuration because it is the parent of all other profiles. It is also possible to add a special profile for SSL configuration and add the following properties to that. Using a special SSL profile means that you must assign the profile to each container in the cluster. The following example edits the default profile: 
    > profile-edit --bundle 'blueprint:file:///opt/fuse/keystore/jmx.keystore.xml;start-level=29' default 1.1
> profile-edit --pid org.apache.karaf.management/secured=true default 1.1
> profile-edit --pid org.apache.karaf.management/secureProtocol=TLSv1 default 1.1
> profile-edit --pid org.apache.karaf.management/keyAlias=server default 1.1
> profile-edit --pid org.apache.karaf.management/keyStore=server_keystore default 1.1
> profile-edit --pid org.apache.karaf.management/trustStore=server_keystore default 1.1
> profile-edit --pid org.apache.karaf.management/keyStoreAvailabilityTimeout=60000 default 1.1
> profile-edit --pid org.apache.karaf.management/fabric.config.merge=true default 1.1
To ensure availability of the certificate when the JMX connector is initialized, the profile specifies start-level for the keystore blueprint configuration. To ensure correct operation on child containers, the fabric.config.merge property is enabled. To make the profile portable, the profile uses the generic key alias of server.

Upgrade Fuse containers

To apply the SSL JMX configuration, upgrade each container in turn. The recommendation is to wait between container upgrades to avoid breaking Zookeeper quorum. For example: 
> container-upgrade 1.1 node11
> container-upgrade 1.1 node12
> container-upgrade 1.1 node13

Test the configuration

Test the configuration by starting jconsole with a trust store that contains the CA certificates used to sign the server keys or with a trust store that contains all self-signed server keys, for example: 
jconsole -J-Djavax.net.ssl.trustStore=client1-chain.jks -J-Djavax.net.ssl.trustStoreType=JKS -J-Djavax.net.ssl.trustStorePassword=changepass
You should be able to negotiate an SSL connection by specifying the JMX address for the container and supplying the username and password of an administrator (admin/admin). You should not receive a warning that prompts you to indicate whether you want to try an unencrypted connection. For example: 
service:jmx:rmi://node11.redhat.com:44444/jndi/rmi://node11.redhat.com:1099/karaf-node11
To add new containers to the cluster, specify the version. Once provisioned, the new containers should be enabled for JMX over SSL. For example, the following command creates an SSH container: 
container-create-ssh --resolver manualip --manual-ip node1.redhat.com --bind-address node1.redhat.com --zookeeper-password admin --user rlee --private-key /home/rlee/.ssh/id_rsa --host node1.redhat.com --profile fabric --profile jboss-fuse-full --jvm-opts '-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass' --path /opt/fuse/containers --version 1.1 node1
The following command creates a child container: 
container-create-child --resolver manualip --manual-ip node11.redhat.com --version 1.1 --bind-address node11.redhat.com --zookeeper-password admin --jmx-user rlee --jmx-password admin --jvm-opts '-Djavax.net.ssl.trustStore=/opt/fuse/keystore/server-chain.jks -Djavax.net.ssl.trustStorePassword=changepass' node11 apps11
The recommendation is to specify the resolver and the bind address and use fully-qualified names. While doing this might not be required, experience shows that it lets you avoid issues with JMX and, in particular, Zookeeper.