Chapter 3. Configuring Java applications

To enable Cryostat to gather, store, and analyze Java Flight Recorder (JFR) data about target applications that run on Java Virtual Machine (JVM)s, you must configure the applications so that Cryostat can detect and connect to them.

You can configure the applications in any of the following ways:

  • By using the Cryostat agent component for detection and connectivity, which is implemented as a Java Instrumentation Agent and acts as a plug-in for applications that run on the JVM
  • By configuring the applications to allow Java Management Extensions (JMX) connections and using an OpenShift Service for detection and JMX for connectivity
  • By using the Cryostat agent for detection and JMX for connectivity

Cryostat agent

The Cryostat agent provides a read-only HTTP API for Cryostat functions and removes the requirement for target applications to expose a JMX port.

The agent performs the following actions:

  • Fetches data from the JVM and sends it back to Cryostat over HTTP by looking for data within itself and the application to which it is plugged in
  • Locates Cryostat applications that run on the network and initiates a connection to those applications
  • Communicates back to Cryostat about itself and provides information on how to reach it
  • Pushes its own JFR data to Cryostat by initiating a network connection with Cryostat, which then analyzes and saves the data

The agent supports a limited set of JFR operations and you must configure JFR recordings in advance before the attached application starts. You can configure the JFR data you want the agent to collect, a schedule for pushing that data to Cryostat, and the Cryostat server URL to which you want to push.

You can use the command line to install the agent on your target JVM. You must add the agent to your application builds, then rebuild and deploy the application.

After the agent is installed to the running JVM instance, the agent starts collecting data and sending it to Cryostat for analysis.

Remote Java Management Extensions (JMX) connections

JMX is a standard feature on a JVM with which you can monitor and manage target applications that run on the JVM. For Cryostat to use JMX, you must enable and configure JMX when you start the JVM because Cryostat requires the target applications to expose a JMX port.

Cryostat communicates with the target applications over this JMX port to start and stop JFR recordings and to pull JFR data over the network, enabling Cryostat to store and analyze this JFR data. Remote monitoring requires security to ensure that unauthorized persons cannot access application. Cryostat prompts you to enter your credentials before Cryostat can access any of the application’s JFR recordings.

Cryostat agent and JMX hybrid

You can configure your target applications to use a hybrid approach where you use both the Cryostat agent and JMX. With this approach, you use the Cryostat agent to detect the target applications and JMX to expose the JFR data to Cryostat, which allows for more flexibility.

For example, you can use the agent to detect the applications without needing to depend on specific port numbers and also use the JMX connections to start and stop JFR flight recordings on demand.

3.1. Configuring applications by using the Cryostat agent

You can use the Cryostat agent, implemented as a Java Instrumentation Agent, to configure target applications so that Cryostat can detect the applications, collect data, and send the data to Cryostat for analysis.

Prerequisites

  • Logged in to your Cryostat web console.
  • Installed JDK version 11 or later.

Procedure

  1. Install the Cryostat agent. Choose one of the following options, depending on your application build:

    • Using Maven:

      Update the application pom.xml file with the Cryostat agent JAR file information.

      Example pom.xml

      <project>
        ...
        <repositories>
          <repository>
            <id>redhat-maven-repository</id>
            <url>https://maven.repository.redhat.com/earlyaccess/all/</url>
          </repository>
        </repositories>
        ...
        <build>
          <plugins>
            <plugin>
              <artifactId>maven-dependency-plugin</artifactId>
              <version>3.3.0</version>
              <executions>
                <execution>
                  <phase>prepare-package</phase>
                  <goals>
                    <goal>copy</goal>
                  </goals>
                  <configuration>
                    <artifactItems>
                      <artifactItem>
                        <groupId>io.cryostat</groupId>
                        <artifactId>cryostat-agent</artifactId>
                        <version>0.2.2.redhat-00001</version>
                      </artifactItem>
                    </artifactItems>
                    <stripVersion>true</stripVersion>
                  </configuration>
                </execution>
              </executions>
            </plugin>
          </plugins>
          ...
        </build>
        ...
      </project>

      The next time you build your application, the Cryostat agent JAR file is available at target/dependency/cryostat-agent.jar.

    • Using Gradle:

      Update the build.gradle file.

      Example build.gradle file

      repositories {
      	…
      maven {
          	url "https://maven.repository.redhat.com/earlyaccess/all/"
          	credentials {
            		username "myusername"
              	password "mytoken"
          	}
      	}
      }

      How you package the agent JAR file into the application depends on the Gradle plug-ins that you use for the build. For example, if you are using the Jib plug-in, update the build.gradle file as follows:

      Example build.gradle file

      plugins {
      	id 'java'
      	id 'application'
      	id 'com.google.cloud.tools.jib' version '3.3.1'
      	id 'com.ryandens.javaagent-jib' version '0.5.0'
      }
      …
      dependencies {
      	…
      	javaagent 'io.cryostat:cryostat-agent:0.2.2.redhat-00001'

  2. Update the Docker file. The following example uses the JAVA_OPTS environment variable to pass the relevant JVM information.

    Example

    ...
    COPY target/dependency/cryostat-agent.jar /deployments/app/
    ...
    ENV JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent.jar"

  3. Rebuild the container image that is specific to your application.

    docker build -t docker.io/myorg/myapp:latest -f src/main/docker/Dockerfile
  4. To supply the JVM system properties or environment variables that you need to configure the Cryostat agent, push the updated image and then modify your application deployment.

    Example

    apiVersion: apps/v1
    kind: Deployment
    ...
    spec:
      ...
      template:
        ...
        spec:
          containers:
            - name: sample-app
              image: docker.io/myorg/myapp:latest
              env:
                - name: CRYOSTAT_AGENT_APP_NAME
                  value: "myapp"
                  # Replace this with the Kubernetes DNS record
                  # for the Cryostat Service
                - name: CRYOSTAT_AGENT_BASEURI
                  value: "http://cryostat.mynamespace.mycluster.svc:8181"
                - name: POD_IP
                  valueFrom:
                    fieldRef:
                      fieldPath: status.PodIP
                - name: CRYOSTAT_AGENT_CALLBACK
                  value: "http://$(POD_IP):9977" 1
                  # Replace "abcd1234" with a base64-encoded authentication token
                - name: CRYOSTAT_AGENT_AUTHORIZATION 2
                  value: "Bearer abcd1234"
              ports:
                - containerPort: 9977
                  protocol: TCP
              resources: {}
          restartPolicy: Always
    status: {}

    • <1>: Port number 9977 is the default HTTP port that the agent exposes for the internal web server that services Cryostat requests. You can change this port number if it conflicts with your target application into which the agent is installed.
    • <2>: The CRYOSTAT_AGENT_AUTHORIZATION value shows the credentials that the agent includes in the API requests to Cryostat to advertise its own presence or to push JFR data. You can also create a Kubernetes Service Account for this purpose and replace abcd1234 with the base64-encoded authentication token associated with the service account.

3.2. Configuring applications by using JMX connections

For Cryostat to detect and communicate with your target Java applications, you can configure the applications to allow remote Java Management Extensions (JMX) connections.

Prerequisites

  • Logged in to your Cryostat web console.
  • Created a Cryostat instance in your project.

Procedure

  1. To enable remote JMX connections, complete the following steps:

    1. On your application, define the following Java system property:

      -Dcom.sun.management.jmxremote.port=<port_num>
      Note

      To add the -Dcom.sun.management.jmxremote.port=<port_num> property without having to rebuild the target application, you can set the JAVA_OPTS_APPEND environment variable on the application. JAVA_OPTS_APPEND is an environment variable that is used by Red Hat Universal Base Images (UBI) only.

      If you use Red Hat UBI to build application images, set the JAVA_OPTS_APPEND variable at build time in the application Docker file, or at runtime by running the following command:

      oc set env deployment <name> JAVA_OPTS_APPEND="..."

      If you do not use Red Hat UBI to build application images, refer to the documentation for your base image for information about how to add the Java system properties at build time or runtime.

    2. Specify that the application listens for remote JMX connections by allowing traffic to the application. Use an Red Hat OpenShift Service and specify the following values for its remote JMX port:

      Example service.yaml

      apiVersion: v1
      kind: Service
      ...
      spec:
        ports:
          - name: "jfr-jmx"
            port: 9091
            targetPort: 9091
      ...

  2. Secure the remote JMX connections:

    1. Enable and configure authentication and SSL/TLS for the remote JMX connections in your application:

      -Dcom.sun.management.jmxremote.port=<port_num>
      
       # enable JMX authentication
      -Dcom.sun.management.jmxremote.authenticate=true
      
      # define users for JMX auth
      -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password>
      
      # set permissions for JMX users
      -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access>
      
       # enable JMX SSL
       -Dcom.sun.management.jmxremote.ssl=true
      
      # enable JMX registry SSL
      -Dcom.sun.management.jmxremote.registry.ssl=true
      
      # set your SSL keystore
      -Djavax.net.ssl.keyStore=</path/to/keystore>
      
      # set your SSL keystore password
      -Djavax.net.ssl.keyStorePassword=<password>
    2. Configure Cryostat to trust the application TLS certificate. Create a secret for the application in the same namespace as your Cryostat application and configure Cryostat to refer to the secret. To create a secret for the certificate, run the following command:

      oc create secret generic myapp-cert --from-file=tls.crt=/path/to/cert.pem
      Note

      The certificate must be in a .pem file format.

    3. When you create your Cryostat instance, add the secret to the list of trusted TLS certificates. For more information, see Configuring TLS certificates.
    4. To allow your applications to verify that Cryostat is connecting to them by a means other than password authentication, enable TLS client authentication:

      -Dcom.sun.management.jmxremote.ssl.need.client.auth=true
      -Djavax.net.ssl.trustStore=</path/to/truststore>
      -Djavax.net.ssl.trustStorePassword=<password>
      Note

      TLS client authentication requires the cert-manager operator for Red Hat OpenShift.

    5. If you use TLS client authentication for remote JMX connections, the application truststore must contain a Cryostat certificate. The Cryostat operator cert-manager integration creates a self-signed certificate for the Cryostat deployment. This certificate is located in the <cryostat>-tls secret, where <cryostat> is the name of the Cryostat instance you created.

      Note

      The cert-manager Operator also places a Java keystore truststore in the secret.

      To mount this truststore in your application deployment, run the following command, replacing "<myapp>" with the name of your application deployment and "<cryostat>" with the name of your Cryostat instance:

      oc set volumes deploy <myapp> --add --name=truststore \
        --secret-name=<cryostat>-tls --sub-path=truststore.p12 \
        --mount-path=/var/run/secrets/<myapp>/truststore.p12
    6. The Cryostat operator generates the truststore password, which you can find in the <cryostat>-keystore secret. To mount this as an environment variable in your application deployment, run the following command:

      oc set env deploy <myapp> --from='secret/<cryostat>-keystore'
    7. Configure the Java arguments for the container. Run the following commands:

      -Dcom.sun.management.jmxremote.ssl.need.client.auth=true
      -Djavax.net.ssl.trustStore=/var/run/secrets/<myapp>/truststore.p12
      -Djavax.net.ssl.trustStorePassword="$(KEYSTORE_PASS)"
      Warning

      If you deployed Cryostat and your applications in a testing environment, you might want to configure the target applications without any JMX or TLS authentication. You can do so by using the following set of Java system properties, however, this configuration is not secure and not recommended.

      -Dcom.sun.management.jmxremote.port=<port_num>
      -Dcom.sun.management.jmxremote.ssl=false
      -Dcom.sun.management.jmxremote.authenticate=false

3.3. Configuring applications by using the Cryostat agent and JMX connections

You can configure target applications that run on Java Virtual Machines (JVM) to use a combination of the Cryostat agent and Java Management Extensions (JMX) connections to detect and communicate with the target applications.

You use the Cryostat agent to detect and communicate with the target application and use JMX to expose the Java Flight Recorder (JFR) data.

You must configure the Cryostat agent to communicate with Cryostat about itself and that the agent is reachable through JMX rather than through HTTP.

Prerequisites

  • Logged in to your Cryostat web console.
  • Created a Cryostat instance in your project.

Procedure

  1. Install the Cryostat agent. For application builds that use Maven, update the application pom.xml file with the Cryostat agent JAR file information.

    Example pom.xml file

    <project>
      ...
      <repositories>
        <repository>
          <id>redhat-maven-repository</id>
          <url>https://maven.repository.redhat.com/earlyaccess/all/</url>
        </repository>
      </repositories>
      ...
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>3.3.0</version>
            <executions>
              <execution>
                <phase>prepare-package</phase>
                <goals>
                  <goal>copy</goal>
                </goals>
                <configuration>
                  <artifactItems>
                    <artifactItem>
                      <groupId>io.cryostat</groupId>
                      <artifactId>cryostat-agent</artifactId>
                      <version>0.2.2.redhat-00001</version>
                    </artifactItem>
                  </artifactItems>
                  <stripVersion>true</stripVersion>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
        ...
      </build>
      ...
    </project>

  2. Modify your application deployment:

    Example

    apiVersion: apps/v1
    kind: Deployment
    ...
    spec:
      ...
      template:
        ...
        spec:
          containers:
            - name: sample-app
              image: docker.io/myorg/myapp:latest
              env:
                - name: CRYOSTAT_AGENT_APP_NAME
                  value: "myapp"
                  # Replace this with the Kubernetes DNS record
                  # for the Cryostat Service
                - name: CRYOSTAT_AGENT_BASEURI
                  value: "http://cryostat.mynamespace.mycluster.svc:8181"
                - name: POD_IP
                  valueFrom:
                    fieldRef:
                      fieldPath: status.PodIP
                - name: CRYOSTAT_AGENT_CALLBACK
                  value: "http://$(POD_IP):9977"
                - name: CRYOSTAT_AGENT_AUTHORIZATION
                  # Replace "abcd1234" with a base64-encoded authentication token
                  value: "Bearer abcd1234"
                  # This environment variable is key to the Cryostat agent
                  # and JMX "hybrid" setup.
                  # Set the Cryostat agent to register itself with Cryostat
                  # as reachable through JMX, rather than reachable through HTTP.
                - name: CRYOSTAT_AGENT_REGISTRATION_PREFER_JMX
                  value: "true"
                  # Configure the application to load the agent JAR file and
                  # to enable JMX, so that the Cryostat agent can register
                  # itself as reachable through JMX.
                  # To configure authentication and SSL/TLS for the JMX
                  # connections, see <1>.
                - name: JAVA_OPTS
                  value: >-
                    -javaagent:/deployments/app/cryostat-agent.jar
                    -Dcom.sun.management.jmxremote.port=9091 1
              ports:
                - containerPort: 9977
                  protocol: TCP
              resources: {}
          restartPolicy: Always
    status: {}

    <1>: To configure authentication and SSL/TLS for the JMX connections and view further configuration options, see Configuring applications by using JMX connections.

  3. To enable Cryostat to detect the target application and connect to the Cryostat agent, configure an application Service:

    Example

    apiVersion: v1
    kind: Service
    ...
    spec:
      ports:
        - name: "jfr-jmx"
          port: 9091
          targetPort: 9091
        - name: "cryostat-agent"
          port: 9977
          targetPort: 9977
    ...