Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Red Hat supports a major version of Cryostat for a minimum of 6 months. Red Hat bases this figure on the time that the product gets released on the Red Hat Customer Portal.

You can install and deploy Cryostat on Red Hat OpenShift Container Platform 4.8 or a later version that runs on an x86_64 architecture.

Cryostat 2.0 introduces new features that enhance your use of the Cryostat product.

API permissions for the Cryostat application

The Cryostat Operator creates a service account and then configures the Cryostat application to use the service account. The Cryostat Operator checks the service account to ensure its permissions remain in sync with those expected by the authorization check.

In Cryostat 1, the Cryostat Operator applied its own service account’s permissions to a Cryostat application. This provided excessive permissions to the Cryostat application.

Automated rules

In Cryostat 2.0, you can use automated rules to enable JFR to continuously monitor a running target application. You do not need to restart or redeploy the application.

You could continue to enable always-on continuous monitoring for an application by setting JVM flags, but this method requires an application restart.

Cryostat 2.0 includes continuous monitoring event templates that you can use to create automated rules and templates. By using continuous monitoring event templates, you can reduce any downtime for specifying a JFR to continuously monitoring an application.

You must specify of the following automated rule definitions for an automated rule:

After you upload an automated rule to Cryostat in JSON format, Cryostat validates the definition file.

You can immediately activate a rule definition for any previous defined matching targets. You do not need to re-add or restart targets after you create a rule definition.

You can delete a rule definition by adding the HTTP query clean=true parameter to your DELETE requests for rule definitions. This parameter directs Cryostat to delete only active recordings. Archived recordings do not get targeted for deletion.

cert-manager API

You can now use the Enable cert-manager integation toggle button on the Create Cryostat page of your Cryostat Operator to enable or disable end-to-end encryption by using cert-manager for your Cryostat application.

Figure 3.1. The Enable cert-manager integation toggle button

Additionally, the Cryostat Operator now automatically checks that the cert-manager API exists before it deploys Cryostat. This ensures that the Cryostat Operator securely deploys Cryostat. Depending on your needs, you can explicitly set the Cryostat Operator cert-manager API check to disabled.

If Cryostat Operator detects a Cryostat instance that requires cert-manager, but you did not install cert-manager, then you’ll receive a warning event for the Cryostat instance on the Events page of the OpenShift web console.

Deployment scenario tree view

After you deploy your application on OpenShift or any supported Kubernetes distribution, you can display your Cryostat’s information in a JSON-formatted tree view. You can view the following components on the tree view:

Endpoint resource watch

The OpenShiftPlatformClient automatically enables the endpoint resource watch to set an asynchronous target discovery for your target JVMs.

Event search with TargetEventGetHandler

You can use the TargetEventGetHandler to list or search for event types produced by your target JVM.

HTTP API behavior change

Prior to Cryostat 2.0, HTTP request handlers might override the default isAsync method setting of true when determining whether to install a handler for the Vertx server.

Each HTTP handler contains code that defines an HTTP API endpoint for Cryostat. The isAsync method defines whether an HTTP handler gets handled by either a separate worker thread or a main webserver thread.

New environment variables

Cryostat 2.0 includes the following two new environment variables:

New handlers

Cryostat 2.0 introduces the following V2 handlers:

New event template available on Cryostat

You can now select the cryostat.jfc event template from the list of available Target JVM templates on the Cryostat web console.

By selecting the cryostat.jfc event template, you automatically enable the following Cryostat settings:

Pre-configured Grafana dashboard image

Cryostat 2.0 includes a pre-configured Grafana dashboard image for viewing your application’s metrics. You can also create your own Grafana dashboard image to meet your needs.

Cryostat 1 provided a Grafana image that offered basic customization by the OpenShift Operator during deployment of a Cryostat container.

Response status code message

Response status code messages for a ClientUrlGetHandler error message now display in the front of the error message. This is the default setting. The custom exception message remains in the response body of the error message.

Support for client-defined connection protocols

Cryostat 2 now adds a mountpoint for a clientlib directory, to which you can add additional JAR files to be loaded into the Cryostat classpath. This supports client-defined connection protocols as alternatives to JMX RMI.

Cryostat 2.0 includes feature enhancements that build upon the Cryostat 1 offerings.

Cryostat web console GUI

You can access to the Cryostat information page through the Help icon in the upper-right corner of the Cryostat web console. On this console page, you can view your version of Cryostat.

Additionally, an archived record’s generated name now includes a target alias. The target alias improves record retrieval by linking an alias to the record’s generated name.

Cryostat Operator topology view

The Cryostat Operator now applies the correct app.openshift.io/connects-to annotation to a Cryostat deployment. This configuration links the Cryostat deployment with the Cryostat Operator deployment in the topology view on the OpenShift Container Platform web console.

Cryostat Operator controller manager

If the Cryostat Operator failed to deploy Cryostat because you did not install cert-manager, the Cryostat Operator no longer hangs when it deletes Cryostat custom resources.

Cryostat cluster ConsoleLink namespace

In Cryostat 2.0, a Cryostat Operator cluster might create a duplicate ConsoleLink namespace for a Cryostat cluster. To avoid this issue, Cryostat Operator now replaces the GenerateName object with a SH256 hash value defined in Name object of the ConsoleLink resource definition.

Custom event templates

You can configure any Cryostat JFR recordings by using custom event templates. You can use custom event templates in the following ways:

Encode SSL/TLS certificates with supported formats

You can add an SSL/TLS certificate on the Cryostat web console for your target JVM application.

Valid SSL/TLS certificates are in DER-encoded base-64 or binary formats. Either format supports the following extensions:

You can now specify a generated certificate at runtime, so that your target JVM application can use the SSL/TLS for JMX connections. Cryostat attempts to open a JMX connection to a target JVM that uses an SSL/TLS certificate. For a successful JMX connection, Cryostat must pass all its authentication checks on the target JVM certificate that you provided at runtime .

You can use the POST handler to accept, validate and write the certificate.

Fabric8 Kubernetes and OpenShift clients

Cryostat 2.0 supports version 4.12.0 of Fabric8 Kubernetes and of OpenShift clients. This version enhances application compatibility with Cryostat and reduces downstream build errors.

Grafana container version

Cryostat 2.0 replaces version 6.4.4 of the Grafana container with version 7.3.6.

Grafana error messages

The 502 error message now relates to a failed JMX connection, while the 512 error messages relates to invalid responses sent to the JFR container.

Health check resource definition

Cryostat Operator replaces the api/v1/clienturl resource definition with the health resource definition. The Cryostat Operator now uses the health endpoint when performing containerized JVM health checks.

initialization-resource annotation

Cryostat Operator now includes an initialization-resource annotation in its CSV file’s configuration. This annotation enhances the Cryostat Operator instance running on the OpenShift Container Platform web console by providing you with graphical hints to create a Cryostat custom resource for your Cryostat cluster.

OLM bundle descriptors

For Cryostat 2.0, an OLM bundle no longer requires setting an integer value in its eventOptions descriptor for checking a JFR recording’s duration. Instead, the OLM bundle now uses requires duration formatted values that are defined in its EventOptions string.

Supported duration units include s, m, and h. For example, 2h15m denotes a JFR recording length of 2 hours and 15 minutes.

Security context constraint (SCC)

The Cryostat Operator now defaults to using the restricted SCC setting. A pod contained in a Cryostat cluster can now use any permitted fsGroup value listed under the restricted SCC setting. Meaning that Cryostat pods mounted to a persistent storage (PV) can now have read/write access to their directories.

A pod’s read/write access level depends on the pod’s fsGroup GID value, which the Cryostat Operator configures to adhere to the built-in restricted SecurityContextConstraint. A SecurityContext element contains pod-level security attributes.

Before Cryostat 2.0, the Cryostat Operator was set to the default setting, which caused fsGroup access issues for a pod running in a Cryostat cluster

For more information about the permitted range of fsGroup values for your Cryostat cluster, see About pre-allocated security context constraints values in the OpenShift documentation website.

ServiceRef definition

A ServiceRef definition includes the following new properties that Cryostat includes in all ServiceRef objects returned from GET communications with HTTP API handlers:

You can use the following two handlers create or delete a ServiceRef definition:

Subprocess management

Before Cryostat generates an automated analysis report, Cryostat creates a child subprocess. A child subprocess protects the parent process by accumulating any large memory loads consumed by the report generation.

For example, on a Linux operating system the out-of-memory (OOM) Killer detects a process that tries to request additional memory that is not available from the system and stops the process. If a child subprocess exists, the OOM Killer stops this subprocess but does not interfere with the running parent process.

You can set a minimum or maximum JVM heap size for a child subprocess by using the Cryostat environment variable: CRYOSTAT_REPORT_GENERATION_MAX_HEAP. Be aware that a low minimum value might stop a child subprocess before it generates a report, while a high maxumim value might cause additional memory constraints on the parent process.

Subprocess report generation

Cryostat 2.0 provides an enhancement to using either ActiveRecordingReportCache or ArchivedRecordingReportCache to generate a subprocessed report for your Java application. These classes are simplified as follows:

A subprocess no longer needs to complete the following tasks:

The removal of these tasks improves a subprocess’s workload in the following ways:

truststore environment variable

Cryostat 2.0 replaces TRUSTSTORE environment variable name with SSL_TRUSTSTORE_DIR.

TargetConnectionManager handler

The TargetConnectionManager now support concurrent connections by using a JMX connection timed cache value for any connections between a targeted request and Cryostat. By default, the timed cache is 90 seconds.

Vertx server response management

The Vertx server can now use a TimeoutHandler implementation to automatically end a delayed response message sent by a Cryostat request handler. This prevents a client from having to wait indefinitely for a response from a Cryostat request handler.

The TimeoutHandler implementation throws a 500 error message when it detects a delayed response from a Cryostat request handler.

WebSockets

The WebSocket includes the following updates:

When your WebSocket client connects to a one-way push NC, the client automatically receives information about actions performed by other connected clients on the same channel.

Cryostat 2.0 removes some features because of their high maintenance costs, low community interest, and better alternative solutions.

ContainerJFR variants

The Cryostat namespace replaces the ContainerJFR namespace, so Cryostat 2.0 no longer specifies any variants of the ContainerJFR namespace.

Event specifier string syntax

Cryostat 2.0 does not include any event specific string syntax. Event templates offer a better alternative to event specific string syntax.

Interactive mode commands

The HTTP API and Notification Channels replace the Command Channel. You cannot use any Command Channel implementations or their CommandExecutor interfaces in Cryostat 2.0.

TTY/TCP execution modes

Cryostat 2.0 does not contain any TTY and TCP socket execution modes or any mode’s configurations. These execution modes did not support platform authentication or target authentication.

You can continue to use the supported WebSocket execution mode.

WebSocket commands

Cryostat 2.0 removed all WebSocket commands. These commands were deprecated in the Cryostat 1 release.

WebSever IO cleanup

Cryostat 2.0 removed the low-memory pressure mode for embedded WebSever IOs. You can no longer use this mode to apply a constant memory size mapping to an embedded WebServer IO.

Sometimes a Cryostat release might contain an issue or issues that Red Hat acknowledges and might fix at a later stage during the product’s development. Review each known issue for its description and its resolution.

OpenShift SDN default CNI network provider

Cryostat 2.0 cannot connect an OpenShift cluster that has the following configurations for JVMs located on nodes different to those running on the Cryostat node:

You can resolve this known issue by configuring your cluster to use the Open Virtual Network (OVN) method instead of the SDN method. The OVN method contains the following similar configurations to that of the SDN method:

Archive upload API Out-Of-Memory (OOM)

Cryostat 2.0 consumes more memory than expected when a client sends a request to the Cryostat HTTP POST /api/v1/recordings handler. This handler points to the /opt/cryostat.d/recordings.d directory on your Cryostat 2.0 instance and you can use the handler to upload .jfr binary files to this directory.

The Cryostat Operator sets a default memory limit of 512 MB for a Cryostat instance deployed in an OpenShift project. If you upload a 150 MB or larger .jfr file to your Cryostat 2.0 instance, the OpenShift cluster’s Out of Memory (OOM) killer stops the pod that contains your deployed Cryostat instance.

You can resolve this known issue by copying your .jfr binary file to the persistent storage location on your Cryostat 2.0 instance. By using this method, you do not need to send a client request to the Cryostat HTTP POST /api/v1/recordings handler to store the .jfr binary in the /opt/cryostat.d/recordings.d directory.

You can issue the following commands to copy your .jfr binary file to the persistent storage location on your Cryostat 2.0 instance:

oc exec -i -n <your_namespace> -c <cryostat_container_name> <cryostat_pod_name> – mkdir /opt/cryostat.d/recordings.d/unlabelled

oc cp vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006t152006z.jfr <cryostat_pod_name>:/opt/cryostat.d/recordings.d/unlabelled/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006t152006z.jfr -c <cryostat_container_name>

After you copy the .jfr binary file directly into the PVC archives location, you can use a curl, an httpie, or a wget command to verify that the .jfr file exists on your Cryostat 2.0 instance.

The following example demonstrates using a curl command to verify that Cryostat recognizes the uploaded file that was copied to the persistent storage location with the oc cp command. The <cryostat_url> value in the example resolves to https://cryostat-sample-myproject.apps-crc.testing:443, but you can replace the <cryostat_url> value with your application’s URL. You can determine your application’s URL by issuing the oc status command.

$ curl -kv -H Authorization:"Bearer $(oc whoami -t)" <cryostat_url>/api/v1/recordings

File upload limit for the integrated Grafana component

For Cryostat 2.0, the integrated View in Grafana​ component cannot accept JFR files larger than 10 MB, because of a configuration issue with the jfr-datasource container.

A deployed Cryostat 2.0 pod’s jfr-datasource container uses default Quarkus settings, which includes the default quarkus.http.limits.max-body-size parameter. This parameter sets the maximum size limit for a file on Quarkus, and the parameter has a default value of 10 MB.

If a client attempts to upload a JFR file larger than 10 MB, the jfr-datasource web server rejects the file and throws an HTTP 413 error message.

You can resolve this known issue by completing the following steps:

Cryostat 2.0 includes security fixes that improve upon security offered from the previous release of Cryostat. Review each security fix for its description.

jsoup Java library

The jsoup Java library that is pre-installed with Cryostat 2.0 now upgraded from version 1.13.1 to version 1.14.2. This new version improves upon the jsoup Java library’s handling of HTML and XML syntax.

Revised on 2022-11-25 08:58:08 UTC