Release notes for Cryostat 2.0
Cryostat is a container-native Java application that connects your Java workloads running inside an OpenShift cluster to your desktop JDK Mission Control application.
The primary feature for Cryostat 2.0 is that you can install and deploy Cryostat through the OpenShift Operator in the OperatorHub of the OpenShift Container Platform web console. After you install and deploy Cryostat, you can access a fully featured Cryostat instance and then explore any of the additional features listed in the Release notes for Cryostat 2.0 guide.
The features listed in the this guide are Technology Preview features. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete for production purposes. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
See Technology Preview Features Support Scope on the Red Hat Customer Portal for information about the support scope for Technology Preview features.
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your feedback on our documentation. To provide feedback, you can highlight the text in a document and add comments.
This section explains how to submit feedback.
- You are logged in to the Red Hat Customer Portal.
- In the Red Hat Customer Portal, view the document in Multi-page HTML format.
To provide your feedback, perform the following steps:
Click the Feedback button in the top-right corner of the document to see existing feedback.Note
The feedback feature is enabled only in the Multi-page HTML format.
- Highlight the section of the document where you want to provide feedback.
Click the Add Feedback pop-up that appears near the highlighted text.
A text box appears in the feedback section on the right side of the page.
Enter your feedback in the text box and click Submit.
A documentation issue is created.
- To view the issue, click the issue tracker link in the feedback view.
Chapter 1. Support policy for Cryostat 2.0
Red Hat supports a major version of Cryostat for a minimum of 6 months. Red Hat bases this figure on the time that the product gets released on the Red Hat Customer Portal.
You can install and deploy Cryostat on Red Hat OpenShift Container Platform 4.6 or later that is running on an x86_64 architecture.
- For more information about the Cryostat life cycle policy, see Cryostat on the Red Hat OpenShift Container Platform Life Cycle Policy web page.
Chapter 2. New features
Cryostat 2.0 introduces new features that enhance your use of the Cryostat product.
API permissions for the Cryostat application
The Cryostat Operator creates a service account and then configures the Cryostat application to use the service account. The Cryostat Operator checks the service account to ensure its permissions remain in sync with those expected by the authorization check.
In Cryostat 1, the Cryostat Operator applied its own service account’s permissions to a Cryostat application. This provided excessive permissions to the Cryostat application.
In Cryostat 2.0, you can use automated rules to enable JFR to continuously monitor a running target application. You do not need to restart or redeploy the application.
You could continue to enable always-on continuous monitoring for an application by setting JVM flags, but this method requires an application restart.
Cryostat 2.0 includes continuous monitoring event templates that you can use to create automated rules and templates. By using continuous monitoring event templates, you can reduce any downtime for specifying a JFR to continuously monitoring an application.
You must specify of the following automated rule definitions for an automated rule:
One-shot rule definition: You can now create a simple configuration of one-shot automated rules for your JFR configuration. The
eventSpecifierproperty contains an
archivesimple string value.
matchExpressiondefinition: Mandatory component of an automated rule that defines JVM targets that must receive an automated rule. Cryostat uses to apply rules to any JVM targets that match regular expressions defined in the
matchExpressionstring expression. Valid string expressions include separator, operator, and simple expressions.
You can apply OpenShift labels or annotations as criteria for a
- You can apply OpenShift labels or annotations as criteria for a
After you upload an automated rule to Cryostat in JSON format, Cryostat validates the definition file.
You can immediately activate a rule definition for any previous defined matching targets. You do not need to re-add or restart targets after you create a rule definition.
You can delete a rule definition by adding the HTTP query
clean=true parameter to your
DELETE requests for rule definitions. This parameter directs Cryostat to delete only active recordings. Archived recordings do not get targeted for deletion.
You can now use the Enable cert-manager integation toggle button on the Create Cryostat page of your Cryostat Operator to enable or disable end-to-end encryption by using
cert-manager for your Cryostat application.
Figure 2.1. The Enable cert-manager integation toggle button
Additionally, the Cryostat Operator now automatically checks that the
cert-manager API exists before it deploys Cryostat. This ensures that the Cryostat Operator securely deploys Cryostat. Depending on your needs, you can explicitly set the Cryostat Operator
cert-manager API check to disabled.
If Cryostat Operator detects a Cryostat instance that requires
cert-manager, but you did not install
cert-manager, then you’ll receive a warning event for the Cryostat instance on the Events page of the OpenShift web console.
Deployment scenario tree view
After you deploy your application on OpenShift or any supported Kubernetes distribution, you can display your Cryostat’s information in a JSON-formatted tree view. You can view the following components on the tree view:
Endpoint resource watch
OpenShiftPlatformClient automatically enables the endpoint resource watch to set an asynchronous target discovery for your target JVMs.
Event search with
You can use the
TargetEventGetHandler to list or search for event types produced by your target JVM.
HTTP API behavior change
Prior to Cryostat 2.0, HTTP request handlers might override the default
isAsync method setting of
true when determining whether to install a handler for the Vertx server.
Each HTTP handler contains code that defines an HTTP API endpoint for Cryostat. The
isAsync method defines whether an HTTP handler gets handled by either a separate worker thread or a main webserver thread.
New environment variables
Cryostat 2.0 includes the following two new environment variables:
CRYOSTAT_TARGET_CACHE_MAX_CONNECTIONS, which the OpenShift Operator sets to a value for specifying the maximum amount of open JMX connections.
CRYOSTAT_TARGET_CACHE_TTL, which the OpenShift Operator sets to a value for specifying the amount of seconds for caching a JMX connection before they close from inactivity.
The OpenShift Operator automatically determines values for these environment variables, so you cannot modify the values specified by the OpenShift Operator for these environment variables.
Cryostat 2.0 introduces the following V2 handlers:
AbstractV2RequestHandlerthat behaves similar to
AbstractAuthenticatedRequestHandler, but maintaining the overall response format among V2 handlers.
TargetRecordingOptionsListGetHandlerthat checks recording options supported by your target JVM. Use this handler before you create a recording, so you do not attempt to use recording options not supported by the target JVM.
TargetSnapshotPostHandlerthat encodes a target response in JSON format for a customized snapshot recording. This V2 snapshot handler also includes the name of the recording, the URL for the downloadable recording file, the URL for the automated analysis report, and much more.
A V1 handler can perform snapshot operations similar to a V2 handler, but a V1 handler’s HTTP response renders only in plain string format and only contains the name of the snapshot recording.
New event template available on Cryostat
You can now select the
cryostat.jfc event template from the list of available Target JVM templates on the Cryostat web console.
By selecting the
cryostat.jfc event template, you automatically enable the following Cryostat settings:
- HTTP request information, such as status code, duration, and so on.
- WebSocket message information, such as duration, message type, and so on.
- JMX connection status, such as opened, closed, or failed.
Pre-configured Grafana dashboard image
Cryostat 2.0 includes a pre-configured Grafana dashboard image for viewing your application’s metrics. You can also create your own Grafana dashboard image to meet your needs.
Cryostat 1 provided a Grafana image that offered basic customization by the OpenShift Operator during deployment of a Cryostat container.
Response status code message
Response status code messages for a
ClientUrlGetHandler error message now display in the front of the error message. This is the default setting. The custom exception message remains in the response body of the error message.
Support for client-defined connection protocols
Cryostat 2 now adds a mountpoint for a
clientlib directory, to which you can add additional JAR files to be loaded into the Cryostat classpath. This supports client-defined connection protocols as alternatives to JMX RMI.
Chapter 3. Feature enhancements
Cryostat 2.0 includes feature enhancements that build upon the Cryostat 1 offerings.
Cryostat web console GUI
You can access to the Cryostat information page through the Help icon in the upper-right corner of the Cryostat web console. On this console page, you can view your version of Cryostat.
Additionally, an archived record’s generated name now includes a target alias. The target alias improves record retrieval by linking an alias to the record’s generated name.
Cryostat Operator topology view
The Cryostat Operator now applies the correct
app.openshift.io/connects-to annotation to a Cryostat deployment. This configuration links the Cryostat deployment with the Cryostat Operator deployment in the topology view on the OpenShift Container Platform web console.
Cryostat Operator controller manager
If the Cryostat Operator failed to deploy Cryostat because you did not install
cert-manager, the Cryostat Operator no longer hangs when it deletes Cryostat custom resources.
In Cryostat 1, a Cryostat Operator cluster might create a duplicate
ConsoleLink namespace for a Cryostat cluster. To avoid this issue, Cryostat Operator now replaces the
GenerateName object with a
SH256 hash value defined in
Name object of the
ConsoleLink resource definition.
Custom event templates
You can configure any Cryostat JFR recordings by using custom event templates. You can use custom event templates in the following ways:
Instruct the Cryostat Operator to pre-configure Cryostat custom event templates by providing them with template files from stored
- Upload custom event templates by using Cryostat web applications.
Encode SSL/TLS certificates with supported formats
You can add an SSL/TLS certificate on the Cryostat web interface for your target JVM application.
Valid SSL/TLS certificates are in DER-encoded
binary formats. Either format supports the following extensions:
You can now specify a generated certificate at runtime, so that your target JVM application can use the SSL/TLS for JMX connections. Cryostat attempts to open a JMX connection to a target JVM that uses an SSL/TLS certificate. For a successful JMX connection, Cryostat must pass all its authentication checks on the target JVM certificate that you provided at runtime .
You can use the
POST handler to accept, validate and write the certificate.
Fabric8 Kubernetes and OpenShift clients
Cryostat 2.0 supports version 4.12.0 of Fabric8 Kubernetes and of OpenShift clients. This version enhances application compatibility with Cryostat and reduces downstream build errors.
Grafana container version
Cryostat 2.0 replaces version 6.4.4 of the Grafana container with version 7.3.6.
Grafana error messages
502 error message now relates to a failed JMX connection, while the
512 error messages relates to invalid responses sent to the JFR container.
Health check resource definition
Cryostat Operator replaces the
api/v1/clienturl resource definition with the
health resource definition. The Cryostat Operator now uses the
health endpoint when performing containerized JVM health checks.
Cryostat Operator now includes an
initialization-resource annotation in its CSV file’s configuration. This annotation enhances the Cryostat Operator instance running on the OpenShift Container Platform web console by providing you with graphical hints to create a Cryostat custom resource for your Cryostat cluster.
OLM bundle descriptors
For Cryostat 2.0, an OLM bundle no longer requires setting an integer value in its
eventOptions descriptor for checking a JFR recording’s duration. Instead, the OLM bundle now uses requires duration formatted values that are defined in its
Supported duration units include
h. For example,
2h15m denotes a JFR recording length of 2 hours and 15 minutes.
Security context constraint (SCC)
The Cryostat Operator now defaults to using the
restricted SCC setting. A pod contained in a Cryostat cluster can now use any permitted
fsGroup value listed under the
restricted SCC setting. Meaning that Cryostat pods mounted to a persistent storage (PV) can now have read/write access to their directories.
A pod’s read/write access level depends on the pod’s
fsGroup GID value, which the Cryostat Operator configures to adhere to the built-in restricted
SecurityContext element contains pod-level security attributes.
Before Cryostat 2.0, the Cryostat Operator was set to the
default setting, which caused
fsGroup access issues for a pod running in a Cryostat cluster
For more information about the permitted range of
fsGroup values for your Cryostat cluster, see About pre-allocated security context constraints values in the OpenShift documentation website.
ServiceRef definition includes the following new properties that Cryostat includes in all
ServiceRef objects returned from
GET communications with HTTP API handlers:
- Annotations, such as the Java application name, labels, or port number.
ServiceRefvalues, such as an alias or
You can use the following two handlers create or delete a
POST, which creates a new
ServiceRefif no existing targets with an identical service URL exist.
DELETE, which removes any
ServiceRefdefinitions from the
CustomTargetPlatformClientif it matches the value specified in the
Before Cryostat generates an automated analysis report, Cryostat creates a child subprocess. A child subprocess protects the parent process by accumulating any large memory loads consumed by the report generation.
For example, on a Linux operating system the out-of-memory (OOM) Killer detects a process that tries to request additional memory that is not available from the system and stops the process. If a child subprocess exists, the OOM Killer stops this subprocess but does not interfere with the running parent process.
You can set a minimum or maximum JVM heap size for a child subprocess by using the Cryostat environment variable:
CRYOSTAT_REPORT_GENERATION_MAX_HEAP. Be aware that a low minimum value might stop a child subprocess before it generates a report, while a high maxumim value might cause additional memory constraints on the parent process.
Subprocess report generation
Cryostat 2.0 provides an enhancement to using either
ArchivedRecordingReportCache to generate a subprocessed report for your Java application. These classes are simplified as follows:
ActiveRecordingReportCachenow automatically includes the path to disk recordings.
ArchivedRecordingReportCachecalls a utility method that uses the connection manager of a parent process to copy a record to a local disk file. The class passes the local path location to the subprocess.
A subprocess no longer needs to complete the following tasks:
- Establish a JMX connection.
- Handle an SSL/TLS certificate.
- Handle JMX authorization credentials.
The removal of these tasks improves a subprocess’s workload in the following ways:
- Quicker end-to-end report generation.
- Reduction in code complexity so that the codebase is easier to maintain.
- Proper and secure handling of SSL/TLS authentication and JMX authentication mechanisms.
truststore environment variable
Cryostat 2.0 replaces
TRUSTSTORE environment variable name with
TargetConnectionManager now support concurrent connections by using a JMX connection timed cache value for any connections between a targeted request and Cryostat. By default, the timed cache is 90 seconds.
Vertx server response management
The Vertx server can now use a
TimeoutHandler implementation to automatically end a delayed response message sent by a Cryostat request handler. This prevents a client from having to wait indefinitely for a response from a Cryostat request handler.
TimeoutHandler implementation throws a
500 error message when it detects a delayed response from a Cryostat request handler.
The WebSocket includes the following updates:
- WebSocket connection upgraded from a two-way interactive channel to a one-way push Notification Channel (NC).
WebSocket Notification Channel changed from
api/v1/notfications, because the WebSocket now uses this channel for one-way push notifications.
- WebSocket can send events on the Notification Channel when you use recordings and event templates for analyzing your JFR data.
When your WebSocket client connects to a one-way push NC, the client automatically receives information about actions performed by other connected clients on the same channel.
Chapter 4. Unsupported and deprecated features
Cryostat 2.0 removes some features because of their high maintenance costs, low community interest, and better alternative solutions.
The Cryostat namespace replaces the
ContainerJFR namespace, so Cryostat 2.0 no longer specifies any variants of the
Event specifier string syntax
Cryostat 2.0 does not include any event specific string syntax. Event templates offer a better alternative to event specific string syntax.
Interactive mode commands
The HTTP API and Notification Channels replace the Command Channel. You cannot use any Command Channel implementations or their
CommandExecutor interfaces in Cryostat 2.0.
TTY/TCP execution modes
Cryostat 2.0 does not contain any TTY and TCP socket execution modes or any mode’s configurations. These execution modes did not support platform authentication or target authentication.
You can continue to use the supported WebSocket execution mode.
Cryostat 2.0 removed all WebSocket commands. These commands were deprecated in the Cryostat 1 release.
WebSever IO cleanup
Cryostat 2.0 removed the low-memory pressure mode for embedded WebSever IOs. You can no longer use this mode to apply a constant memory size mapping to an embedded WebServer IO.
Chapter 5. Known issues
Sometimes a Cryostat release might contain an issue or issues that Red Hat acknowledges and might fix at a later stage during the product’s development. Review each known issue for its description and its resolution.
OpenShift SDN default CNI network provider
Cryostat 2.0 cannot connect an OpenShift cluster that has the following configurations for JVMs located on nodes different to those running on the Cryostat node:
- Use the software-defined networking (SDN) method as the unified cluster network for communication between pods on the cluster.
- Use the default Container Network Interface (CNI) network provider for writing plug-ins to configure network interfaces for containers.
You can resolve this known issue by configuring your cluster to use the Open Virtual Network (OVN) method instead of the SDN method. The OVN method contains the following similar configurations to that of the SDN method:
- Uses the Open vSwitch (OVS) to manage network traffic.
- Uses a plug-in that sets the CNI network provider as default.
- For more information about the SND method, see About the OpenShift SDN default CNI network provider in the Red Hat OpenShift documentation.
- For more information about the OVN method, see About the OVN-Kubernetes default Container Network Interface (CNI) network provider in the Red Hat OpenShift documentation.
Archive upload API Out-Of-Memory (OOM)
Cryostat 2.0 consumes more memory than expected when a client sends a request to the Cryostat HTTP
/api/v1/recordings handler. This handler points to the
/opt/cryostat.d/recordings.d directory on your Cryostat 2.0 instance and you can use the handler to upload
.jfr binary files to this directory.
The Cryostat Operator sets a default memory limit of 512 MB for a Cryostat instance deployed in an OpenShift project. If you upload a 150 MB or larger
.jfr file to your Cryostat 2.0 instance, the OpenShift cluster’s Out of Memory (OOM) killer stops the pod that contains your deployed Cryostat instance.
You can resolve this known issue by copying your
.jfr binary file to the persistent storage location on your Cryostat 2.0 instance. By using this method, you do not need to send a client request to the Cryostat HTTP
/api/v1/recordings handler to store the
.jfr binary in the
You can issue the following commands to copy your
.jfr binary file to the persistent storage location on your Cryostat 2.0 instance:
oc exec -i -n <your_namespace> -c <cryostat_container_name> <cryostat_pod_name> – mkdir /opt/cryostat.d/recordings.d/unlabelled oc cp vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006t152006z.jfr <cryostat_pod_name>:/opt/cryostat.d/recordings.d/unlabelled/vertx-fib-demo-6f4775cdbf-82dvl_150mb_20211006t152006z.jfr -c <cryostat_container_name>
The previously stated
oc exec command fails if an
unlabelled directory already exists in your
/opt/cryostat.d/recordings.d/ path. You can choose to ignore the failed command message and continue with the
oc cp command.
After you copy the
.jfr binary file directly into the PVC archives location, you can use a
httpie, or a
wget command to verify that the
.jfr file exists on your Cryostat 2.0 instance.
The following example demonstrates using a
curl command to verify that Cryostat recognizes the uploaded file that was copied to the persistent storage location with the
oc cp command. The <cryostat_url> value in the example resolves to
https://cryostat-sample-myproject.apps-crc.testing:443, but you can replace the <cryostat_url> value with your application’s URL. You can determine your application’s URL by issuing the
oc status command.
$ curl -kv -H Authorization:"Bearer $(oc whoami -t)" <cryostat_url>/api/v1/recordings
- For more information about the commands that you can use to resolve the Cryostat container memory limit issue, see OPENJDK-495 in the Red Hat OpenJDK Jira project.
File upload limit for the integrated Grafana component
For Cryostat 2.0, the integrated View in Grafana component cannot accept JFR files larger than 10 MB, because of a configuration issue with the
A deployed Cryostat 2.0 pod’s
jfr-datasource container uses default Quarkus settings, which includes the default
quarkus.http.limits.max-body-size parameter. This parameter sets the maximum size limit for a file on Quarkus, and the parameter has a default value of 10 MB.
If a client attempts to upload a JFR file larger than 10 MB, the
jfr-datasource web server rejects the file and throws an
HTTP 413 error message.
You can resolve this known issue by completing the following steps:
- Navigate to your listed active or archive recording in the Recordings menu on your Cryostat 2.0 instance.
- From the overflow menu for your target recording, click the Download Recording option.
- Save the file to your preferred location on your local system.
- Open the downloaded JFR file in your Java Mission Control (JMC) desktop application.
- For more information about the download functionality on the Recordings menu, see Downloading an active recording or an archived recording in the Getting started with Cryostat 2.0 guide.
Chapter 6. Security fixes
Cryostat 2.0 includes security fixes that improve upon security offered from the previous release of Cryostat. Review each security fix for its description.
jsoup Java library
jsoup Java library that is pre-installed with Cryostat 2.0 now upgraded from version 1.13.1 to version 1.14.2. This new version improves upon the
jsoup Java library’s handling of HTML and XML syntax.
For more information about the bug fixes associated with the
jsoup 1.14.2 release, see jsoup Java HTML Parser release 1.14.2 on the
jsoup documentation website.
Revised on 2021-11-29 17:01:10 UTC