Release notes for the Red Hat build of Cryostat 2.2

Red Hat build of Cryostat 2

Red Hat Customer Content Services

Abstract

The Release notes for the Red Hat build of Cryostat 2.2 document provides an overview of new features in Cryostat 2.2 and a list of potential known issues and possible workarounds.

Preface

The Red Hat build of Cryostat is a container-native implementation of JDK Flight Recorder (JFR) that you can use to securely monitor the Java Virtual Machine (JVM) performance in workloads that run on an OpenShift Container Platform cluster. You can use Cryostat 2.2 to start, stop, retrieve, archive, import, and export JFR data for JVMs inside your containerized applications by using a web console or an HTTP API.

Depending on your use case, you can store and analyze your recordings directly on your Red Hat OpenShift cluster by using the built-in tools that Cryostat provides or you can export recordings to an external monitoring application to perform a more in-depth analysis of your recorded data.

Important

Red Hat build of Cryostat is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Providing feedback on Red Hat documentation

We appreciate your feedback on our documentation. To provide feedback, you can highlight the text in a document and add comments. Follow the steps in the procedure to learn about submitting feedback on Red Hat documentation.

Prerequisites

  • Log in to the Red Hat Customer Portal.
  • In the Red Hat Customer Portal, view the document in Multi-page HTML format.

Procedure

  1. Click the Feedback button to see existing reader comments.

    Note

    The feedback feature is enabled only in the Multi-page HTML format.

  2. Highlight the section of the document where you want to provide feedback.
  3. In the prompt menu that opens near the text you selected, click Add Feedback.

    A text box opens in the feedback section on the right side of the page.

  4. Enter your feedback in the text box and click Submit.

    You have created a documentation issue.

  5. To view the issue, click the issue tracker link in the feedback view.

Chapter 1. Support policy for Cryostat

Red Hat supports a major version of Cryostat for a minimum of 6 months. Red Hat bases this figure on the time that the product gets released on the Red Hat Customer Portal.

You can install and deploy Cryostat on Red Hat OpenShift Container Platform 4.8 or a later version that runs on an x86_64 architecture.

Additional resources

  • For more information about the Cryostat life cycle policy, see Red Hat build of Cryostat on the Red Hat OpenShift Container Platform Life Cycle Policy web page.

Chapter 2. New features

Cryostat 2.2 introduces new features that enhance your use of the Cryostat product.

Automatic archival of JFR recordings

Cryostat 2.2 adds the Archive on Stop checkbox to the Recordings menu, so that you can configure Cryostat to automatically archive new JFR recordings to the persistent volume claim (PVC) storage location on Red Hat OpenShift. By default, Cryostat selects the Archive on Stop checkbox. If you select the Continuous checkbox, Cryostat clears the Archive on Stop checkbox.

Additionally, you can issue a curl command in your CLI that requests Cryostat to automatically archive a single JFR recording to the PVC storage location on Red Hat OpenShift. The following example of a curl command specifies Cryostat to automatically archive any JFR recording file with a given name, duration, and event template.

$ curl --data "recordingName=foo2&duration=5&events=template=ALL&archiveOnStop=true" localhost:8181/api/v1/targets/localhost/recordings

 

GraphQL API

Cryostat 2.2 includes a GraphQL API endpoint, /api/v2.2/graphql, so that you can use the API to run shorter and simpler queries against target JVMs. You can also run these queries against any of a target JVM’s active and archived recordings.

The API can run queries against general Cryostat archives. You can then customize the queries to automate the following tasks for active or archived recordings:

  • Archive
  • Delete
  • Start
  • Stop

Cryostat 2.2 introduces the following query capabilities:

  • New filter options that apply to all query types.
  • The GraphQL API can update JFR recording labels. Before Cryostat 2.2, you could only update a JFR recording’s label by using the HTTP REST API, which has limited workload capabilities.

The GraphQL API is more powerful than the HTTP REST API, which has limited workload capabilities. For example, the HTTP REST API would require you to create an API request for each copy of a recording that you want to start on each scaled replica that is inside a container on Red Hat OpenShift. The GraphQL API can achieve this task in one API request, which improves the API’s performance and reduces any network traffic for your Cryostat instance.

Cryostat 2.2 supports the HTTP REST API and includes some enhancements for this API, such as including new endpoints for downloading JFR files by using JWT tokens. However, this API requires more user intervention, such as requiring you to write custom clients to parse API JSON responses when performing iterative actions on response data.

Pluggable Discovery API

Cryostat supports the /api/v2.2/discovery endpoint, so you can use the Pluggable Discovery API to register external clients with Cryostat.

The API can register or unregister a client based on Cryostat’s environment.

Note

The Cryostat Operator on Red Hat OpenShift creates a service account for Cryostat. After you create an application on Red Hat OpenShift, the application can connect with Cryostat’s Discovery Plugin API through the service account.

You can disable Cryostat built-in target discovery mechanism, so that Cryostat uses the Discovery Plugin API as the primary target discovery mechanism. To disable Cryostat’s built-in discovery mechanism, you would set the following environment variable to true:

CRYOSTAT_DISABLE_BUILTIN_DISCOVERY=true

The Discovery Plugin API contains all the capabilities of Cryostat’s built-in target discover mechanism.

Permission mapping with ConfigMap

You can now use create a ConfigMap resource that includes specific permissions from a properties file. The Cryostat Operator can then apply the permissions to your Cryostat instance. You no longer need to use the default permissions file that the Cryostat Operator mounts to Cryostat.

Security context in a Cryostat CRD

You can now specify security contexts for a Cryostat Custom Resource (CRD) on Red Hat OpenShift, so that you can configure your Cryostat Operator to better distribute Cryostat workloads across Red Hat OpenShift clusters.

Before the Cryostat 2.2, the Cryostat Operator would use a default security context for Cryostat, which would limit permissions for your Cryostat Operator when distributing Cryostat workloads across Red Hat OpenShift clusters.

Support for uploading automated rule definitions in JSON

By uploading automated rule definitions in JSON, you can reuse a rule definition for your automated rule. This functionality is similar to how you would upload a recording to the Archives menu.

From the Automated Rules menu on the Cryostat web console, you can now complete the following tasks:

  • Configure a rule definition file with your preferred text editor or make additional copies of the file.
  • Disable a rule definition, which prevents the rule from interacting with a JFR recording. You can enable the rule definition at a later stage.
  • Download a rule definition file, such as continuous, in JSON format. Functionality is similar to how you would download a recording from the Recordings menu.
  • Upload a rule definition in JSON format. By using this function, you can reuse a rule definition for your automated rule. Functionality is similar to how you would upload a recording to the Archives menu.

Chapter 3. Feature enhancements

Cryostat 2.2 includes feature enhancements that build upon the Cryostat 2.1 offerings.

Automated rule behavior after JMX credential change

Before Cryostat 2.2, if you created an automated rule to monitor a target JVM application before you enter the application’s JMX credentials on the Security menu, the automated rule would fail without any warning.

Cryostat 2.2 resolves this issue. With this update, you can enter JMX credentials after you created the automated rule without experiencing issues. You do not need to re-create the automated rule, because Cryostat retries the automated rule after you enter the correct JMX credentials for the application.

Note

Cryostat 2.2 encrypts and stores JMX credentials for a target JVM application in a database that is stored on a persistent volume claim (PVC) on Red Hat OpenShift.

The Cryostat Operator stores JMX credentials in memory for the duration of establishing a connection between Cryostat and the target JVM application. Cryostat supports SSL/TLS on the HTTP request that adds JMX credentials to the database and on the JMX connection that uses those credentials to connect to the target application. Cryostat also encrypts the credentials within the database by using a passphrase that is either provided by the user or that is generated by the Cryostat Operator.

Deletion prompt on the Cryostat web console

Cryostat 2.2 updates the Delete function on the Cryostat web console, so that after you click Delete, the following prompt opens on your web console:

Figure 3.1. The Delete prompt that opens on the Cryostat web console

The *Delete* prompt that opens on the Cryostat web console

You can access this prompt when you complete any of the following delete operations on the web console:

  • An automated rule from the Automated Rules menu.
  • An active or an archived recording from either the Recordings menu or the Archives menu.
  • An event template, an event type, or a custom target from numerous locations on the Cryostat web console, such as the Events menu.
  • JMX credentials from the Security menu.
Important

The prompt does not open when you attempt to delete recording labels for either your active recording or archived recording.

The prompt informs you that after you delete the recording, Cryostat removes all data associated with the recording.

The prompt provides you with the following options:

  • To proceed with deleting the recording, click Delete.
  • To keep the recording, click Cancel.

After you click either option, Cryostat returns to the previous menu.

Edit Recording Labels pane

Cryostat 2.2 removes the Edit Metadata option from the Recordings menu, and adds an Edit Labels button under the Active Recordings tab and the Archived Recordings tab.

Figure 3.2. The Edit Recordings Labels pane on the Cryostat web console

The *Edit Recordings Labels* pane on the Cryostat web console

After you click the Edit Labels button, an Editing Recordings Labels pane opens on your Cryostat web console.

Enhancement to the Archives menu

The Archives menu separates recordings into three nested tables: All Targets, All Archives, and Uploads. Each table lists results in a chronological order.

After you archive a recording, Cryostat lists the associated target JVM in the All Targets table. You can click the Hide targets with zero recordings checkbox to remove any target JVM entry that does not have an archived recording.

After you click on the twistie icon (v) beside the JVM target entry, you can access a filter function, where you can edit labels to enhance your filter or click the Delete button to remove the filter.

The All Archives table looks similar to the All Targets table, but the All Archives table lists target JVM applications from files that Cryostat archived.

From the Uploads table, you can view all your uploaded JFR recordings. The Uploads table also includes a filtering mechanism similar to the All Targets table.

You can also use the filtering mechanism on the Archives menu to find an archived file that might have no recognizable target JVM application.

Fixed issue with Archive Recordings table

Cryostat 2.2 fixes the Cryostat 2.1 issue, where Cryostat would delete one of the duplicate files from under the Archive Recordings table. This issue is caused when you complete the following steps:

  1. Archived a JFR recording that belongs to a specific target. As an example, Cryostat names the archived file as my_recording_9093_20220322T172832Z.jfr.
  2. Archived the same JFR recording again or uploaded the same file to Cryostat’s archives location.
  3. Cryostat might remove one of the files in error.

You would notice the incorrect deleted file when you complete one of the following actions:

  • View a generated automated analysis report.
  • View application metrics on Grafana.
  • Edit a recording label for the existing JFR recording.

If Cryostat 2.2 detects JFR recordings files with the same name, but each file has a different target, Cryostat does not delete one of the files. This behavior also applies when you re-upload a file with the same name as the archived file belonging to a target JVM application.

For more information about the source of the issue, see Duplicate file name displays under the Archived Recordings table (Release notes for the Red Hat build of Cryostat 2.1).

Filter recordings

From either the Active Recordings tab or the Archived Recordings tab in the Recordings menu, you can filter listed JFR recordings by selecting checkboxes that open beside each JFR recording entry.

After you click a checkbox, Cryostat enables buttons, such as Create, Archive, Edit Labels, and so on.

With Cryostat’s filtering functionality, you can create a filter that accurately finds your target JFR recordings.

The following image shows an Active Recordings table with three listed active recordings:

Figure 3.3. Example of an Active Recordings table that shows three listed active recordings

cryostat filter listed recordings 1

The following example shows a filter with defined template.type:TARGET label and the DurationSeconds: continuous label criteria. After the filter query completes, two results show that match the filter’s label criteria.

Figure 3.4. Example of a completed filter under the Active Recordings tab

cryostat filter listed recordings 2

 

OpenJDK 17 and Eclipse Vert.x support

Cryostat 2.2 is built with the Vert.x 4 framework. This framework improves performance, fixes legacy bugs, and builds new features for Cryostat.

Additionally, Cryostat 2.2 is built with OpenJDK 17. This OpenJDK implementation improves performance and reduces memory requirements for Cryostat.

Note

If you run your Cryostat application on OpenJDK 17, Cryostat can still interact with a target JVM application that was built with a different release of Cryostat that supports the JFR technology, such as OpenJDK 11.0.17.

The initialDelay automated rule property

Before the Cryostat 2.2 release, if you created an automated rule that copies recordings into Cryostat’s archives, you would need to create a schedule by specifying a value for the archivalPeriodSeconds property. This configuration limits an automated rule to only move a recording copy to archives based at specific time intervals. You cannot stagger the archival period with this archivalPeriodSeconds property.

Cryostat 2.2 includes the initialDelay property, which you can define in rule definition. Your automated rule can then stagger the archival schedule to meet your needs. Consider a situation where you would like to immediately archive a recording during application startup. Thereafter, the archival interval could be scheduled to occur every 30 seconds.

Updated Archived Recordings table

Cryostat 2.2 updates the Archived Recording table that opens on the Recordings menu. The table now includes a Size column, where you can view an archived file size in kilobytes (KB) units.

Additionally, with this release, you can use a scroll bar on the Archived Recording table to quickly locate an archived file. The scroll bar visibility is unique to each table type. For example, from the Active Recordings table, you can access the scroll bar when the table lists 5 or more active JFR recording files.

Chapter 4. Unsupported and deprecated features

Cryostat 2.2 removes some features because of their high maintenance costs, low community interest, and better alternative solutions.

Removed support for OpenShift Operator CRDs

You cannot use the FlightRecorder or the Recording custom resources definitions (CRD) with Cryostat 2.2. Cryostat includes the HTTP REST API and the GraphQL API that provide the same functionality as the OpenShift Operator’s CRDs. Additionally, the Cryostat APIs include more features than these CRDs.

Chapter 5. Fixed issues

The Cryostat release might include fixes for issues that were identified in earlier releases of Cryostat. Review each fixed issue note for a description of the issue and the subsequent fix.

Issues fixed in Cryostat 2.2.1

The following issues have been fixed in Cryostat 2.2.1 release:

Successful authentication of JMX session

Before Cryostat 2.2.1, the JMX authentication connection failed when you attempted to connect Cryostat to a target JVM application over JMX, and you specified JMX credentials over the X-JMX-Authorization header. With this update, the JMX authentication connection failure issue is resolved, so that you can connect Cryostat to a target JVM application over JMX.

Deployment recreation is applied with configuration changes

Before Cryostat 2.2.1, Cryostat failed to perform a rolling update on itself in response to changes made to its configuration. This issue existed because of a conflict on a lock file in the new JMX credentials database. With this update, Cryostat does not attempt a rolling update but instead recreates its deployments in response to changes made to its configuration.

Issues fixed in Cryostat 2.2.0

The following issues have been fixed in Cryostat 2.2.0 release:

Automated analysis report results

Before Cryostat 2.2.0, Cryostat did not update analysis reports to reflect a stopped JFR recording. The following sequence of steps demonstrates how you could reach this behavior in a previous version of Cryostat:

  1. Start a JFR recording.
  2. Generate an analysis report.
  3. Stop the JFR recording.
  4. View the latest generated analysis report.

The latest generated report would show results of a snapshot from the last active recording and not the stopped recording. Cryostat does not cache a copy of the report based on the stopped recording’s data, so you can only view a report from the last active recording snapshot.

Cryostat 2.2.0 fixes this issue by generating a report on data cached from the stopped JFR recording.

Fixed access issue with the Create recording button

After Cryostat loses a connection with a target JVM and then re-establishes this connection, Cryostat no longer either blocks your access to the Create button in the Recordings menu or outputs a 500 NoRouteToHost error message.

Revised on 2022-12-23 11:20:09 UTC

Legal Notice

Copyright © 2022 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.