Introduction to Red Hat OpenShift Service Registry

Guide
  • Red Hat OpenShift Service Registry 1
  • Updated 02 December 2021
  • Published 24 September 2021

Introduction to Red Hat OpenShift Service Registry

Guide
Red Hat OpenShift Service Registry 1
  • Updated 02 December 2021
  • Published 24 September 2021

Discover the features and functions available in the Red Hat OpenShift Service Registry cloud service.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code and documentation. We are beginning with these four terms: master, slave, blacklist, and whitelist. Due to the enormity of this endeavor, these changes will be gradually implemented over upcoming releases. For more details on making our language more inclusive, see our CTO Chris Wright’s message.

What is Service Registry?

Service Registry is a datastore for sharing standard event schemas and API designs across event-driven and API architectures. You can use Service Registry to decouple the structure of your data from your client applications, and to share and manage your data types and API descriptions at runtime using a REST interface.

For example, client applications can dynamically push or pull the latest schema updates to or from Service Registry at runtime without needing to redeploy. Developer teams can query the registry for existing schemas required for services already deployed in production, and can register new schemas required for new services in development.

You can enable client applications to use schemas and API designs stored in Service Registry by specifying the registry URL in your client application code. For example, the registry can store schemas used to serialize and deserialize messages, which are then referenced from your client applications to ensure that the messages that they send and receive are compatible with those schemas.

Using Service Registry to decouple your data structure from your applications reduces costs by decreasing overall message size, and creates efficiencies by increasing consistent reuse of schemas and API designs across your organization. Service Registry provides a web console to make it easy for developers and administrators to manage registry content.

In addition, you can configure optional rules to govern the evolution of your registry content. For example, these include rules to ensure that uploaded content is syntactically and semantically valid, or is backwards and forwards compatible with other versions. Any configured rules must pass before new versions can be uploaded to the registry, which ensures that time is not wasted on invalid or incompatible schemas or API designs.

Service Registry is based on the Apicurio Registry open source community project. For details, see https://github.com/apicurio/apicurio-registry.

Service Registry capabilities

  • Deployed as an easy-to-use managed cloud service without having to install, configure, or run your own Service Registry instances

  • Full integration with Red Hat OpenShift Streams for Apache Kafka managed cloud service, including Kafka topic-to-schema mapping

  • Multiple payload formats for standard event schema and API specifications such as Apache Avro, JSON Schema, Protobuf, AsyncAPI, OpenAPI, and more

  • Rules for content validation and version compatibility to govern how registry content evolves over time

  • Registry content management using web console, REST API, command line, Maven plug-in, or Java client

  • Full Apache Kafka schema registry support, including integration with Kafka Connect for external systems

  • Kafka client serializers/deserializers (SerDes) to validate message types at runtime

  • Compatibility with existing Confluent or IBM schema registry client applications

Schema and API artifacts and groups in Service Registry

The items stored in Service Registry, such as event schemas and API designs, are known as registry artifacts. The following shows an example of an Apache Avro schema artifact in JSON format for a simple share price application:

{
   "type": "record",
   "name": "price",
   "namespace": "com.example",
   "fields": [
       {
           "name": "symbol",
           "type": "string"
       },
       {
           "name": "price",
           "type": "string"
       }
   ]
}

When a schema or API design is added as an artifact in the registry, client applications can then use that schema or API design to validate that the client messages conform to the correct data structure at runtime.

Service Registry supports a wide range of message payload formats for standard event schemas and API specifications. For example, supported formats include Apache Avro, Google Protobuf, GraphQL, AsyncAPI, OpenAPI, and others.

Schema and API groups

An artifact group is an optional named collection of schema or API artifacts. Each group contains a logically related set of schemas or API designs, typically managed by a single entity, belonging to a particular application or organization.

You can create optional artifact groups when adding your schemas and API designs to organize them in Service Registry. For example, you could create groups to match your development and production application environments, or your sales and engineering organizations.

Schema and API groups can contain multiple artifact types. For example, you could have Protobuf, Avro, JSON Schema, OpenAPI, and AsyncAPI schema and API artifacts all in the same group.

You can create schema and API artifacts and optional groups using the Service Registry web console, core REST API, command line, Maven plug-in, or Java client application. The following simple example shows using the registry core REST API:

$ curl -X POST -H "Content-type: application/json; artifactType=AVRO" \
  -H "X-Registry-ArtifactId: share-price" \
  --data '{"type":"record","name":"price","namespace":"com.example", \
   "fields":[{"name":"symbol","type":"string"},{"name":"price","type":"string"}]}' \
  https://my-registry.example.com/apis/registry/v2/groups/my-group/artifacts

This example creates an artifact group named my-group and adds an Avro schema with an artifact ID of share-price.

Specifying a group is optional when using the Service Registry web console, where a default group is automatically created. When using the REST API or Maven plug-in, you can specify the default group in the API path if you do not want to create a unique group.
Additional resources

Govern registry content using rules

To govern the evolution of registry content, you can configure optional rules for artifact content added to the registry. All configured global rules or artifact rules must pass before a new artifact version can be uploaded to the registry. Configured artifact rules override any configured global rules.

The goal of these rules is to prevent invalid content from being added to the registry. For example, content can be invalid for the following reasons:

  • Invalid syntax for a given artifact type (for example, AVRO or PROTOBUF)

  • Valid syntax, but semantics violate a specification

  • Incompatibility, when new content includes breaking changes relative to the current artifact version

You can enable optional content rules using the Service Registry web console, REST API commands, or a Java client application.

When rules are applied

Rules are applied only when content is added to the registry. This includes the following REST operations:

  • Adding an artifact

  • Updating an artifact

  • Adding an artifact version

If a rule is violated, Service Registry returns an HTTP error. The response body includes the violated rule and a message showing what went wrong.

If no rules are configured for an artifact, the set of currently configured global rules are applied, if any.

How rules work

Each rule has a name and configuration information. The registry maintains the list of rules for each artifact and the list of global rules. Each rule in the list consists of a name and configuration for the rule implementation.

A rule is provided with the content of the current version of the artifact (if one exists) and the new version of the artifact being added. The rule implementation returns true or false depending on whether the artifact passes the rule. If not, the registry reports the reason why in an HTTP error response. Some rules might not use the previous version of the content. For example, compatibility rules use previous versions, but syntax or semantic validity rules do not.

Manage Service Registry content using client tools

You can manage Service Registry content using the Service Registry web console, core REST API, command line, Maven plug-in, or Java client.

Service Registry web console

You can use the Service Registry web console to browse and search the schema and API artifacts, groups, and versions stored in the registry, and to add new schema and API artifacts, groups, and versions. You can also configure optional validation or compatibility rules for registry content, both globally and for each schema and API artifact.

Service Registry web console
Figure 1. Service Registry web console

Core registry REST API

Using the Service Registry core REST API, client applications can manage the schema and API artifacts in Service Registry. This API provides create, read, update, and delete operations for artifacts, versions, metadata, and rules. It also provides search and admin operations, such as managing global rules or importing and exporting registry data.

Service Registry provides compatibility with other schema registry APIs, such as Confluent Schema Registry and CNCF CloudEvents Schema Registry.

rhoas CLI

You can use the Red Hat OpenShift Application Services (rhoas) command line interface to create, read, and delete Service Registry instances, and to manage schema and API artifacts stored Service Registry. The rhoas CLI provides create, read, update, and delete operations for registry artifacts, versions, and metadata.

Service Registry Java client

You can use the Service Registry Java client classes to create, read, update, or delete artifacts stored in Service Registry. You can also perform admin functions using the Java client, such as managing global rules or importing and exporting registry data.

Service Registry Maven plug-in

You can use the Service Registry Maven plug-in to add, download, or test Service Registry artifacts in a Maven build. This includes adding artifacts during a build, or downloading artifacts when generating code from a schema. If content rules are configured, you can test that artifacts can be added before making changes.

Map Kafka topics to schemas

Service Registry provides full integration with Red Hat OpenShift Streams for Apache Kafka to map your Kafka topics to the appropriate schemas.

You can use the OpenShift Streams for Apache Kafka web console to:

  • Map your Kafka topics to specific schemas stored in Service Registry

  • Check the schemas that are used for specific Kafka topic key and topic values

You can also create mappings between Kafka topics and schemas automatically when registering schemas with Service Registry in your Kafka client application code. You can do this using the Service Registry Kafka client serializers/deserializers (SerDes) in your Java Kafka application code.

For example, you can configure that the serializer automatically tries to create the schema artifact in Service Registry. In addition, the default schema lookup strategy used by the serializer is the TopicIdStrategy. This looks for schema artifacts in the registry with the same name as the Kafka topic receiving the messages.

Additional resources

Validate Kafka messages using schemas and Java client serializers/deserializers

Kafka producer applications can use serializers to encode messages that conform to a specific event schema. Kafka consumer applications can then use deserializers to validate that messages have been serialized using the correct schema, based on a specific schema ID.

Registry SerDes architecture
Figure 2. Service Registry and Kafka client SerDes architecture

Service Registry provides Kafka client serializers/deserializers (SerDes) to validate the following message types at runtime:

  • Apache Avro

  • Google protocol buffers

  • JSON Schema

The Service Registry Maven repository includes the Kafka SerDes implementations for these message types, which Kafka client developers can use to integrate with the registry.

These implementations include custom Java classes for each supported message type, for example, io.apicurio.registry.serde.avro, which client applications can use to pull schemas from the registry at runtime for validation.

Stream data to external systems with Kafka Connect converters

You can use Service Registry with Apache Kafka Connect to stream data between Kafka and external systems. Using Kafka Connect, you can define connectors for different systems to move large volumes of data into and out of Kafka-based systems.

Registry and Kafka Connect architecture
Figure 3. Service Registry and Kafka Connect architecture

Service Registry provides the following features for Kafka Connect:

  • Storage for Kafka Connect schemas

  • Kafka Connect converters for Apache Avro and JSON Schema

  • Registry REST API to manage schemas

You can use the Avro and JSON Schema converters to map Kafka Connect schemas into Avro or JSON schemas. Those schemas can then serialize message keys and values into the compact Avro binary format or human-readable JSON format. The converted JSON is also less verbose because the messages do not contain the schema information, only the schema ID.

Service Registry can manage and track the Avro and JSON schemas used in the Kafka topics. Because the schemas are stored in Service Registry and decoupled from the message content, each message must only include a tiny schema identifier. For an I/O bound system like Kafka, this means more total throughput for producers and consumers.

The Avro and JSON Schema serializers and deserializers (SerDes) provided by Service Registry are also used by Kafka producers and consumers in this use case. Kafka consumer applications that you write to consume change events can use the Avro or JSON Serdes to deserialize these change events. You can install these SerDes into any Kafka-based system and use them along with Kafka Connect, or with Kafka Connect-based systems such as Debezium and Camel Kafka Connector.

Service Registry Java libraries

Service Registry provides Java libraries for client applications to connect to Service Registry instances, manage their registry content, or serialize/deserialize and validate Kafka messages. Service Registry Java client applications can:

  • Use a Java client or Maven plug-in to manage schema and API artifacts stored in Service Registry

  • Use Kafka serializers/deserializers (SerDes) for Apache Avro, Protobuf, or JSON schema to serialize/derserialize and validate Kafka message content

  • Use Kafka Connect converters to map Kafka Connect schemas into Avro or JSON schemas when streaming data to external systems

Apicurio Registry libraries in Maven

The following Apicurio Registry community-based Java libraries are available from Maven Central:

Service Registry demonstration examples

Service Registry provides open source example applications that demonstrate how to use the registry in different use case scenarios. For example, these include storing schemas used by Kafka serializer and deserializer (SerDes) classes. These Java classes fetch the schema from the registry for use when producing or consuming operations to serialize, deserialize, or validate the Kafka message payload.

These example applications include the following:

  • Simple Avro

  • Simple JSON Schema

  • Confluent SerDes integration

  • Avro bean

  • Custom ID strategy

  • Simple Avro Maven

  • REST client

  • Mix Avro schemas

  • Cloud Events

Additional resources

Try Red Hat OpenShift Service Registry

Service Registry is available with Red Hat OpenShift Streams for Apache Kafka and Red Hat OpenShift API Management.

To try out Service Registry, you can create a Service Registry trial instance using the web console. To learn more, go to console.redhat.com.