Chapter 1. Introduction to Camel Kafka Connector
This chapter introduces the features, concepts, and distributions provided by Camel Kafka Connector:
Camel Kafka Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production.
These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview.
1.1. Camel Kafka Connector overview
Apache Camel is a highly flexible open source integration framework for connecting a wide range of different systems, which is based on standard Enterprise Integration Patterns (EIPs). Apache Kafka Connect is the Kafka-native approach for connecting to external systems, which is specifically designed for event-driven architectures.
Camel Kafka Connector enables you to use standard Camel components as Kafka Connect connectors. This widens the scope of possible integrations beyond the external systems supported by Kafka Connect connectors alone. Camel Kafka Connector works as an adapter that makes the popular Camel component ecosystem available in Kafka-based AMQ Streams on OpenShift.
Camel Kafka Connector provides a user-friendly way to configure Camel components directly in the Kafka Connect framework. Using Camel Kafka Connector, you can leverage Camel components for integration with different systems by connecting to or from Camel Kafka sink or source connectors. You do not need to write any code, and can include the appropriate connector JARs in your Kafka Connect image and configure connector options using custom resources.
Camel Kafka Connector is built on Apache Camel Kafka Connector, which is a subproject of the Apache Camel open source community. Camel Kafka Connector is fully integrated with OpenShift Container Platform, AMQ Streams, and Kafka Connect.
Camel Kafka Connector is available with the Red Hat Integration - Camel K distribution for cloud-native integration on OpenShift. Camel K is a lightweight integration framework built from Apache Camel K that runs natively in the cloud on OpenShift. Camel K is specifically designed for serverless and microservice architectures.
1.2. Camel Kafka Connector features
The Camel Kafka Connector Technology Preview includes the following main features:
- OpenShift Container Platform 4.4 or 4.3
- AMQ Streams 1.5
- Kafka Connect 2.5
- Camel 3.1
- Selected Camel Kafka connectors
Table 1.1. Camel Kafka connectors in Technology Preview
| Connector | Sink/source |
|---|---|
| Amazon AWS Kinesis | Sink and source |
| Amazon AWS S3 | Sink and source |
| Cassandra Query Language (CQL) | Sink only |
| Elasticsearch | Sink only |
| Java Message Service (JMS) | Sink and source |
| Salesforce | Source only |
| Syslog | Source only |
1.3. Camel Kafka Connector architecture
AMQ Streams is a distributed and scalable streaming platform based on Apache Kafka that includes a publish/subscribe messaging broker. Kafka Connect provides a framework to integrate Kafka-based systems with external systems. Using Kafka Connect, you can configure source and sink connectors to stream data from external systems into and out of a Kafka broker.
Camel Kafka Connector reuses the flexibility of Camel components and makes them available in Kafka Connect as source and sink connectors that you can use to stream data into and out of AMQ Streams. For example, you can ingest data from Amazon Web Services for processing using an AWS S3 source connector, or consolidate events stored in Kafka into an Elasticsearch instance for analytics using an Elasticsearch sink connector.
The following diagram shows a simplified view of the Camel Kafka Connector cloud-native integration architecture based on AMQ Streams on OpenShift:
Figure 1.1. Camel Kafka Connector architecture
Kafka Connect concepts
The main Kafka Connect concepts include:
- Source connector
- Source connectors work like consumers and pull data from external systems into Kafka topics to make the data available for stream processing. For example, these external source systems include Amazon Web Services or Java Message Service.
- Sink connector
- Sink connectors work like producers and push data from Kafka topics into external systems for offline analysis. For example, these external sink systems include Cassandra, Syslog, or Elasticsearch.
- Sink/source task
- Tasks are typically created by a sink or source connector and are responsible for handling the data.
- Key/value converter
- Key/value converters can serialize/deserialize the key or value of a Kafka message in various formats.
- Transformer
- Transformers can manipulate Kafka message content.
Camel Kafka Connector configuration
You can use Camel Kafka Connector configuration to specify the following:
- Kafka Connect configuration options
- Camel route definitions
- Camel configuration options
Additional resources
1.4. Camel Kafka Connector distributions
The Camel Kafka Connector distributions are bundled with Red Hat Integration - Camel K:
Table 1.2. Camel Kafka Connector available distributions
| Distribution | Description | Location |
|---|---|---|
| Maven repository | Maven artifacts for Camel Kafka Connector | |
| Source code | Source code for Camel Kafka Connector | |
| Demonstration examples | Camel Kafka Connector examples and Debezium and Apache Camel Kafka Connector community example |
You must have a subscription for Red Hat Integration and be logged into the Red Hat Customer Portal to access the Camel Kafka Connector distributions available with Red Hat Integration - Camel K.
