Chapter 2. Connecting to Kafka with Kamelets

1. Log in to your OpenShift cluster’s web console.
2. Create or open a project in which you plan to create your integration, for example my-camel-k-kafka.
3. Install the Camel K operator and Camel K CLI as described in Installing Camel K.

4. Install the AMQ streams operator:

   1. From any project, select Operators > OperatorHub.
   2. In the Filter by Keyword field, type AMQ Streams.

   3. Click the Red Hat Integration - AMQ Streams card and then click Install.

      The Install Operator page opens.

   4. Accept the defaults and then click Install.
5. Select Operators > Installed Operators to verify that the Camel K and AMQ Streams operators are installed.
6. Set up Kafka authentication as described in Managing secure access to Kafka.

1. Log in to your OpenShift cluster’s web console.
2. Select Projects and then click the project in which you installed the Red Hat Integration - AMQ Streams operator. For example, click the my-camel-k-kafka project.
3. Select Operators > Installed Operators and then click Red Hat Integration - AMQ Streams.

4. Create a Kafka cluster:

   1. Under Kafka, click Create instance.
   2. Type a name for the cluster, for example kafka-test.

   3. Accept the other defaults and then click Create.

      The process to create the Kafka instance might take a few minutes to complete.

      When the status is ready, continue to the next step.

5. Create a Kafka topic:

   1. Select Operators > Installed Operators and then click Red Hat Integration - AMQ Streams.
   2. Under Kafka Topic, click Create Kafka Topic.
   3. Type a name for the topic, for example test-topic.
   4. Accept the other defaults and then click Create.

Procedure

1. Log in to your OpenShift web console with a cluster admin account.

2. Create the OpenShift project for your Camel K or kamelets application.

   1. Select Home > Projects.
   2. Click Create Project.
   3. Type the name of the project, for example my-camel-k-kafka, then click Create.

3. Install the RHOAS operator:

   1. From any project, select Operators > OperatorHub.
   2. In the Filter by Keyword field, type RHOAS.

   3. Click the OpenShift Application Services (RHOAS) card (it is a community operator) and then click Install.

      The Install Operator page opens.

   4. Accept the default mode (All namespaces on the cluster) or select the namespace for your project, then click Install.
4. Download and install the RHOAS CLI as described in Getting started with the rhoas CLI.
5. Install the Camel K operator and Camel K CLI as described in Installing Camel K.
6. To verify that the Red Hat Integration - Camel K and OpenShift Application Services (RHOAS) operators are installed, click Operators > Installed Operators.

1. From the command line, log in to your OpenShift cluster.

2. Open your project, for example:

   oc project my-camel-k-kafka

3. Verify that the necessary operators are installed in your project:

   oc get csv

   The result lists the Red Hat Camel K and RHOAS operators and indicates that they are in the Succeeded phase.

4. Prepare and connect a Kafka instance to RHOAS:

   1. Login to the RHOAS CLI by using this command:

      rhoas login

   2. Create a kafka instance, for example kafka-test:

      rhoas kafka create kafka-test

      The process to create the Kafka instance might take a few minutes to complete.

5. To check the status of your Kafka instance:

   rhoas status

   You can also view the status in the web console:

   https://cloud.redhat.com/beta/application-services/streams/kafkas/

   When the status is ready, continue to the next step.

6. Create a new Kafka topic:

   rhoas kafka topic create test-topic

7. Connect your Kafka instance (cluster) with the Openshift Application Services instance:

   rhoas cluster connect

8. Follow the script instructions for obtaining a credential token.

   You should see output similar to the following:

   Token Secret "rh-cloud-services-accesstoken-cli" created successfully
   Service Account Secret "rh-cloud-services-service-account" created successfully
   KafkaConnection resource "kafka-test" has been created
   KafkaConnection successfully installed on your cluster.

   The RHOAS operator sets up the KafkaConnection custom resource named kafka-test.

Procedure

1. Obtain the Kafka Broker URL (Bootstrap URL):

   rhoas status kafka

   This command returns output similar to the following:

     Kafka
     ---------------------------------------------------------------
     ID:                     1ptdfZRHmLKwqW6A3YKM2MawgDh
     Name:                   my-kafka
     Status:                 ready
     Bootstrap URL:        my-kafka--ptdfzrhmlkwqw-a-ykm-mawgdh.kafka.devshift.org:443

2. To obtain a username and password, create a service account by using the following syntax:

   rhoas service-account create --name "<account-name>" --file-format json

   Note

   When creating a service account, you can choose the file format and location to save the credentials. For more information, type rhoas service-account create --help

   For example:

   rhoas service-account create --name "my-service-acct" --file-format json

   The service account is created and saved to a JSON file.

3. To verify your service account credentials, view the credentials.json file:

   cat credentials.json

   This command returns output similar to the following:

   {"user":"srvc-acct-eb575691-b94a-41f1-ab97-50ade0cd1094", "password":"facf3df1-3c8d-4253-aa87-8c95ca5e1225"}

Procedure

1. Edit the application.properties file and add the Kafka credentials.

   application.properties file

   camel.component.kafka.brokers = <YOUR-KAFKA-BOOTSTRAP-URL-HERE>
   camel.component.kafka.security-protocol = SASL_SSL
   camel.component.kafka.sasl-mechanism = PLAIN
   camel.component.kafka.sasl-jaas-config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<YOUR-SERVICE-ACCOUNT-ID-HERE>' password='<YOUR-SERVICE-ACCOUNT-SECRET-HERE>';
   consumer.topic=<TOPIC-NAME>
   producer.topic=<TOPIC-NAME>

2. Run the following command to create a secret that contains the sensitive properties in the application.properties file:

   oc create secret generic kafka-props --from-file application.properties

   You use this secret when you run a Camel K integration.

Procedure

1. Edit the application-oauth.properties file and add the Kafka credentials.

   application-oauth.properties file

   camel.component.kafka.brokers = <YOUR-KAFKA-BOOTSTRAP-URL-HERE>
   camel.component.kafka.security-protocol = SASL_SSL
   camel.component.kafka.sasl-mechanism = OAUTHBEARER
   camel.component.kafka.sasl-jaas-config = org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
   oauth.client.id='<YOUR-SERVICE-ACCOUNT-ID-HERE>' \
   oauth.client.secret='<YOUR-SERVICE-ACCOUNT-SECRET-HERE>' \
   oauth.token.endpoint.uri="https://identity.api.openshift.com/auth/realms/rhoas/protocol/openid-connect/token" ;
   consumer.topic=<TOPIC-NAME>
   producer.topic=<TOPIC-NAME>

2. Run the following command to create a secret that contains the sensitive properties in the application.properties file:

   oc create secret generic kafka-props --from-file application-oauth.properties

   You use this secret when you run a Camel K integration.