Chapter 9. Connecting to Kafka

Apache Kafka is a distributed streaming platform that you can use to obtain and publish data. In an integration, you can subscribe for data from a Kafka topic that you specify or publish data to a Kafka topic that you specify. To do this, create a connection to Kafka and then add that connection to an integration. Details are in the following topics:

9.1. Create a connection to a Kafka broker

To create a Kafka connection:

  1. In Fuse Online, in the left panel, click Connections to display any available connections.
  2. In the upper right, click Create Connection to display connectors.
  3. Click the Kafka connector.
  4. In the Kafka bootstraps URI field, enter a comma separated list of Kafka broker URIs. Each URI should be in the form host:port.
  5. Click Validate. Fuse Online immediately tries to validate the connection and displays a message that indicates whether validation is successful. If validation fails, revise the input parameter and try again.
  6. If validation is successful, click Next.
  7. In the Connection Name field, enter your choice of a name that helps you distinguish this connection from any other connections. For example, you might enter Kafka West.
  8. In the Description field, optionally enter any information that is helpful to know about this connection.
  9. In the upper right, click Create to see that the connection you created is now available. If you entered the example name, you would see that Kafka West appears as a connection that you can choose to add to an integration.

9.2. Add a Kafka connection to an integration

You must create a Kafka connection before you can add it to an integration. If you did not already create a Kafka connection, see Section 9.1, “Create a connection to a Kafka broker”.

The procedure for adding a Kafka connection to an integration varies according to whether you want to use the Kafka connection to obtain data or publish data. See the following topics:

9.2.1. Obtain data from a Kafka broker

To trigger execution of an integration based on receiving data from a Kafka broker, add a Kafka connection as the start connection. When the integration is running, the Kafka connection continuously watches for data in the Kafka topic that you specify. When the connection finds new data, it passes that data to the next step in the integration.

To execute an integration each time a Kafka topic receives a message:

  1. In the Fuse Online panel on the left, click Integrations.
  2. Click Create Integration.
  3. On the Choose a Start Connection page, click the Kafka connection that you want to use to start the integration.
  4. On the Choose an Action page, click the Subscribe action to receive data from the topic that you specify.
  5. In the Topic Name field, click the down carat to display a list of topics and click the topic that you want to subscribe to.
  6. Click Next to specify the action’s output type. See Specifying connection action input and output types.

9.2.2. Publish data to a Kafka broker

In an integration, you can publish data to a Kafka broker to finish an integration. To do this, add a Kafka connection as the integration’s finish connection. To publish data to a Kafka broker in the middle of integration, add a Kafka connection to an integration after the start connection and before the finish connection.

To add a Kafka connection that publishes data:

  1. Start creating the integration.
  2. Add and configure the start connection.

  3. On the Choose a Finish Connection page, do one of the following:

    • To finish an integration by publishing data, click the Kafka connection that you want to use.
    • To publish data in the middle of an integration, click the connection that you want to use to finish the integration. Configure that connection. When the finish connection is part of the integration, in the left panel, hover over the plus sign where you want to add a Kafka connection and click Add a connection. Click the Kafka connection that you want to use to publish a message in the middle of an integration.
  4. On the Choose an Action page, click Publish.
  5. In the Topic Name field, click the down carat to display a list of topics and click the topic that you want to publish to.
  6. Click Next to specify the action’s input type. See Specifying connection action input and output types.