Chapter 2. Getting started

1. Add a new kafka user and group:

   groupadd kafka
   useradd -g kafka kafka
   passwd kafka

2. Extract and move the contents of the amq-streams-<version>-bin.zip file into the /opt/kafka directory.

   unzip amq-streams-<version>-bin.zip -d /opt
   mv /opt/kafka*redhat* /opt/kafka

   This step requires admin privileges.

3. Change the ownership of the /opt/kafka directory to the kafka user:

   chown -R kafka:kafka /opt/kafka

4. Create directory /var/lib/zookeeper for storing ZooKeeper data and set its ownership to the kafka user:

   mkdir /var/lib/zookeeper
   chown -R kafka:kafka /var/lib/zookeeper

5. Create directory /var/lib/kafka for storing Kafka data and set its ownership to the kafka user:

   mkdir /var/lib/kafka
   chown -R kafka:kafka /var/lib/kafka

   You can now run a default configuration of Kafka as a single-node cluster.

   You can also use the installation to run other Kafka components, like Kafka Connect, on the same host.

   To run other components, specify the hostname and port to connect to the Kafka broker using the bootstrap.servers property in the component configuration.

   Example bootstrap servers configuration pointing to a single Kafka broker on the same host

   bootstrap.servers=localhost:9092

   However, we recommend installing and running Kafka components on separate hosts.

6. (Optional) Install Kafka components on separate hosts.

   1. Extract the installation files to the /opt/kafka directory on each host.
   2. Change the ownership of the /opt/kafka directory to the kafka user.

   3. Add bootstrap.servers configuration that connects the component to the host (or hosts in a multi-node environment) running the Kafka brokers.

      Example bootstrap servers configuration pointing to Kafka brokers on different hosts

      bootstrap.servers=kafka0.<host_ip_address>:9092,kafka1.<host_ip_address>:9092,kafka2.<host_ip_address>:9092

      You can use this configuration for Kafka Connect, MirrorMaker 2.0, and the Kafka Bridge.

Running the cluster

1. Edit the ZooKeeper configuration file /opt/kafka/config/zookeeper.properties. Set the dataDir option to /var/lib/zookeeper/:

   dataDir=/var/lib/zookeeper/

2. Edit the Kafka configuration file /opt/kafka/config/server.properties. Set the log.dirs option to /var/lib/kafka/:

   log.dirs=/var/lib/kafka/

3. Switch to the kafka user:

   su - kafka

4. Start ZooKeeper:

   /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties

5. Check that ZooKeeper is running:

   jcmd | grep zookeeper

   Returns:

   number org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/kafka/config/zookeeper.properties

6. Start Kafka:

   /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

7. Check that Kafka is running:

   jcmd | grep kafka

   Returns:

   number kafka.Kafka /opt/kafka/config/server.properties

Procedure

1. Start the Kafka console producer and configure it to send messages to a new topic:

   /opt/kafka/bin/kafka-console-producer.sh --broker-list <bootstrap_address> --topic <topic-name>

   For example:

   /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

2. Enter several messages into the console. Press Enter to send each individual message to your new topic:

   >message 1
   >message 2
   >message 3
   >message 4

   When Kafka creates a new topic automatically, you might receive a warning that the topic does not exist:

   WARN Error while fetching metadata with correlation id 39 :
   {4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

   The warning should not reappear after you send further messages.

3. In a new terminal window, start the Kafka console consumer and configure it to read messages from the beginning of your new topic.

   /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server <bootstrap_address> --topic <topic-name> --from-beginning

   For example:

   /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

   The incoming messages display in the consumer console.

4. Switch to the producer console and send additional messages. Check that they display in the consumer console.
5. Stop the Kafka console producer and then the consumer by pressing Ctrl+C.

Procedure

1. Stop the Kafka broker.

   su - kafka
   /opt/kafka/bin/kafka-server-stop.sh

2. Confirm that the Kafka broker is stopped.

   jcmd | grep kafka

3. Stop ZooKeeper.

   su - kafka
   /opt/kafka/bin/zookeeper-server-stop.sh