Kafka CLI Tools and Essential Commands

Apache Kafka is a powerful distributed event streaming platform, but to manage and interact with it effectively, you need to master its Command Line Interface (CLI) tools. These CLI tools are shipped directly with the Kafka distribution and allow developers and administrators to manage topics, produce and consume messages, inspect consumer groups, and troubleshoot cluster states.

In this guide, we will explore the most essential Kafka CLI commands from the ground up. Whether you are setting up a local development environment or managing a production cluster, these commands are fundamental to your daily workflow.

Understanding the Kafka CLI Architecture

Before executing commands, it is crucial to understand how the CLI tools interact with your Kafka cluster. In older versions of Kafka, many CLI tools communicated directly with ZooKeeper. In modern Kafka (including KRaft mode), all CLI tools communicate directly with the Kafka Brokers using the bootstrap server address.

+-------------------------------------------------------------+
|                        Kafka CLI Tools                      |
|  (kafka-topics, kafka-console-producer, kafka-consumer)    |
+-------------------------------------------------------------+
                               |
                               | TCP Connection (Port 9092)
                               v
+-------------------------------------------------------------+
|                    Kafka Bootstrap Server                   |
|                   (Broker 1, Broker 2, ...)                 |
+-------------------------------------------------------------+
    

When executing any command, you must provide the --bootstrap-server flag followed by the host and port of at least one broker in your cluster (usually localhost:9092 for local development).

1. Topic Management: kafka-topics.sh

The kafka-topics.sh tool is used to create, alter, list, describe, and delete topics within your Kafka cluster.

Creating a Kafka Topic

To create a new topic, you must specify the topic name, the number of partitions, and the replication factor. The replication factor cannot exceed the number of active brokers in your cluster.

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic order-events --partitions 3 --replication-factor 1

Listing All Topics

To verify that your topic was created successfully, you can list all active topics in the cluster:

kafka-topics.sh --bootstrap-server localhost:9092 --list

Describing Topic Details

The describe command provides detailed configuration information about a topic, including its partitions, leader replicas, in-sync replicas (ISR), and custom configurations.

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic order-events

The output of the describe command looks like this:

Topic: order-events   TopicId: x87yHz... PartitionCount: 3   ReplicationFactor: 1
Topic: order-events   Partition: 0    Leader: 1   Replicas: 1   Isr: 1
Topic: order-events   Partition: 1    Leader: 1   Replicas: 1   Isr: 1
Topic: order-events   Partition: 2    Leader: 1   Replicas: 1   Isr: 1

Modifying Partitions

You can increase the number of partitions for an existing topic. Note that you cannot decrease the number of partitions.

kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic order-events --partitions 5

Deleting a Topic

To delete a topic and all of its associated data, use the delete command:

kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic order-events

2. Producing Messages: kafka-console-producer.sh

The kafka-console-producer.sh tool allows you to send messages to a Kafka topic directly from your terminal. This is highly useful for testing consumer applications.

Sending Simple Messages

Run the following command to open an interactive prompt where every line you type and send with Enter is produced as a message value:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic order-events

Once the prompt opens, type messages like this:

> First order placed
> Second order completed
> Payment received

Sending Messages with Keys

By default, the console producer sends messages with null keys. To send key-value pairs, you must configure a key separator:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic order-events --property parse.key=true --property key.separator=:

Now you can type messages as key-value pairs:

> orderId123:{"status":"PENDING"}
> orderId456:{"status":"COMPLETED"}

3. Consuming Messages: kafka-console-consumer.sh

The kafka-console-consumer.sh tool reads messages from a Kafka topic and prints them to your standard output.

Consuming Live Messages

This command starts consuming messages that are produced *after* the consumer starts up:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-events

Consuming All Historical Messages

To read all messages from the very beginning of the topic's log, append the --from-beginning flag:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-events --from-beginning

Displaying Keys and Metadata

By default, only the message values are printed. To view keys, partition numbers, and timestamps, use the formatter properties:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-events --from-beginning --property print.key=true --property print.partition=true --property print.timestamp=true

4. Managing Consumer Groups: kafka-consumer-groups.sh

In production, consumers are grouped together in Consumer Groups to share the processing load. The kafka-consumer-groups.sh tool is essential for monitoring and managing group offsets.

Listing Active Consumer Groups

To see all active consumer groups in your cluster:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Describing Consumer Group Lag

This is one of the most critical commands for production monitoring. It shows the current offset, log end offset, and "lag" (how far behind the consumer is from the latest message) for each partition.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-billing-service

Example Output:

GROUP              TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
my-billing-service order-events   0          1045            1050            5               client-1        /127.0.0.1      client-1
my-billing-service order-events   1          2011            2011            0               client-1        /127.0.0.1      client-1

Resetting Consumer Group Offsets

If your application encounters a bug and you need to reprocess data, you can reset the consumer group offsets. Note that the consumer group must be completely inactive (stopped) to execute this command.

To reset offsets to the earliest available message:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-billing-service --reset-offsets --to-earliest --topic order-events --execute

To shift offsets back by a specific duration (e.g., 30 minutes):

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-billing-service --reset-offsets --by-duration PT0H30M0S --topic order-events --execute

Real-World Use Cases

• Production Debugging: If a downstream service fails, developers can spin up a kafka-console-consumer.sh with the --from-beginning flag to inspect the exact payload sent to Kafka and identify malformed data.
• CI/CD Automation: DevOps engineers use kafka-topics.sh in bash scripts inside Jenkins or GitHub Actions pipelines to automatically provision topics before deploying microservices.
• Performance Testing: Using console producers to stream test payloads into a staging cluster to benchmark consumer application throughput.

Common Mistakes and How to Avoid Them

• Using the --zookeeper Flag: Older tutorials recommend using --zookeeper localhost:2181. This is deprecated and removed in modern Kafka. Always use --bootstrap-server pointing to your brokers.
• Running Reset Offsets on Active Groups: If you try to reset offsets while your application instances are running, the command will fail with a GroupNotEmptyException. Always stop your consumers first.
• Replication Factor Exceeding Broker Count: Attempting to create a topic with --replication-factor 3 on a single-node local cluster will fail. Ensure your replication factor is less than or equal to your active broker count.

Interview Notes & Quick Reference

• Question: What is consumer lag and how do you check it?
• Answer: Consumer lag is the difference between the last produced message offset and the last committed offset by a consumer group. It can be checked using the command: kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group-name>.
• Question: Can you decrease the partitions of an existing topic using CLI?
• Answer: No. You can increase partitions using --alter, but decreasing them is not supported because it would result in data loss and break partition-key routing semantics.
• Question: How do you read messages from a specific partition using CLI?
• Answer: You can use kafka-console-consumer.sh with the --partition flag, but note that you must also specify the --offset flag or omit --group when targeting a single partition.

Summary

Kafka CLI tools are indispensable for any developer working with event-driven architectures. By mastering kafka-topics.sh, kafka-console-producer.sh, kafka-console-consumer.sh, and kafka-consumer-groups.sh, you gain absolute control over your messaging pipelines, enabling seamless debugging, administrative operations, and monitoring.