Installing and Configuring Apache Kafka: A Step-by-Step Guide

Last Updated: May 28, 2026

This guide explains how to install, configure, and run Apache Kafka using both ZooKeeper mode and modern KRaft mode with practical CLI examples.

Setting up Apache Kafka correctly is the foundation of building high-throughput, fault-tolerant event streaming pipelines. Whether you are setting up a local development environment or preparing a production-ready cluster, understanding the installation steps and core configuration files is essential.

In this guide, we will walk through the process of installing Apache Kafka on your local machine, configuring it using both the traditional ZooKeeper mode and the modern KRaft (Kafka Raft) mode, and verifying the installation with basic command-line tools.

If you are new to Kafka architecture and event streaming concepts, start with Introduction to Event Streaming and Apache Kafka .

You should also understand Kafka internals before installation by reading Understanding Apache Kafka Architecture and Core Concepts .

Table of Contents

• Prerequisites for Apache Kafka
• Apache Kafka Installation Flow
• Installing Java JDK
• Downloading Apache Kafka
• Kafka Directory Structure
• ZooKeeper vs KRaft Setup
• Verifying the Installation
• Troubleshooting Common Errors
• Summary

Prerequisites for Apache Kafka

Before installing Apache Kafka, your system must meet certain requirements. Since Apache Kafka is built using Java and Scala, the primary prerequisite is the Java Runtime Environment (JRE) or Java Development Kit (JDK).

• Java Development Kit (JDK): Kafka requires JDK 8, 11, or 17. JDK 11 or 17 is highly recommended for modern Kafka versions.
• Operating System: Linux and macOS are natively supported. Windows is supported via Windows Subsystem for Linux (WSL) or using specific batch scripts, though Linux-based environments are standard for production.
• Disk Space and Memory: A minimum of 4GB RAM is recommended for local development.

Apache Kafka Installation Flow

The following diagram illustrates the high-level steps required to get Apache Kafka up and running on your system:

+------------------------------------------+
|       Install Java JDK (8, 11, or 17)    |
+------------------------------------------+
                     │
                     ▼
+------------------------------------------+
|     Download & Extract Apache Kafka      |
+------------------------------------------+
                     │
                     ▼
+------------------------------------------+
|   Choose Mode: ZooKeeper or KRaft Mode   |
+------------------------------------------+
                     │
                     ▼
+------------------------------------------+
|      Configure Property Files            |
|  (server.properties / broker.properties) |
+------------------------------------------+
                     │
                     ▼
+------------------------------------------+
|    Start Metadata Controller & Broker   |
+------------------------------------------+
                     │
                     ▼
+------------------------------------------+
|    Verify Setup (Create Topic & Test)    |
+------------------------------------------+

Step 1: Installing Java JDK

First, verify if Java is already installed on your system. Open your terminal and run the following command:

java -version

If Java is not installed, or if the version is older than Java 8, download and install OpenJDK from your system package manager. For example, on Ubuntu/Debian systems:

sudo apt update
sudo apt install default-jdk

On macOS, you can install it using Homebrew:

brew install openjdk@17

Step 2: Downloading and Extracting Apache Kafka

Navigate to the official Apache Kafka downloads page. Always download the binary recommended release. Ensure you download the binary version (which contains pre-compiled Scala classes) and not the source code version.

To download Kafka via the command line, use wget or curl. Replace the version number with the latest stable release:

wget https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz

Once downloaded, extract the archive using the tar command:

tar -xzf kafka_2.13-3.6.1.tgz

Move the extracted directory to a convenient location, such as /usr/local/kafka or your user home directory, and navigate into it:

mv kafka_2.13-3.6.1 ~/kafka
cd ~/kafka

Step 3: Understanding the Kafka Directory Structure

Once extracted, you will see several directories inside the Kafka home folder. Understanding these directories helps during configuration and troubleshooting:

• bin/: Contains all the shell scripts required to start, stop, and manage Kafka brokers, topics, consumers, and producers. Windows-specific batch scripts are located in bin/windows/.
• config/: Contains configuration files for Kafka brokers, ZooKeeper, consumers, producers, and connect workers.
• libs/: Contains all the JAR dependencies required by Kafka to run.
• licenses/: Contains license agreements for the software components.

Step 4: Choosing Your Architecture Mode

Historically, Apache Kafka relied on Apache ZooKeeper to manage cluster metadata, coordinate brokers, and elect leaders. However, modern versions of Kafka introduce KRaft (Kafka Raft) mode, which removes the dependency on ZooKeeper, allowing Kafka to manage its own metadata.

In this guide, we cover both methods. KRaft is recommended for new installations, while ZooKeeper remains highly relevant for legacy systems.

Option A: Configuring and Running Kafka with KRaft (Modern Method)

KRaft simplifies Kafka administration by consolidating metadata management directly inside the Kafka cluster. This eliminates the need to run a separate ZooKeeper process.

1. Configure KRaft Properties: Open the KRaft configuration file located at config/kraft/reformat.sh or config/kraft/server.properties. Review the default configurations. Key properties include:

• process.roles=broker,controller: Defines that this node acts as both a storage broker and a metadata controller.
• node.id=1: A unique identifier for the node within the cluster.
• controller.quorum.voters=1@localhost:9093: Defines the controller nodes used for voting and consensus.
• log.dirs=/tmp/kraft-combined-logs: The directory where Kafka event logs and metadata are stored.

2. Generate a Cluster UUID: KRaft requires a unique identifier for the cluster. Generate one using the formatting tool:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

3. Format the Log Directory: Format the storage directory using the generated UUID:

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

4. Start the Kafka Server: Start the broker in KRaft mode:

bin/kafka-server-start.sh config/kraft/server.properties

Option B: Configuring and Running Kafka with ZooKeeper (Legacy Method)

If your architecture requires ZooKeeper, you must run ZooKeeper first, followed by the Kafka broker.

1. Configure ZooKeeper: The default configuration file is config/zookeeper.properties. The default port is 2181, and data is stored in /tmp/zookeeper.

2. Start ZooKeeper: Open a terminal window and run:

bin/zookeeper-server-start.sh config/zookeeper.properties

3. Configure Kafka Broker: Open config/server.properties. Ensure the following properties are set correctly:

• broker.id=0: Unique identifier for the broker.
• log.dirs=/tmp/kafka-logs: Directory where the actual message logs are stored.
• zookeeper.connect=localhost:2181: Connection string pointing to your running ZooKeeper instance.

4. Start the Kafka Broker: Open a new terminal window, navigate to your Kafka directory, and start the broker:

bin/kafka-server-start.sh config/server.properties

Step 5: Verifying the Installation

To verify that your Kafka broker is functioning correctly, you can create a test topic, publish a message using the command-line producer, and read it back using the command-line consumer.

1. Create a Topic

Open a new terminal window and run the following command to create a topic named test-events with a single partition and replication factor of 1:

bin/kafka-topics.sh --create --topic test-events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

To understand topics, partitions, and replication in depth, explore Working with Kafka Topics and Partitions .

2. Start a Console Producer

Run the console producer tool to send messages to your newly created topic. Type a few messages and press Enter after each one:

bin/kafka-console-producer.sh --topic test-events --bootstrap-server localhost:9092
> Hello Kafka
> This is my first event message
> Exit with Ctrl+C

Next, learn how producers work internally in Understanding Kafka Producers and Sending Messages .

3. Start a Console Consumer

Open another terminal window and start the console consumer to read the messages from the beginning of the topic:

bin/kafka-console-consumer.sh --topic test-events --from-beginning --bootstrap-server localhost:9092

You should see the messages you typed in the producer console appear in real-time in the consumer console.

You can also learn how consumers manage offsets and process events in Understanding Kafka Consumers and Reading Messages .

Common Mistakes and Troubleshooting

Setting up Kafka for the first time can lead to a few common configuration errors. Here is how to resolve them:

• Address Already in Use (BindException): This happens if port 9092 (Kafka) or 2181 (ZooKeeper) is already being used by another process. Identify and kill the process using lsof -i :9092 or change the ports in the respective properties files.
• Out of Memory Error (Java Heap Space): By default, Kafka allocates 1GB of heap memory. If your local machine is constrained, you can temporarily lower the memory limit before starting Kafka by setting the environment variable: export KAFKA_HEAP_OPTS="-Xmx512M -Xms256M".
• KRaft Directory Not Formatted: If you get an error stating KafkaStorageException: The directory is not formatted, it means you skipped the kafka-storage.sh format step required for KRaft mode.
• Windows Path Limits: When running Kafka on Windows, long directory paths can cause script crashes. Keep your Kafka installation folder close to the root drive (e.g., C:\kafka) and run commands using the bin\windows\ directory scripts.

Real-World Use Cases

Understanding installation and configuration is crucial for several real-world scenarios:

• Local Development Environments: Developers run single-node Kafka brokers locally (often via Docker or manual installation) to write and test event-driven microservices before deploying them to staging environments.
• Production Cluster Topologies: System administrators configure multiple brokers across different physical racks or availability zones. They modify server.properties to define custom replication factors, partition strategies, and security protocols (SSL/SASL).

Interview Notes

• What is the default port for Apache Kafka and Apache ZooKeeper? Kafka defaults to port 9092, and ZooKeeper defaults to port 2181.
• Why is Apache Kafka moving away from ZooKeeper (KRaft)? ZooKeeper introduces scalability bottlenecks, metadata synchronization delays, and operational complexity. KRaft handles metadata directly inside Kafka, allowing clusters to scale to millions of partitions with faster controller failover times.
• What is the purpose of the log.dirs configuration? It specifies the directory paths where Kafka persists its commit log files. It can accept a comma-separated list of directories to distribute data across multiple physical disks.
• How do you increase the partition count of an existing topic? You can use the kafka-topics.sh --alter command. Note that while you can increase partitions, you cannot easily decrease them without recreating the topic.

Who Should Learn Kafka Installation?

• Backend Developers
• DevOps Engineers
• Cloud Engineers
• Microservices Developers
• Platform Engineers
• Site Reliability Engineers (SREs)

Frequently Asked Questions

Does Kafka require ZooKeeper?

Older Kafka versions require ZooKeeper, but modern Kafka versions use KRaft mode which removes the ZooKeeper dependency.

What is the default Kafka port?

Kafka brokers use port 9092 by default, while ZooKeeper uses port 2181.

Can Kafka run on Windows?

Yes. Kafka can run on Windows using batch scripts or Windows Subsystem for Linux (WSL), though Linux environments are preferred for production systems.

Why must KRaft storage be formatted?

KRaft requires formatted metadata storage directories to initialize cluster metadata and controller quorum information.

Next Step

Now that your Kafka environment is ready, continue learning essential Kafka administration commands in Kafka CLI Tools and Essential Commands .

Continue Learning Apache Kafka

• Kafka CLI Tools and Essential Commands
• Working with Kafka Topics and Partitions
• Kafka Security, Authentication, Authorization, and Encryption

Summary

In this guide, we successfully downloaded, installed, and configured Apache Kafka. We explored the differences between legacy ZooKeeper configurations and modern KRaft-based installations. Finally, we verified our environment by producing and consuming messages through the command-line interface. In the next topic, we will dive deep into the essential Kafka CLI tools to manage topics, producers, and consumers like a pro.

About the Author

This Apache Kafka tutorial is designed for developers and engineers who want practical hands-on understanding of Kafka installation, broker configuration, and event streaming systems used in enterprise applications.