Understanding Kafka Producers: Sending Messages

Last Updated: May 28, 2026

Learn how Apache Kafka Producers work internally, how messages are serialized and routed to partitions, how acknowledgments and retries ensure reliability, and how enterprise systems publish millions of events efficiently using Kafka producers.

Before learning producers, you should first understand Kafka architecture and topic partitioning concepts:

• Understanding Apache Kafka Architecture and Core Concepts
• Working with Kafka Topics and Partitions

In Apache Kafka, producers are responsible for publishing events into Kafka topics. Every event-driven system starts with producers generating and sending data into the streaming platform. Whether you are processing financial transactions, IoT telemetry, application logs, or real-time analytics streams, producers act as the entry point into the Kafka ecosystem.

Kafka producers are designed for extremely high throughput, low latency, batching efficiency, asynchronous communication, and fault-tolerant delivery. Understanding how producers work internally is essential for building scalable and reliable distributed systems.

Table of Contents

• What is a Kafka Producer?
• Kafka Producer Internal Workflow
• Serialization Process
• Message Routing and Partitioning
• Message Delivery Patterns
• Understanding Acknowledgments (acks)
• Batching and Performance Optimization
• Idempotent Producers
• Java Producer Example
• Real-World Use Cases
• Common Mistakes
• Interview Questions
• Summary

What is a Kafka Producer?

A Kafka Producer is a client application responsible for publishing records to Kafka topics. Producers communicate directly with Kafka brokers and continuously stream data into the Kafka cluster.

Each produced message contains:

• Topic name
• Optional partition number
• Optional message key
• Message value (payload)
• Timestamp
• Headers (optional metadata)

Producers are optimized for:

• High-throughput event streaming
• Asynchronous non-blocking communication
• Efficient batching
• Network optimization
• Reliable delivery guarantees
• Partition-aware message routing

Kafka Producer Internal Workflow

When an application sends a message using KafkaProducer.send(), the message does not immediately travel to the broker. Instead, it goes through multiple optimized internal stages designed for performance and reliability.

+--------------------------------------------------------------------------------+
|                                KAFKA PRODUCER                                  |
|                                                                                |
|  [ProducerRecord]                                                               |
|        |                                                                       |
|        v                                                                       |
|  +--------------+     +--------------+     +--------------+                    |
|  | Serializer   | --> | Partitioner  | --> | Accumulator  |                    |
|  +--------------+     +--------------+     +--------------+                    |
|                                                     |                          |
|                                                     v   (Batches per partition)                        |
|                                              +--------------+                  |
|                                              | Sender Thread|                  |
|                                              +--------------+                  |
+--------------------------------------------------------------------------------+
                                                      |
                                                      v     (Network I/O)
                                             +-------------+
                                             | Kafka Broker|
                                             +-------------+
    

1. ProducerRecord Creation

The producer first creates a ProducerRecord object containing the topic name, optional key, and message payload.

2. Serialization

Kafka only understands raw bytes. The producer converts Java objects into byte arrays using serializers.

Common serializers include:

• StringSerializer
• ByteArraySerializer
• IntegerSerializer
• JSON serializers
• Avro serializers
• Protobuf serializers

3. Partition Selection

The partitioner determines which partition receives the message.

If a key exists:

hash(key) % number_of_partitions

This guarantees ordering for messages sharing the same key.

If no key exists, Kafka uses Sticky Partitioning to maximize batching efficiency.

4. Record Accumulator

Instead of sending messages immediately, Kafka temporarily stores records inside memory buffers grouped by partition. This component is called the Record Accumulator.

Batching significantly improves:

• Network efficiency
• Broker throughput
• Compression ratio
• CPU utilization

5. Sender Thread

A dedicated background sender thread continuously pulls batches from the accumulator and transmits them to Kafka brokers using TCP socket communication.

Serialization Process

Serialization converts application objects into byte arrays before transmission.

Without serialization, Kafka cannot persist or transfer data.

Example:

String message = "Order Created";

Gets converted into:

[79, 114, 100, 101, 114, ...]

Enterprise systems commonly use:

• JSON: Human-readable but larger payload size
• Avro: Compact binary format with schema evolution support
• Protocol Buffers: High-performance compact serialization
• Thrift: Efficient RPC serialization format

Message Routing and Partitioning

Partitions are critical for scalability and ordering guarantees.

Key-Based Routing

Using keys ensures related events always land in the same partition.

Example:

Key = customer-101

All records for customer-101 always route to the same partition.

This guarantees event ordering for:

• Bank account transactions
• Order lifecycle events
• User activity streams
• IoT sensor data

Sticky Partitioning

When no key is provided, Kafka batches records into a single partition temporarily before switching. This sticky partitioning strategy dramatically improves batching efficiency and throughput.

Message Delivery Patterns

1. Fire-and-Forget

The producer sends records without waiting for acknowledgment.

Advantages:

• Maximum throughput
• Minimal latency

Disadvantages:

• Possible message loss
• No failure visibility

2. Synchronous Send

The application blocks until the broker confirms delivery.

producer.send(record).get();

Advantages:

• Strong delivery confirmation

Disadvantages:

• Poor scalability
• Higher latency

3. Asynchronous Send

The producer immediately continues processing while Kafka handles delivery in the background.

Callbacks handle success or failure notifications asynchronously.

This is the recommended production strategy.

Understanding Acknowledgments (acks)

The acks configuration controls durability guarantees.

acks=0

Producer does not wait for any acknowledgment.

Fastest but least reliable.

acks=1

Producer waits for leader acknowledgment only.

Moderate reliability and performance.

acks=all

Producer waits until all In-Sync Replicas acknowledge the record.

Strongest durability guarantee.

Recommended for financial systems, banking applications, and critical enterprise workloads.

Batching and Performance Optimization

Kafka producers achieve high throughput primarily through batching.

Important batching configurations:

• batch.size — Maximum batch size in bytes
• linger.ms — Time to wait before sending partial batches
• compression.type — Compression algorithm
• buffer.memory — Total producer memory buffer

Compression Types

• gzip
• snappy
• lz4
• zstd

Compression improves:

• Network utilization
• Disk storage efficiency
• Broker throughput

Idempotent Producers

Network failures may cause duplicate message retries.

Kafka solves this problem using idempotent producers.

enable.idempotence=true

Idempotence guarantees:

• No duplicate writes during retries
• Exactly-once producer semantics
• Ordered retry handling

Kafka internally assigns:

• Producer IDs
• Sequence numbers
• Broker-side duplicate detection

Production-Ready Kafka Producer Example

package com.example.kafka;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class OrderProducer {

    public static void main(String[] args) {

        Properties properties = new Properties();

        properties.put(
            ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
            "localhost:9092"
        );

        properties.put(
            ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
            StringSerializer.class.getName()
        );

        properties.put(
            ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
            StringSerializer.class.getName()
        );

        properties.put(
            ProducerConfig.ACKS_CONFIG,
            "all"
        );

        properties.put(
            ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,
            "true"
        );

        KafkaProducer<String, String> producer =
                new KafkaProducer<>(properties);

        ProducerRecord<String, String> record =
                new ProducerRecord<>(
                        "order-events",
                        "customer-101",
                        "Order Created"
                );

        producer.send(record, (metadata, exception) -> {

            if (exception == null) {

                System.out.println("Message Sent Successfully");

                System.out.println(
                        "Partition: " + metadata.partition()
                );

                System.out.println(
                        "Offset: " + metadata.offset()
                );

            } else {

                System.err.println(
                        "Producer Error: " + exception.getMessage()
                );
            }
        });

        producer.flush();
        producer.close();
    }
}

Real-World Use Cases

E-Commerce Platforms

Order lifecycle events are streamed into Kafka for inventory management, payments, notifications, and analytics pipelines.

Banking Systems

Transaction events require strict ordering and strong durability guarantees using keys and idempotent producers.

IoT Sensor Streaming

Millions of devices continuously stream telemetry data into Kafka topics for real-time analytics.

Centralized Logging

Microservices asynchronously publish logs into Kafka clusters for monitoring and observability systems.

Common Mistakes

• Ignoring producer callbacks
• Using synchronous sends in high-volume systems
• Not enabling retries
• Not enabling idempotence
• Hardcoding partition numbers
• Not closing producers properly
• Using oversized messages
• Using too many small batches

Interview Questions

• How does Kafka guarantee ordering?
  Kafka guarantees ordering only within a partition.
• What is the purpose of acks configuration?
  It controls delivery durability guarantees.
• What is idempotent producer?
  A producer configuration preventing duplicate messages during retries.
• What is sticky partitioning?
  A batching optimization strategy for messages without keys.
• Why are callbacks important?
  Callbacks help applications detect delivery success or failures asynchronously.

Summary

Kafka producers are high-performance distributed messaging clients designed for scalable event ingestion. By understanding serialization, batching, acknowledgments, partition routing, retries, idempotence, and asynchronous delivery models, developers can build highly reliable real-time streaming systems capable of handling millions of events per second.

Next Step

Continue learning how Kafka consumers read and process messages:

Understanding Kafka Consumers and Reading Messages