Duplicate Messages Are Coming from Kafka or RabbitMQ — How Will You Handle Duplicate Processing?
Duplicate message processing is a very common problem in distributed systems and event-driven microservices architecture.
Kafka and RabbitMQ provide:
- At-Least-Once Delivery Guarantee
This means:
A message can be delivered more than once
If duplicate handling is not implemented properly:
- Money may be deducted twice
- Orders may be created multiple times
- Inventory may become inconsistent
- Duplicate notifications may be sent
Real-Time Banking Example
Scenario
Customer Transfers ₹5000
↓
Kafka Publishes Event
↓
Payment Consumer Processes Event
Problem Scenario
Suppose:
- Consumer processed message successfully
- Database updated successfully
- Before offset acknowledgment, consumer crashed
What Happens?
Kafka re-delivers the same message.
Same Payment Event Processed Again
Dangerous Result
₹5000 deducted twice
This is called:
Duplicate Processing
Why Duplicate Messages Occur?
| Reason | Description |
|---|---|
| Consumer Crash | Offset not committed |
| Network Failure | Acknowledgment lost |
| Retries | Same message processed again |
| Broker Rebalancing | Message reassigned to another consumer |
| Producer Retry | Producer sends duplicate event |
Production-Level Solutions
- Idempotency
- Unique Transaction IDs
- Database Constraints
- Deduplication Table
- Exactly Once Semantics
- Distributed Cache
- Event Versioning
- Consumer Transaction Management
Most Important Solution — Idempotency
Idempotency means:
Processing same request multiple times produces same result
Example
If same payment message comes 10 times:
Money should deduct only once
Real Banking Example
Message
Transaction ID = TXN123 Amount = ₹5000
First Processing
TXN123 Processed Successfully
Duplicate Message Comes Again
Before processing:
Check: Was TXN123 already processed?
If Already Processed
Ignore Duplicate Message
Database Table Example
processed_transactions ID | TRANSACTION_ID 1 | TXN123
Idempotency Check Example
public void process(TransactionEvent event) {
boolean exists =
repository.existsByTransactionId(
event.getTransactionId()
);
if(exists) {
return;
}
saveTransaction(event);
}
Database Unique Constraint
Even if multiple consumers process simultaneously, database constraint prevents duplicates.
SQL Example
ALTER TABLE transactions ADD CONSTRAINT unique_txn UNIQUE(transaction_id);
Benefits
- Strong duplicate protection
- Simple implementation
- Production-ready solution
Redis-Based Deduplication
High-scale systems often use Redis for fast duplicate checks.
Flow
Message Received
↓
Check Redis Key
↓
If Exists → Ignore
If Not Exists → Process
Redis Example
String key = "txn:" + transactionId;
Boolean exists =
redisTemplate.hasKey(key);
if(Boolean.TRUE.equals(exists)) {
return;
}
redisTemplate.opsForValue()
.set(key, "PROCESSED");
Benefits
- Very fast
- Works across multiple instances
- Suitable for distributed systems
Kafka Exactly Once Semantics (EOS)
Kafka supports Exactly Once Processing.
Producer Configuration
spring:
kafka:
producer:
properties:
enable.idempotence: true
Transactional Configuration
spring:
kafka:
producer:
transaction-id-prefix: txn-
Benefits
- No duplicate publishing
- No duplicate consumption
- Reliable event processing
Transactional Consumer Example
@Transactional
@KafkaListener(topics = "payment-topic")
public void consume(String message) {
processMessage(message);
}
RabbitMQ Duplicate Handling
RabbitMQ also supports acknowledgments.
Problem Scenario
Consumer Processes Message
↓
Before ACK → Consumer Crashes
↓
RabbitMQ Re-delivers Message
Solution
- Manual acknowledgment
- Idempotent processing
- Deduplication storage
RabbitMQ Manual ACK Example
@RabbitListener(queues = "payment-queue")
public void receive(Message message,
Channel channel) throws Exception {
try {
process(message);
channel.basicAck(
message.getMessageProperties()
.getDeliveryTag(),
false
);
} catch(Exception ex) {
channel.basicNack(
message.getMessageProperties()
.getDeliveryTag(),
false,
true
);
}
}
Deduplication Table Pattern
Maintain a separate processed message table.
Example Table
processed_messages MESSAGE_ID | PROCESSED_AT
Flow
Message Received
↓
Check MESSAGE_ID
↓
Already Exists?
↓
YES → Ignore
NO → Process
Kafka Partitioning
Kafka guarantees ordering within a partition.
Production Recommendation
Use business key as partition key.
Example
kafkaTemplate.send(
"payment-topic",
accountId,
event
);
Benefit
All events for same account go to same partition.
This reduces concurrency conflicts.
Outbox Pattern
Outbox Pattern prevents duplicate event publishing.
Problem Without Outbox
DB Commit Success
↓
Kafka Publish Retry
↓
Duplicate Event Published
Outbox Solution
Store event in database first, then publish safely.
Event Versioning
Version numbers help detect stale or duplicate events.
Example
Account Updated Version 1 Version 2 Version 3
Older versions ignored automatically.
Monitoring Duplicate Processing
Production systems should monitor:
- Duplicate message count
- Retry count
- DLQ messages
- Consumer restarts
- Rebalance events
Monitoring Tools
- Grafana
- Prometheus
- ELK Stack
- Datadog
Real Production Incident
Issue
A wallet application deducted money twice during Kafka consumer restarts.
Root Cause
- No idempotency handling
- Duplicate Kafka message processing
Fixes Applied
- Added transaction ID uniqueness
- Implemented Redis deduplication
- Enabled Kafka idempotent producer
- Added manual acknowledgment
- Introduced processed message table
Final Result
Before: Duplicate money deductions After: No duplicate processing Reliable event handling
Production Best Practices
| Technique | Purpose |
|---|---|
| Idempotency | Prevent duplicate processing |
| Unique Transaction ID | Identify duplicate events |
| DB Unique Constraint | Prevent duplicate inserts |
| Redis Deduplication | Fast duplicate detection |
| Exactly Once Semantics | Reliable Kafka processing |
| Manual ACK | Controlled message acknowledgment |
| Outbox Pattern | Prevent duplicate publishing |
| Monitoring | Track duplicate events |
Final Interview Answer
If duplicate messages are coming from Kafka or RabbitMQ, I would handle duplicate processing using idempotency. Each message should contain a unique transaction ID or event ID. Before processing, the consumer checks whether the message was already processed. If yes, it ignores the duplicate message. I would also enforce database unique constraints, use Redis for fast deduplication checks, and maintain a processed messages table. In Kafka, I would enable Exactly Once Semantics and idempotent producers. Additionally, I would use manual acknowledgments and transactional processing to ensure reliable event handling in production systems.