How Will You Achieve Exactly-Once Processing in Event-Driven Microservices?
Exactly-once processing is one of the most important challenges in event-driven microservices architecture.
Exactly-once means:
Each event must be processed only once without duplication and without data loss
Why Is Exactly-Once Processing Important?
In distributed systems:
- Messages can be duplicated
- Consumers may crash
- Network failures may occur
- Retries may happen
- Offsets may not commit properly
Without proper handling:
- Money may deduct twice
- Orders may create multiple times
- Inventory may reduce incorrectly
- Duplicate notifications may be sent
Real-Time Banking Example
Scenario
Customer Transfers ₹5000
↓
Kafka Event Published
↓
Payment Consumer Processes Event
↓
Database Updated
Problem Scenario
Suppose:
- Consumer updates database successfully
- Before offset commit, consumer crashes
What Happens?
Kafka re-delivers the same message.
Same Payment Processed Again
Dangerous Result
₹5000 deducted twice
Why Exactly-Once Is Difficult?
Because multiple systems are involved:
- Kafka broker
- Consumer application
- Database
- Network
All systems must remain consistent.
Delivery Semantics in Messaging Systems
| Type | Behavior |
|---|---|
| At Most Once | No duplicates but message loss possible |
| At Least Once | No message loss but duplicates possible |
| Exactly Once | No duplicates and no data loss |
Production-Level Techniques for Exactly-Once Processing
- Idempotency
- Kafka Transactions
- Idempotent Producer
- Transactional Consumer
- Manual Offset Commit
- Outbox Pattern
- Deduplication Table
- Unique Constraints
- Saga Pattern
Most Important Technique — Idempotency
Idempotency means:
Processing same event multiple times produces same final result
Real Banking Example
Event
Transaction ID = TXN1001 Amount = ₹5000
First Processing
TXN1001 Processed Successfully
Duplicate Event Comes Again
Before processing:
Check: Was TXN1001 already processed?
If Already Processed
Ignore Event
Idempotency Example
public void process(TransactionEvent event) {
boolean exists =
repository.existsByTransactionId(
event.getTransactionId()
);
if(exists) {
return;
}
saveTransaction(event);
}
Database Unique Constraint
Database-level protection is also important.
SQL Example
ALTER TABLE transactions ADD CONSTRAINT unique_txn UNIQUE(transaction_id);
Kafka Idempotent Producer
Kafka provides idempotent producer support.
Configuration
spring:
kafka:
producer:
properties:
enable.idempotence: true
Benefits
- Prevents duplicate publishing
- Ensures safe retries
- Maintains ordering guarantees
Kafka Transactions
Kafka supports transactional messaging.
Goal
Either:
- All operations succeed
- Or everything rolls back
Configuration
spring:
kafka:
producer:
transaction-id-prefix: txn-
Transactional Producer Example
@Transactional
public void publishEvent() {
kafkaTemplate.send(
"payment-topic",
event
);
}
Transactional Consumer
Consumer processing and offset commit should happen atomically.
Consumer Example
@Transactional
@KafkaListener(topics = "payment-topic")
public void consume(String message) {
saveToDatabase(message);
}
How It Works
If database update fails:
- Offset not committed
- Message reprocessed safely
Manual Offset Commit
Never commit Kafka offsets before successful processing.
Disable Auto Commit
spring:
kafka:
consumer:
enable-auto-commit: false
Manual ACK Example
@KafkaListener(topics = "payment-topic")
public void consume(
String message,
Acknowledgment ack) {
try {
saveToDatabase(message);
ack.acknowledge();
} catch(Exception ex) {
log.error("Processing Failed", ex);
}
}
Outbox Pattern
Outbox Pattern ensures reliable event publishing.
Problem Without Outbox
DB Commit Success
↓
Kafka Publish Failed
Or:
Kafka Published
↓
DB Rollback Happened
Outbox Solution
Save:
- Business data
- Event data
inside same database transaction.
Outbox Table
CREATE TABLE outbox_events (
id BIGINT PRIMARY KEY,
event_type VARCHAR(100),
payload TEXT,
status VARCHAR(20)
);
Transactional Save Example
@Transactional
public void createPayment() {
paymentRepository.save(payment);
outboxRepository.save(event);
}
Outbox Publisher
@Scheduled(fixedDelay = 5000)
public void publishEvents() {
List<OutboxEvent> events =
repository.findPending();
for(OutboxEvent event : events) {
kafkaTemplate.send(
event.getTopic(),
event.getPayload()
);
event.setStatus("COMPLETED");
}
}
Deduplication Table
Maintain processed event IDs.
Example Table
processed_events EVENT_ID | PROCESSED_AT
Flow
Event Received
↓
Check EVENT_ID
↓
Already Exists?
↓
YES → Ignore
NO → Process
Redis-Based Deduplication
High-scale systems use Redis for fast duplicate detection.
Redis Example
String key = "txn:" + transactionId;
Boolean exists =
redisTemplate.hasKey(key);
if(Boolean.TRUE.equals(exists)) {
return;
}
redisTemplate.opsForValue()
.set(key, "PROCESSED");
Kafka Partitioning
Kafka guarantees ordering within a partition.
Production Recommendation
Use business key as partition key.
Example
kafkaTemplate.send(
"payment-topic",
accountId,
event
);
Benefit
All events for same account go to same partition.
This reduces concurrency issues.
Saga Pattern
For distributed transactions across microservices, Saga Pattern ensures consistency.
Example
Order Service
↓
Payment Service
↓
Inventory Service
If failure occurs:
- Compensation transaction executed
Monitoring and Observability
Production systems should monitor:
- Duplicate events
- Retry counts
- DLQ messages
- Consumer lag
- Offset commit failures
Monitoring Tools
- Grafana
- Prometheus
- ELK Stack
- Datadog
- Splunk
Real Production Incident
Issue
A wallet system processed duplicate Kafka messages during consumer restart.
Impact
- Duplicate money deduction
- Financial inconsistency
Root Cause
- No idempotency
- Auto offset commit enabled
- No deduplication logic
Fixes Applied
- Implemented idempotency
- Enabled Kafka transactions
- Added outbox pattern
- Used manual acknowledgment
- Added Redis deduplication
- Configured unique DB constraints
Final Result
Before: Duplicate transaction processing After: Reliable exactly-once processing No duplicate deductions
Production Best Practices
| Technique | Purpose |
|---|---|
| Idempotency | Prevent duplicate processing |
| Kafka Transactions | Atomic processing |
| Idempotent Producer | Prevent duplicate publishing |
| Manual Offset Commit | Commit after successful processing |
| Outbox Pattern | Reliable event publishing |
| Unique Constraints | Prevent duplicate inserts |
| Redis Deduplication | Fast duplicate detection |
| Saga Pattern | Distributed consistency |
Final Interview Answer
To achieve exactly-once processing in event-driven microservices, I would combine multiple techniques such as idempotency, Kafka transactions, idempotent producers, manual offset commits, and the Outbox Pattern. Every event should contain a unique transaction ID, and consumers should verify whether the event was already processed before executing business logic. I would disable auto offset commit and acknowledge messages only after successful database transactions. Additionally, I would use database unique constraints, Redis-based deduplication, and Kafka Exactly Once Semantics to prevent duplicate processing and ensure reliable event handling in production systems.