Database of One Microservice Goes Down — Will the Entire System Fail? How Will You Handle It?
In microservices architecture, every microservice usually has its own separate database.
Example Architecture
User Service → User DB Order Service → Order DB Payment Service → Payment DB Notification Service → Notification DB
Important Principle
Failure of one database should NOT bring down the entire system.
Why?
Because microservices are:
- Loosely coupled
- Independently deployable
- Independently scalable
- Failure isolated
Monolithic Architecture Problem
Single Application
↓
Single Shared Database
If DB Fails
Entire Application Down
Microservices Advantage
Independent Services Independent Databases
Real-Time Banking Example
Customer Performs UPI Transfer
↓
Transaction Service
↓
Payment Service
↓
Payment DB Down
Wrong System Behavior
Entire Banking System Crashes
Correct Production Behavior
| Feature | Status |
|---|---|
| Payment Transfer | Temporarily Unavailable |
| Balance Check | Working |
| Login | Working |
| Mini Statement | Working |
| Notifications | Working |
Main Goal
Isolate Failure Prevent Cascading Failure Maintain Partial Availability
Production-Level Techniques to Handle DB Failure
- Database Per Service Pattern
- Circuit Breaker
- Retries with Backoff
- Fallback Mechanism
- Bulkhead Pattern
- Graceful Degradation
- Event-Driven Architecture
- Database Replication
- Failover Strategy
- Redis Cache
- Monitoring & Alerts
Step 1: Database Per Service Pattern
Each service should own its own database.
Correct Architecture
Order Service → Order DB Payment Service → Payment DB User Service → User DB
Benefits
- Failure isolation
- Independent scaling
- Technology flexibility
- Reduced coupling
Step 2: Use Circuit Breaker Pattern
Circuit breaker prevents continuous calls to failed services.
Problem Without Circuit Breaker
Order Service
↓
Payment Service
↓
Payment DB Down
↓
Continuous Retry
↓
Thread Exhaustion
↓
Entire System Slow
Solution
Circuit Opens
↓
Stop Calling Failed Service
Spring Boot Resilience4j Example
@CircuitBreaker(
name = "paymentService",
fallbackMethod = "fallbackMethod"
)
public String processPayment() {
return paymentClient.pay();
}
Fallback Example
public String fallbackMethod(Exception ex) {
return "Payment Service Temporarily Unavailable";
}
Benefits
- Prevents cascading failures
- Protects resources
- Improves resilience
Step 3: Graceful Degradation
Only impacted functionality should fail.
Example
Payment Feature Down Other Features Continue
Benefits
- Better customer experience
- Reduced business impact
- Partial availability maintained
Step 4: Retry Temporary Failures
Some DB issues are temporary.
Examples
- Short network issue
- Temporary DB overload
- Transient connectivity problem
Retry Example
@Retry(
name = "paymentRetry",
maxAttempts = 3
)
Best Practices
- Limited retries
- Exponential backoff
- Combine with circuit breaker
Problem With Unlimited Retry
Retry Storm
↓
Database Completely Crashes
Step 5: Use Bulkhead Pattern
Isolate resources between services.
Problem Without Bulkhead
Payment DB Failure
↓
Threads Blocked
↓
Entire Application Slow
Solution
Separate Thread Pools Separate Resource Allocation
Benefits
- Failure isolation
- Resource protection
- Improved stability
Step 6: Use Event-Driven Architecture
Avoid tight synchronous dependency between services.
Synchronous Problem
Order Service
↓
Payment Service
↓
Immediate Failure
Better Asynchronous Design
Order Created Event
↓
Kafka/RabbitMQ
↓
Payment Service Processes Later
Benefits
- Loose coupling
- Asynchronous recovery
- Improved resilience
Kafka Example
Topic: payment-processing-topic
If Payment DB Down
Message Remains in Kafka Consumer Retries Later
Benefits
- No data loss
- Automatic recovery
- Reliable processing
Step 7: Database Replication and Failover
Production databases should have replicas.
Architecture
Primary DB
↓
Replica DB
If Primary DB Fails
Automatic Failover to Replica
Popular Solutions
- MySQL Replication
- PostgreSQL Streaming Replication
- MongoDB Replica Set
- AWS Aurora Multi-AZ
Benefits
- High availability
- Reduced downtime
- Automatic recovery
Step 8: Use Redis Cache
Cache can temporarily serve read requests.
Example
User Request
↓
Redis Cache
↓
Return Cached Data
Benefits
- Reduced DB dependency
- Improved performance
- Temporary fault tolerance
Step 9: Health Checks and Kubernetes Recovery
Kubernetes should automatically detect unhealthy services.
Spring Boot Health Endpoint
/actuator/health
Kubernetes Flow
Service Unhealthy
↓
Pod Restarted Automatically
Benefits
- Automatic healing
- Reduced downtime
Step 10: Monitoring and Alerts
Production systems should detect failures immediately.
Monitor
- DB availability
- Connection pool usage
- Slow queries
- Error rate
- Replication lag
Monitoring Tools
- :contentReference[oaicite:0]{index=0}
- :contentReference[oaicite:1]{index=1}
- :contentReference[oaicite:2]{index=2}
- :contentReference[oaicite:3]{index=3}
Real Production Incident
Issue
Payment database became unavailable because of disk corruption.
Impact
- Payment transactions delayed
- Money transfer feature affected
- Other banking services continued working
Why Entire System Did Not Fail?
- Independent databases
- Circuit breakers enabled
- Kafka-based asynchronous processing
- Graceful degradation implemented
- Separate thread pools used
Fixes Applied
- Replica DB promoted automatically
- Pending Kafka events reprocessed
- Redis cache served temporary reads
- Alerts triggered immediately
Final Result
Before: Single DB issue risked platform outage After: Only impacted service degraded Entire system remained operational
Production Best Practices
| Technique | Purpose |
|---|---|
| Database Per Service | Failure isolation |
| Circuit Breaker | Prevent cascading failure |
| Retries | Handle temporary failures |
| Bulkhead Pattern | Resource isolation |
| Event-Driven Architecture | Loose coupling |
| Replication | High availability |
| Redis Cache | Temporary read support |
| Monitoring | Early failure detection |
Final Interview Answer
No, the entire system should not fail if one microservice database goes down. In a properly designed microservices architecture, each service owns its own database, which isolates failures to a specific domain. If one database becomes unavailable, only the dependent microservice should be impacted while other services continue functioning normally. To handle this scenario, I would implement circuit breakers to prevent cascading failures, retries with exponential backoff for temporary issues, graceful degradation to maintain partial functionality, and bulkhead patterns for resource isolation. I would also use event-driven communication with Kafka or RabbitMQ to decouple services, database replication and automatic failover for high availability, Redis caching for temporary read operations, and monitoring tools like :contentReference[oaicite:4]{index=4} and :contentReference[oaicite:5]{index=5} for proactive detection and recovery in production systems.