How Will You Rollback a Failed Microservice Deployment in Production?
Rollback means restoring the previous stable version when a new deployment causes failures in production.
Why Rollback Is Important?
Even after testing, production deployments may fail because of:
- Application bugs
- Memory leaks
- Database migration issues
- API incompatibility
- Configuration problems
- Traffic spikes
- Third-party failures
- Unexpected production behavior
Main Goal
Restore Stable Service Quickly With Minimal Customer Impact
Real Production Example
Suppose:
Payment Service v2 Deployed
After Deployment
Payment Failures Increase Latency Becomes High CPU Usage Spikes Orders Start Failing
Immediate Requirement
Rollback To Previous Stable Version
Production Rollback Strategies
- Kubernetes Rollback
- Blue-Green Rollback
- Canary Rollback
- Feature Flag Rollback
- Traffic Routing Rollback
- Database Rollback Strategy
- GitOps Rollback
- Immutable Deployments
- CI/CD Automated Rollback
- Observability-Based Rollback
- Service Mesh Rollback
- Configuration Rollback
1. Kubernetes Rollback
Most modern microservices run on Kubernetes.
Deployment Flow
Deploy v2
↓
Monitor Production
↓
Errors Detected
↓
Rollback To Previous ReplicaSet
Kubernetes Rollback Command
kubectl rollout undo deployment payment-service
How Kubernetes Works Internally
Old ReplicaSet Preserved New ReplicaSet Created
Rollback Simply Switches Back
v2 → v1
Benefits
- Fast rollback
- Simple execution
- Minimal downtime
View Rollout History
kubectl rollout history deployment payment-service
Rollback To Specific Revision
kubectl rollout undo deployment payment-service --to-revision=2
2. Blue-Green Rollback
One of the safest rollback strategies.
Architecture
Blue → Current Stable Version Green → New Version
Deployment Flow
Deploy v2 To Green
↓
Switch Traffic To Green
↓
Errors Detected
↓
Switch Traffic Back To Blue
Benefits
- Instant rollback
- Very low risk
- Near-zero downtime
Rollback Time
Few Seconds
Production Example
Blue → Payment Service v1 Green → Payment Service v2
Failure Detected
Load Balancer Routes Back To Blue
Benefits
- No redeployment required
- Fast recovery
3. Canary Rollback
Rollback only affects small percentage initially.
Canary Deployment Example
95% Traffic → v1 5% Traffic → v2
If v2 Fails
Stop Canary All Traffic → v1
Benefits
- Reduced blast radius
- Safer deployments
- Early issue detection
Production Example
New Payment Logic Causes Errors
Only 5% Users Impacted
Rollback immediately:
100% Traffic → Stable Version
4. Feature Flag Rollback
One of the fastest rollback methods.
Idea
Disable Feature Without Redeployment
Example
if(featureFlagEnabled) {
newPaymentFlow();
}
else {
oldPaymentFlow();
}
If Problem Occurs
Turn Off Feature Flag
Benefits
- Instant rollback
- No redeployment needed
- Very low risk
Popular Tools
- :contentReference[oaicite:0]{index=0}
- :contentReference[oaicite:1]{index=1}
5. Load Balancer Traffic Rollback
Traffic routing controlled centrally.
Architecture
Users ↓ Load Balancer ↓ Service Versions
Rollback Flow
Traffic To v2 Causes Errors
↓
Load Balancer Routes Back To v1
Benefits
- Fast traffic switching
- No DNS delay
- Controlled rollback
Popular Load Balancers
- :contentReference[oaicite:2]{index=2}
- :contentReference[oaicite:3]{index=3}
- :contentReference[oaicite:4]{index=4}
6. Service Mesh Rollback
Modern microservices often use service mesh.
Popular Tools
- :contentReference[oaicite:5]{index=5}
- :contentReference[oaicite:6]{index=6}
Benefits
- Traffic splitting
- Canary rollback
- Advanced traffic control
- Observability
Traffic Example
90% → v1 10% → v2
Rollback
100% → v1
7. Database Rollback Strategy
Database rollback is the most dangerous part.
Problem
App Rolled Back But Database Schema Changed
Result
Old App Cannot Work
Production Solution
Backward-Compatible Database Changes
Best Practice
Expand And Contract Pattern
Correct Flow
Step 1: Add New Columns Step 2: Deploy New App Step 3: Migrate Data Slowly Step 4: Remove Old Columns Later
Benefits
- Supports rollback safely
- Old and new versions coexist
Wrong Approach
Drop Existing Column Immediately
Problem
Rollback Impossible
8. GitOps Rollback
Infrastructure and deployments managed through Git.
Popular Tools
- :contentReference[oaicite:7]{index=7}
- :contentReference[oaicite:8]{index=8}
Rollback Flow
Revert Git Commit
↓
GitOps Sync
↓
Previous Stable Version Restored
Benefits
- Version-controlled rollback
- Auditability
- Consistent deployments
9. Immutable Deployments
Never modify running servers directly.
Idea
Deploy New Image Instead Of Updating Existing One
Rollback
Redeploy Previous Image
Benefits
- Predictable rollback
- Environment consistency
Docker Example
payment-service:v1 payment-service:v2
Rollback
Deploy payment-service:v1
10. Automated Rollback Through Monitoring
Modern systems rollback automatically.
Monitor Metrics
- Error rate
- Latency
- CPU usage
- Memory usage
- HTTP 5xx errors
- Kafka consumer lag
Monitoring Tools
- :contentReference[oaicite:9]{index=9}
- :contentReference[oaicite:10]{index=10}
- :contentReference[oaicite:11]{index=11}
Automated Flow
Deployment Starts
↓
Metrics Monitored
↓
Threshold Breached
↓
Automatic Rollback Triggered
Benefits
- Faster recovery
- Reduced manual intervention
- Less customer impact
11. CI/CD-Based Rollback
CI/CD pipelines can automate rollback.
Pipeline Flow
Deploy ↓ Smoke Tests ↓ Health Checks ↓ Error Monitoring ↓ If Failed ↓ Rollback Automatically
Popular CI/CD Tools
- :contentReference[oaicite:12]{index=12}
- :contentReference[oaicite:13]{index=13}
- :contentReference[oaicite:14]{index=14}
12. Health Checks Before Traffic Routing
Traffic should only reach healthy instances.
Kubernetes Readiness Probe
readinessProbe:
httpGet:
path: /health
port: 8080
Flow
New Pod Starts
↓
Health Check Passes
↓
Traffic Allowed
If Health Check Fails
Pod Removed From Load Balancer
Benefits
- Prevent unhealthy traffic routing
- Safer deployments
13. Graceful Shutdown During Rollback
Old pods should not terminate immediately.
Flow
Stop Accepting New Requests
↓
Complete Existing Requests
↓
Shutdown Safely
Benefits
- No request loss
- Better user experience
Spring Boot Example
server.shutdown=graceful
14. Real Production Incident
Scenario
New Payment Service deployment introduced memory leak.
Symptoms
- CPU usage increased
- Latency became high
- Pods restarting
- Payments failing
Production Rollback Flow
Canary Deployment Detected Errors
↓
Traffic Immediately Shifted Back
↓
Rollback Triggered
↓
Previous Stable Image Restored
Database Safety
Backward-compatible migrations prevented DB rollback issues.
Final Result
- Recovery in few minutes
- Minimal customer impact
- No data corruption
Production Best Practices
| Practice | Purpose |
|---|---|
| Canary Deployment | Reduce blast radius |
| Blue-Green Deployment | Instant rollback |
| Feature Flags | Fast rollback |
| Kubernetes Rollback | Simple restoration |
| Monitoring | Detect failures quickly |
| Graceful Shutdown | Prevent request loss |
| Backward-Compatible DB | Safe rollback support |
| GitOps | Version-controlled rollback |
Final Interview Answer
To rollback a failed microservice deployment in production, I would first detect the issue quickly using monitoring and observability tools such as :contentReference[oaicite:15]{index=15}, :contentReference[oaicite:16]{index=16}, and distributed tracing systems. Depending on the deployment strategy, I would use Kubernetes rollback commands, blue-green rollback, or canary rollback to restore the previous stable version with minimal customer impact. In Kubernetes, rollback can be performed using ReplicaSet history, while blue-green deployment allows instant traffic switching back to the stable environment. In canary deployments, only a small percentage of traffic is exposed initially, reducing blast radius. I would also use feature flags for instant rollback without redeployment, and service mesh tools like :contentReference[oaicite:17]{index=17} for advanced traffic control. Database changes would follow backward-compatible migration strategies using the expand-and-contract pattern to ensure rollback safety. Additionally, CI/CD pipelines can automate rollback based on health checks and production metrics. The overall goal is to restore stability quickly, avoid downtime, minimize customer impact, and ensure safe recovery in production microservices environments.