One Microservice Consumes Excessive Memory and CPU. How Will You Identify and Resolve the Issue?
High memory and CPU usage is one of the most common production problems in microservices architecture.
Problem Scenario
Suppose:
Payment Service
starts consuming:
CPU = 95% Memory = 90%
Production Symptoms
- Slow API responses
- High latency
- Pod restarts
- OutOfMemoryError
- Request timeouts
- Kafka consumer lag
- Database connection exhaustion
- Entire application slowdown
Main Goal
Identify Root Cause Fix Performance Problem Prevent Future Occurrences
Production Troubleshooting Approach
- Monitor Metrics
- Analyze CPU Usage
- Analyze Memory Usage
- Check Logs
- Perform Distributed Tracing
- Capture Heap Dumps
- Thread Dump Analysis
- Database Query Analysis
- GC Analysis
- Traffic Analysis
- Load Testing
- Code Optimization
- Infrastructure Scaling
1. First Step — Check Monitoring Dashboards
Never guess in production.
Use Monitoring Tools
- :contentReference[oaicite:0]{index=0}
- :contentReference[oaicite:1]{index=1}
- :contentReference[oaicite:2]{index=2}
- :contentReference[oaicite:3]{index=3}
Check Metrics
- CPU usage
- Memory usage
- Pod restarts
- Request latency
- Error rates
- GC pauses
- Heap memory
- Thread count
- Kafka lag
- Database connections
Production Example
CPU suddenly spikes from: 20% → 95%
Possible Reasons
- Infinite loop
- Traffic spike
- Heavy database query
- Memory leak
- Thread contention
- Retry storm
2. Check Application Logs
Logs provide important clues.
Centralized Logging Tools
- :contentReference[oaicite:4]{index=4}
- :contentReference[oaicite:5]{index=5}
- :contentReference[oaicite:6]{index=6}
Look For
- OutOfMemoryError
- GC overhead
- Timeouts
- Retry loops
- Connection pool exhaustion
- Thread starvation
- Infinite recursion
Example Log
java.lang.OutOfMemoryError: Java heap space
Meaning
Application Consuming Excessive Memory
3. Analyze CPU Usage
High CPU usage usually means:
- Heavy computations
- Infinite loops
- Thread contention
- Excessive retries
- Serialization overhead
- Too many requests
Linux Commands
top htop
Kubernetes Example
kubectl top pod
Example Output
payment-service: CPU = 1800m Memory = 3GB
Thread Dump Analysis
Very important for CPU troubleshooting.
Java Command
jstack <PID>
Look For
- Blocked threads
- Deadlocks
- Infinite loops
- High CPU threads
Example Problem
while(true) {
processPayments();
}
Result
CPU Becomes 100%
4. Analyze Memory Usage
High memory usage usually means:
- Memory leaks
- Large object creation
- Unreleased collections
- Cache overflow
- Huge payloads
- Thread leaks
Heap Dump Analysis
Most important for memory issues.
Java Command
jmap -dump:live,format=b,file=heap.hprof <PID>
Analyze Using
- :contentReference[oaicite:7]{index=7}
- :contentReference[oaicite:8]{index=8}
- :contentReference[oaicite:9]{index=9}
Example Memory Leak
static List<String> cache = new ArrayList<>(); cache.add(data);
Problem
List Keeps Growing Forever
Result
OutOfMemoryError
5. GC (Garbage Collection) Analysis
Frequent GC pauses increase CPU usage.
Symptoms
- Application pauses
- High CPU
- Slow response time
Enable GC Logs
-XX:+PrintGCDetails
Check
- GC frequency
- Full GC count
- Pause duration
Problem Example
Large Objects Created Repeatedly
Result
Frequent Full GC CPU Spike
6. Database Query Analysis
Slow queries can increase CPU and memory.
Example Problem
SELECT * FROM transactions
Problem
- Huge data loaded
- Memory consumption increases
- CPU usage increases
Correct Approach
SELECT id, amount FROM transactions LIMIT 100
Production Fixes
- Add indexes
- Pagination
- Optimize joins
- Connection pooling
- Avoid N+1 queries
Database Monitoring Tools
- :contentReference[oaicite:10]{index=10}
- :contentReference[oaicite:11]{index=11}
7. Check Traffic Spike
Sometimes application is healthy but traffic increases suddenly.
Example
Black Friday Sale
Traffic Increases
100 Requests/sec → 20,000 Requests/sec
Result
CPU And Memory Spike
Solution
- Horizontal scaling
- Auto scaling
- Rate limiting
- Caching
- Load balancing
8. Kubernetes Auto Scaling
Automatically add more pods.
Example
CPU > 70%
Then
Increase Pods Automatically
Kubernetes HPA Example
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
averageUtilization: 70
Benefits
- Handles traffic spikes
- Improves availability
9. Distributed Tracing
Sometimes another service causes slowdown.
Example
Payment Service Calling Fraud Service
Fraud Service Slow
Payment Threads Blocked CPU Increases
Use Distributed Tracing
- :contentReference[oaicite:12]{index=12}
- :contentReference[oaicite:13]{index=13}
Benefits
- Identify slow dependencies
- Track request flow
10. Retry Storm Analysis
Excessive retries can destroy systems.
Scenario
Payment Service Calls Inventory Service
Inventory Service Down
Payment Retries Continuously
Result
CPU Spike Thread Exhaustion Memory Increase
Solution
- Circuit breaker
- Retry limits
- Exponential backoff
Popular Tool
- :contentReference[oaicite:14]{index=14}
11. Thread Pool Analysis
Improper thread pool configuration causes problems.
Problem
Too Many Threads
Result
- Context switching overhead
- High CPU usage
- Memory increase
Correct Configuration
corePoolSize = 20 maxPoolSize = 50
12. Caching Problems
Large cache may consume huge memory.
Example
Cache All Transactions Forever
Problem
Memory Usage Keeps Growing
Solution
- TTL expiration
- LRU eviction
- Distributed cache
Popular Cache Tools
- :contentReference[oaicite:15]{index=15}
- :contentReference[oaicite:16]{index=16}
13. Memory Limits in Kubernetes
Protect cluster from unhealthy services.
Kubernetes Example
resources:
limits:
memory: "2Gi"
cpu: "1000m"
Benefits
- Prevent node crashes
- Resource isolation
14. Load Testing
Reproduce issue safely before fixing.
Tools
- :contentReference[oaicite:17]{index=17}
- :contentReference[oaicite:18]{index=18}
- :contentReference[oaicite:19]{index=19}
Benefits
- Identify bottlenecks
- Validate fixes
15. Real Production Incident
Scenario
Payment Service CPU reached 100%.
Investigation
- Grafana showed sudden spike
- Thread dump showed retry loops
- Distributed tracing showed Inventory Service latency
- Logs showed timeout exceptions
Root Cause
Infinite Retries To Slow Dependency
Fix Applied
- Implemented circuit breaker
- Added retry limits
- Enabled exponential backoff
- Scaled Inventory Service
Final Result
- CPU normalized
- Latency reduced
- System stabilized
Production Best Practices
| Practice | Purpose |
|---|---|
| Monitoring | Early detection |
| Heap Dump Analysis | Memory leak detection |
| Thread Dumps | CPU issue analysis |
| Auto Scaling | Handle traffic spikes |
| Circuit Breakers | Prevent retry storms |
| Distributed Tracing | Dependency analysis |
| Resource Limits | Cluster protection |
| Load Testing | Performance validation |
Final Interview Answer
If a microservice consumes excessive memory and CPU in production, I would first analyze monitoring dashboards using tools like :contentReference[oaicite:20]{index=20} and :contentReference[oaicite:21]{index=21} to identify abnormal metrics such as CPU spikes, memory growth, GC pauses, latency, and pod restarts. Then I would investigate application logs and perform thread dump analysis using tools like jstack to identify blocked threads, infinite loops, retry storms, or thread contention. For memory-related issues, I would capture heap dumps using jmap and analyze them with tools like :contentReference[oaicite:22]{index=22} to detect memory leaks or large object retention. I would also analyze database queries, distributed tracing, traffic spikes, cache usage, and external dependencies. If traffic is the cause, I would enable Kubernetes auto scaling and optimize load balancing. To prevent cascading failures, I would implement circuit breakers and retry limits using tools like :contentReference[oaicite:23]{index=23}. Finally, I would validate fixes using load testing tools and configure proper CPU and memory limits in Kubernetes to ensure system stability in production.