One Microservice Becomes Slow and Entire Application Performance Degrades — How Will You Troubleshoot and Fix It?

In a microservices architecture, services communicate with each other using REST APIs, Kafka, RabbitMQ, or gRPC. If one microservice becomes slow, dependent services start waiting, which can eventually degrade the performance of the entire application.

---

Real-Time Production Example

Consider a banking application with the following services:

Mobile App
    ↓
API Gateway
    ↓
Transaction Service
    ↓
Payment Service
    ↓
Fraud Detection Service
    ↓
Notification Service

Suppose the Payment Service becomes slow.

Then:

• Fund transfers become slow
• API response time increases
• Threads remain blocked
• Kafka lag increases
• Customers receive timeout errors
• Entire application performance degrades

---

Step 1: Identify Which Microservice Is Slow

First, identify the bottleneck using monitoring and observability tools.

Production Monitoring Tools

• Prometheus
• Grafana
• Zipkin
• Jaeger
• ELK Stack
• Datadog
• Splunk

Grafana Example

Transaction Service = 100ms
Payment Service     = 12 sec
Fraud Service       = 50ms

Clearly, Payment Service is the bottleneck.

---

Step 2: Use Distributed Tracing

Distributed tracing helps identify where latency occurs across microservices.

Request Flow

Customer Request
      ↓
Transaction Service (100ms)
      ↓
Payment Service (11 sec)
      ↓
Notification Service (40ms)

This confirms that Payment Service is causing the delay.

---

Spring Boot Zipkin Configuration

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

application.yml

spring:
  zipkin:
    base-url: http://zipkin-server:9411

---

Step 3: Analyze Application Logs

Check logs for:

• Timeout exceptions
• Database connection pool exhaustion
• Deadlocks
• Retry storms
• External API failures

Production Error Example

HikariPool-1 - Connection is not available
Request timed out after 30000ms

This indicates database connection exhaustion.

---

Step 4: Analyze Database Performance

Slow database queries are one of the biggest reasons for microservice latency.

Slow Query Example

SELECT * FROM transactions
WHERE transaction_status='SUCCESS';

If the table contains millions of records and no index exists, the database performs a full table scan.

---

Fix: Add Database Index

CREATE INDEX idx_transaction_status
ON transactions(transaction_status);

Performance Improvement

Before Index → 18 sec
After Index  → 120ms

---

Step 5: Check Thread Pool Exhaustion

Each incoming request consumes one thread. If downstream services become slow, threads remain blocked for a long time.

Production Symptoms

Tomcat Threads = 200
Busy Threads   = 200
Queued Requests = 5000

New requests cannot be processed.

---

Thread Pool Optimization

server:
  tomcat:
    threads:
      max: 400
      min-spare: 50

---

Step 6: Analyze External API Latency

Many applications depend on external APIs such as:

• Payment Gateway APIs
• UPI APIs
• Banking APIs
• Credit Score APIs
• Third-party Verification APIs

Problem Example

Third-Party API Response Time = 25 sec

The application waits continuously, causing latency.

---

Fix: Configure Timeout

Spring WebClient Example

@Bean
public WebClient webClient() {

    return WebClient.builder()
            .baseUrl("https://bank-api.com")
            .build();
}

Timeout Configuration

public Mono<String> transfer() {

    return webClient.get()
            .uri("/payment")
            .retrieve()
            .bodyToMono(String.class)
            .timeout(Duration.ofSeconds(3));
}

Now the application stops waiting after 3 seconds.

---

Step 7: Implement Circuit Breaker

Circuit Breaker prevents cascading failures.

Production Scenario

If Payment Service becomes unavailable:

• Threads remain blocked
• CPU usage increases
• Other services become slow
• Entire application may crash

---

Resilience4j Circuit Breaker Example

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
</dependency>

Implementation

@CircuitBreaker(
name = "paymentService",
fallbackMethod = "fallbackResponse")
public String processPayment() {

    return paymentClient.call();
}

public String fallbackResponse(Exception ex) {

    return "Payment Service Temporarily Unavailable";
}

---

How Circuit Breaker Works

Service Healthy
      ↓
Requests Allowed
      ↓
Failures Increase
      ↓
Circuit Opens
      ↓
Requests Blocked Temporarily
      ↓
System Protected

---

Step 8: Implement Retry Mechanism

Retries help recover temporary failures.

Retry Example

@Retry(name = "paymentRetry")
public String makePayment() {

    return paymentClient.call();
}

Retries must be limited to avoid retry storms.

---

Step 9: Use Redis Cache

Frequently accessed data should be cached.

Examples

• Customer profile
• Account summary
• Product details
• Exchange rates

---

Spring Cache Example

@Cacheable(value = "customerCache",
key = "#customerId")
public Customer getCustomer(Long customerId) {

    return repository.findById(customerId).get();
}

Performance Improvement

Before Cache → 700ms
After Cache  → 15ms

---

Step 10: Kafka Consumer Lag Troubleshooting

Microservices often use Kafka for asynchronous communication.

Production Kafka Issue

Topic = payment-events
Consumer Lag = 5 Million

Consumers cannot process records fast enough.

---

Kafka Consumer Scaling

spring:
  kafka:
    listener:
      concurrency: 10

Now multiple consumers process messages in parallel.

---

Kafka Retry Example

@RetryableTopic(
attempts = "3",
backoff = @Backoff(delay = 2000)
)
@KafkaListener(topics = "payment-topic")
public void consume(String message) {

    process(message);
}

---

Step 11: Implement Bulkhead Pattern

Bulkhead isolates failures.

Example

If Loan Service becomes slow, UPI payments should continue working.

---

Bulkhead Example

@Bulkhead(
name = "paymentBulkhead",
type = Bulkhead.Type.THREADPOOL)
public String processPayment() {

    return paymentService.call();
}

---

Step 12: Scale the Microservice

If traffic increases heavily:

CPU Usage = 95%
Memory Usage = 90%

Scale horizontally.

---

Kubernetes Scaling Example

apiVersion: apps/v1
kind: Deployment

spec:
  replicas: 10

---

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler

spec:
  minReplicas: 2
  maxReplicas: 15

  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

---

Step 13: Move to Async Communication

Long synchronous communication chains increase latency.

Bad Design

Transaction Service
      ↓
Payment Service
      ↓
Notification Service
      ↓
Audit Service

Every service waits for the next service.

---

Better Design Using Kafka

Transaction Completed
       ↓
Publish Kafka Event
       ↓
Notification Service Consumes
Audit Service Consumes
Fraud Service Consumes

Services process independently.

---

Step 14: Check JVM and Memory Issues

Symptoms

• Frequent Full GC
• OutOfMemoryError
• High heap usage

---

JVM Monitoring Tools

• VisualVM
• JConsole
• Eclipse MAT
• Heap Dump Analyzer

---

JVM Optimization

JAVA_OPTS="
-Xms2G
-Xmx4G
-XX:+UseG1GC"

---

Production Incident Example

Issue

An online banking system became slow during salary credit processing.

Root Causes

• Missing database index
• Slow payment gateway API
• Kafka lag
• Only 2 pods deployed
• Thread pool exhaustion

---

Fixes Applied

• Added database indexes
• Implemented Redis cache
• Configured timeout and retries
• Added circuit breaker
• Scaled pods from 2 to 12
• Increased Kafka consumers
• Implemented async processing

---

Final Result

Before Optimization
Response Time = 20 sec

After Optimization
Response Time = 180ms

---

Important Production-Level Techniques

Technique	Purpose
Distributed Tracing	Identify bottleneck
Circuit Breaker	Prevent cascading failures
Timeout	Avoid long waits
Retry	Handle temporary failures
Bulkhead	Isolate failures
Redis Cache	Improve response time
Kafka Scaling	Reduce consumer lag
Kubernetes Autoscaling	Handle traffic spikes
DB Indexing	Optimize queries
Thread Pool Tuning	Improve request handling

---

Interview Summary Answer

If one microservice becomes slow, first identify the bottleneck using monitoring and distributed tracing tools. Then analyze logs, database queries, thread pools, Kafka lag, external API latency, CPU, and memory usage. Based on the root cause, apply solutions such as indexing, Redis caching, timeout configuration, retries, circuit breakers, bulkhead isolation, Kafka scaling, Kubernetes autoscaling, and asynchronous communication to prevent cascading failures and improve overall system performance.