Kubernetes Troubleshooting: A Real-World Scenario Walkthrough

Introduction

It’s 2:00 AM. Your phone buzzes with an alert: “Production service degradation detected.” As a DevOps engineer or platform administrator, this scenario is all too familiar. Kubernetes, while powerful, comes with its own set of complex troubleshooting challenges. Today, I’ll walk you through a real-world troubleshooting scenario that combines multiple failure points, providing you with a systematic approach to diagnose and resolve Kubernetes issues.

The Scenario: E-commerce Payment Processing Service Outage

Imagine you’re managing a Kubernetes cluster for an e-commerce platform. The payment processing microservice is reporting intermittent failures, with customers unable to complete purchases. The monitoring dashboard shows increased error rates and latency spikes. Your task: identify the root cause and restore service.

Initial Assessment

When facing any Kubernetes issue, start by understanding the scope of the problem:

# Check overall cluster health
kubectl get nodes
kubectl get pods --all-namespaces -o wide

In our scenario, the output shows:

• All nodes are in Ready status
• Several pods in the payment-system Namespaces are in CrashLoopBackOff state
• The database pods appear to be running, but with restarts

Step 1: Examining the Problematic Pods

First, let’s focus on the pods showing issues:

kubectl get pods -n payment-system

Output:

NAME                           READY   STATUS             RESTARTS   AGE
payment-api-5d7f98b955-2xkfd   0/1     CrashLoopBackOff   7          45m
payment-api-5d7f98b955-9lkj3   0/1     CrashLoopBackOff   7          45m
payment-db-statefulset-0       1/1     Running            2          12h
payment-db-statefulset-1       1/1     Running            2          12h
payment-db-statefulset-2       1/1     Running            3          12h

Let’s check the logs and events:

kubectl logs payment-api-5d7f98b955-2xkfd -n payment-system

The logs reveal connection timeout errors when attempting to connect to the database:

Error: Connection timed out after 5000ms when connecting to payment-db-service:5432
...
Error: Maximum connection retries reached (5)

Step 2: Investigating Network Connectivity

Since we’re seeing connection timeouts, let’s verify network policies and DNS resolution:

# Check if services are correctly defined
kubectl get svc -n payment-system

# Check endpoints for the service
kubectl get endpoints payment-db-service -n payment-system

The service output looks normal, but the endpoints show an issue:

NAME                ENDPOINTS                                      AGE
payment-db-service  <none>                                         12h

No endpoints! This explains why our API pods can’t connect to the database. But why are there no endpoints when the database pods are running?

Step 3: Investigating Service-Pod Connectivity

Let’s examine the service definition and pod labels:

kubectl describe svc payment-db-service -n payment-system

Output:

Name:              payment-db-service
Namespace:         payment-system
Labels:            app=payment-db
Annotations:       <none>
Selector:          app=payment-database
Type:              ClusterIP
IP:                10.100.45.10
Port:              postgresql  5432/TCP
TargetPort:        5432/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

Now let’s check the pod labels:

kubectl get pods -n payment-system --show-labels

Output:

NAME                           READY   STATUS             RESTARTS   AGE   LABELS
payment-api-5d7f98b955-2xkfd   0/1     CrashLoopBackOff   8          55m   app=payment-api
payment-api-5d7f98b955-9lkj3   0/1     CrashLoopBackOff   8          55m   app=payment-api
payment-db-statefulset-0       1/1     Running            2          12h   app=payment-db
payment-db-statefulset-1       1/1     Running            2          12h   app=payment-db
payment-db-statefulset-2       1/1     Running            3          12h   app=payment-db

Aha! There’s a label mismatch. The service is looking for pods with a label app=payment-databaseBut our database pods have app=payment-db.

Step 4: Investigating Recent Changes

When did this issue start? Let’s check the recent changes:

kubectl get events -n payment-system --sort-by='.lastTimestamp'

The events show that a service update occurred shortly before the issues began. Let’s check the deployment history:

kubectl rollout history deployment payment-api -n payment-system

A recent deployment was made that coincides with the start of the problems.

Step 5: Investigating Resource Constraints

While examining the events, we notice some resource-related warnings on the nodes where our payment pods are scheduled:

kubectl describe node worker-node-3

The output reveals high memory pressure, which could be affecting our pods’ performance even after we fix the connection issue.

Root Causes Identified

We’ve now identified two issues:

1. Service-Pod Label Mismatch: The service selector doesn’t match the database pod labels
2. Resource Pressure: One of our worker nodes is experiencing memory pressure

Resolution Steps

Fix 1: Correcting the Service Definition

We need to update the service to use the correct selector:

kubectl patch svc payment-db-service -n payment-system --type='json' -p='[{"op": "replace", "path": "/spec/selector", "value":{"app":"payment-db"}}]'

Verify the change worked:

kubectl get endpoints payment-db-service -n payment-system

Now we should see endpoints populated with our database pod IPs.

Fix 2: Addressing Memory Pressure

For the memory pressure issue, we have a few options:

1. Adjust resource limits for the pods on that node:

kubectl set resources deployment payment-api -n payment-system --limits=memory=512Mi --requests=memory=256Mi

1. Cordon the problematic node to prevent new pods from being scheduled there:

kubectl cordon worker-node-3

1. Drain the node to move existing pods to other nodes (if you have capacity):

kubectl drain worker-node-3 --ignore-daemonsets

Verification

After applying these fixes, monitor the service recovery:

# Watch the pods recover
kubectl get pods -n payment-system -w

# Check application logs for successful connections
kubectl logs payment-api-5d7f98b955-2xkfd -n payment-system

Once the API pods can connect to the database and run stably, verify that the payment service is functioning by running test transactions through your monitoring or synthetic testing system.

Lessons Learned

This troubleshooting exercise highlights several key takeaways:

1. Follow a systematic approach: Move from symptoms to components to configurations
2. Check labels and selectors carefully: Small mismatches can completely break connectivity
3. Monitor resource usage: Resource constraints can cause seemingly unrelated issues
4. Implement change control: The service mismatch likely occurred during a recent deployment
5. Document your fixes: Create runbooks for similar issues in the future

Preventive Measures

To prevent similar issues in the future:

1. Implement CI/CD validation to catch label mismatches before deployment
2. Configure resource quotas at the namespace level to avoid resource exhaustion
3. Set up proactive alerts for endpoint disconnections and resource pressure
4. Use GitOps practices to manage and version control all Kubernetes configurations
5. Perform chaos engineering exercises to test your system’s resilience to similar failures

Conclusion

Kubernetes troubleshooting requires both systematic thinking and deep knowledge of how the components interact. By following the process outlined in this post, you can efficiently diagnose and resolve even complex multi-factor issues.

Get Saraswathi Lakshman’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Remember that effective troubleshooting isn’t just about fixing the immediate problem — it’s about understanding why it happened and implementing safeguards to prevent similar issues in the future.

What Kubernetes troubleshooting challenges have you faced? Share your experiences in the comments!