How to Debug a Production Issue Without Panicking
It was around 2 AM when my phone rang.
I was half asleep, so for a few seconds I just looked at the screen, hoping whoever was calling would call back in the morning.
Then I saw the team name. I picked it up.
“Production mein issue hai. Payment requests fail ho rahi hain.”
Bas. Neend gayi.
I opened my laptop and joined the call. Everyone was already trying to figure things out. Someone was checking logs, someone was looking at the database, and someone was asking about the latest deployment. And honestly, my first thought wasn’t some amazing debugging strategy.
It was simply, “Yaar, ab kya hua?”
That’s the thing about production issues. When something breaks on your local machine, you have time. You can add logs, put breakpoints, restart the application, change things and try again.
Production is different.
There are real users on the other side. Someone is trying to make a payment. Someone is waiting for an order. Someone is staring at an error message. And while you’re trying to understand the problem, people are waiting for an answer.
That’s where I learned one of the most important things about production debugging:
Don’t panic. And don’t start fixing things before you understand what is actually broken.
My First Instinct Used to Be Wrong
Earlier, whenever I saw a production issue, I would immediately open the code.
API failing? Open the controller.
Then the service.
Then the repository.
Then the database query.
Ten minutes later, I would have twenty tabs open and still wouldn’t know what was happening.
The problem wasn’t that I couldn’t debug.
I was just starting from the wrong place.
Instead of asking “Where is the bug?”, I should have been asking “What exactly is happening?”
So that night, I started with the simplest thing: the numbers.
How many requests were failing?
When did they start failing?
Were all payments failing?
Or only some?
That immediately gave us an interesting clue.
Not every payment was failing.
Some were going through perfectly.
Now the problem wasn’t “payment is down.”
Something was failing under a particular condition.
And suddenly, the problem became much smaller.
Don’t Trust the First Explanation
During production incidents, someone will almost always say something like:
“Database down hai.”
“API broken hai.”
“Payment service is down.”
I’ve learned not to take these statements as facts.
Usually, they’re just someone’s observation.
If a payment fails, it doesn’t necessarily mean the payment service is broken. Maybe the database is fine. Maybe the API is fine. Maybe a third-party service is returning something unexpected.
So now, whenever I hear “X is broken”, my first reaction is:
“Okay, let’s verify that.”
That one habit can save a lot of time.
Start With the Timeline
The next thing I usually check is the timeline.
When did the problem actually start? And more importantly, what changed just before it started?
Was there a deployment? A configuration change? A database migration? A traffic spike? A change in another service?
Production systems have a lot of moving parts, and sometimes your code hasn’t changed at all.
Maybe another service was deployed.
Maybe traffic suddenly increased.
Maybe a scheduled job started behaving differently.
Maybe an external API started responding slowly.
That night, we noticed that the failures had started shortly after a deployment. Interesting. But I didn’t want to immediately say, “Deployment caused it.”
It was a clue, not a conclusion.
The Production Debugging Flow
This is roughly how I approach a production issue now. I start broad, collect evidence, and slowly narrow the problem down until I reach the actual failure point.
Follow One Failing Request
At this point, instead of looking at thousands of logs, I picked one failing request. I wanted to follow its entire journey.
The request reached our API. The business logic ran. The database responded normally. Then our service called another service. And that’s where things started looking strange.
This is where I usually step back and look at the bigger picture. A production issue rarely gives you the answer from one place.
I look at the timeline, logs, metrics and request trace together, then compare the evidence before deciding what the actual root cause is.
The downstream service wasn’t completely down. It was responding, but in one particular scenario, the response wasn’t what our application expected.
That changed everything.
We weren’t dealing with a completely broken application.
We had a specific failure happening between two services.
This is something production teaches you pretty quickly: your application doesn’t live alone. It depends on databases, queues, other services, external APIs and infrastructure.
Sometimes your code is perfectly fine.
Something around it isn’t.
Find One That Works
One of the simplest things I do during debugging is compare a failing request with a successful request.
Same API?
Same type of user?
Same data?
Same region?
Same request?
Same downstream response?
Usually, there is a difference somewhere.
Maybe one particular data combination is causing the issue.
Maybe only older records fail.
Maybe only a particular user flow triggers it.
The moment you find that difference, the problem becomes much easier.
Instead of saying:
“The payment system is broken.”
You can say:
“Payments fail when this specific condition occurs.”
That’s a much better problem to have.
Logs Should Tell You a Story
I used to think more logs meant easier debugging.
Not really.
I’ve seen systems where a single request generates hundreds of log lines. Everything is logged, but when production breaks, finding the useful information becomes harder.
What you actually want is a story.
Which request was this?
What operation was happening?
Which service handled it?
Which dependency was called?
What happened after that?
This is where correlation IDs are extremely useful.
If a request travels through multiple services, you should be able to follow that request across the system.
At 2 AM, you don’t want more information.
You want the right information.
Don’t Change Everything at Once
This is probably the easiest mistake to make when everyone is waiting for you.
You increase a timeout.
Restart the service.
Change some configuration.
Modify the query.
Get Himanshu Singour’s stories in your inbox
Join Medium for free to get updates from this writer.
Deploy again.
And suddenly everything works.
Great.
But what actually fixed it?
You don’t know.
Maybe the restart helped. Maybe the configuration change helped. Maybe the external service recovered at the same time.
That’s why I try to make one meaningful change at a time.
Change something.
Observe.
Measure.
Then move forward.
Otherwise, you might fix the problem without understanding it.
And the next time it happens, you’re back at the beginning.
Sometimes Rollback Is the Right Fix
There is sometimes a strange ego around deployments.
Engineers don’t want to rollback because it feels like admitting that the deployment was wrong.
But production doesn’t care about your ego.
If a recent deployment is strongly connected to the incident and rollback is safe, rollback.
The goal during an incident is not to prove that your code is perfect.
The goal is to get users back to a stable system.
You can investigate the real root cause afterward.
Production first. Ego later.
The Hardest Part Is Staying Calm
Honestly, the technical problem isn’t always the hardest part.
The pressure is.
Someone asks, “Any update?”
You say you’re checking.
Five minutes later:
“Any update?”
Another team is waiting. Someone wants to know the impact. Meanwhile, you’re staring at the logs thinking, “I still don’t know what’s happening.”
I’ve been there.
And I’ve learned that not knowing the answer doesn’t mean you need to start moving faster.
Sometimes you need to stop for thirty seconds.
Ask yourself:
What do I know?
What don’t I know?
What have I already ruled out?
What’s the next thing I can check?
That’s enough.
You don’t need to solve the whole incident in your head.
You just need to find the next useful clue.
The Problem Usually Becomes Smaller
At 2 AM, the problem looked huge.
“Payment is failing in production.”
But after checking the metrics, looking at the timeline, following one request and comparing it with a successful request, the problem became much smaller.
It wasn’t the entire payment system.
It wasn’t the database.
It wasn’t even the whole API.
It was a specific interaction between services under a specific condition.
That’s how I think about debugging now.
You don’t magically find the answer.
You keep removing possibilities.
One by one.
Until the giant production problem becomes one service, one request, one dependency and one bug.
And that’s when debugging becomes much easier.
Fixing the Bug Isn’t the End
Once everything starts working again, it’s tempting to close the laptop and forget the whole thing.
But that’s actually when the important questions start.
Why did this happen?
Why didn’t we catch it earlier?
Why didn’t our monitoring detect it?
Could we have handled the failure better?
Do we need better validation?
Better logging?
A timeout?
A circuit breaker?
A test?
The goal isn’t just to fix today’s incident.
It’s to make sure the same problem doesn’t wake you up at 2 AM again.
What I Finally Learned
I used to think a good engineer was someone who could look at an error and immediately know the answer.
I don’t think that anymore.
The engineers I respect most aren’t the ones who know everything.
They’re the ones who stay calm when they don’t know.
They don’t guess.
They don’t randomly change five things.
They don’t assume the first error message is the root cause.
They simply keep asking better questions.
Did the request reach the service?
Did the database respond?
Did the downstream service respond?
When did the problem start?
What changed?
Why did this request fail while another one worked?
Slowly, the huge problem becomes smaller.
And eventually, you find the answer.
So the next time your phone rings at 2 AM and someone says, “Production mein issue hai,” don’t immediately start changing things.
Take a breath.
Understand what is actually happening.
Follow one failing request.
Find one that worked.
Compare them.
Reduce the possibilities.
Then fix the thing that is actually broken.
Because production debugging isn’t about being the smartest person in the room.
It’s about staying calm enough to figure out what the system is trying to tell you.
And honestly, that’s one of the most valuable engineering skills I’ve learned.
Before You Go
One more thing.
If you’re serious about becoming a better software engineer, I’m building something that might be useful for you.
Instead of putting everything into one huge learning platform, I’ve created focused learning paths for different stages of an engineer’s journey.
Master AI Engineering
Learn how to build AI-powered applications, understand AI systems, work with LLMs, agents, RAG, and take AI from an idea to something that actually works.
Master AI Engineering: https://algorithmxlr8.io/ai/engineering/home
Master DevOps Engineering
Understand the engineering behind deployments, containers, CI/CD, cloud infrastructure, monitoring, scalability, and running applications reliably in production.
Master DevOps Engineering: https://algorithmxlr8.io/devops/engineering/home
Master Software Engineer Interviews
DSA, LLD, HLD, System Design and the kind of problems you’re actually expected to solve in software engineering interviews.
Master Software Engineer Interviews: https://algorithmxlr8.io/
The goal isn’t just to help you clear an interview.
It’s to help you become the engineer who actually understands what’s happening behind the code, how systems work, and how to build things properly.
If that sounds like the kind of engineer you want to become, come join me on the journey.
Explore: https://algorithmxlr8.io/
