How I Debug a Production Issue in the First 15 Minutes

The first quarter hour of an incident decides how long it lasts, so here's the exact routine I follow before I touch any code.

The routine moves from confirming the alert to mitigation before anyone hunts for root cause.

My phone buzzed a little before 3 AM. Checkout was broken.

Error rate was at 18%. Normal is under 0.2%.

The first time I handled an incident like that, years ago, I opened the code and started reading. That's the worst thing you can do. The code was the same code that worked fine yesterday.

Since then I've been on call for something like 60 incidents across four teams. Over time I've built a routine for the first 15 minutes. It's not fancy. It just keeps me from wasting time and making things worse.

Minute 0 to 2. Confirm and scope

Before anything else, I check that the alert is real. Alerts lie more often than people admit. A broken metric exporter can look exactly like an outage.

I open the main dashboard and ask three questions.

1. Is this real user impact or just a noisy metric?
2. Is it everything, or one endpoint, one region, one customer segment?
3. When exactly did it start?

That third question is the most valuable one. A sharp start time is a gift, because sharp starts usually mean a change. A slow climb usually means load, a leak, or a dependency degrading.

Minute 2 to 4. Tell people

I post in the incident channel right away, even if I know almost nothing.

Checkout errors at 18% for the last 10 minutes. Investigating. Next update in 10 minutes.

That one message stops five people from DMing me, and it gets the right people watching. If the impact is big, I page an incident commander so I can focus on debugging instead of answering questions.

I learned this one the hard way. During an early incident I stayed quiet for 40 minutes because I wanted to have answers first. My manager found out from a customer.

Minute 4 to 7. What changed

Most production issues are caused by a change. So I go looking for one.

• Deploys in the last few hours, ours and our dependencies.
• Config or feature flag changes.
• Infrastructure work, like node rotations, database failovers, or certificate renewals.
• Traffic changes, like a marketing email or a big partner batch job.

We have a single "changes" panel that pulls deploy events from CI and flag changes from our flag service onto the same timeline as error rate. It's the most useful panel we've ever built. If a deploy lines up with the start of the errors, I've probably got my answer.

Minute 7 to 10. Follow one failing request

Now I pick one actual failed request and follow it. Not the aggregate. One request.

I grab a trace ID from the error logs and open it in our tracing tool. Then I look for the span where it actually broke.

In the checkout incident, the trace showed every failing request spent 5 seconds waiting in the payment client, then threw a timeout. So the problem wasn't checkout at all. Checkout was just where it surfaced.

Get Naveen Metta’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If logs don't have trace IDs, that's a gap to fix after the incident. I add this to every Spring service now.

logging:
  pattern:
    level: "%5p [${spring.application.name},%X{traceId},%X{spanId}]"

With Micrometer Tracing, that traceId field shows up in every log line, so you can jump from a log to a trace in one click.

A slow query from an evening deploy exhausted the payment HikariCP pool and surfaced as checkout timeouts.

Minute 10 to 13. Check the usual suspects

Once I know which service is unhappy, I check the boring stuff. Boring stuff causes most outages.

• Connection pools. Is HikariCP showing hikaricp_connections_pending above zero?
• Thread pools. Are Tomcat threads maxed out?
• JVM. Is GC time spiking or heap close to the limit?
• Dependencies. Is the database, Redis, or Kafka slow?
• Pods. Are any restarting or getting OOM killed?

In the checkout case, the payment service's HikariCP pool was fully used, with 140 threads waiting for a connection. The database itself was fine. A slow query introduced in a deploy that evening was holding connections for 4 seconds each.

When I suspect threads, I take a quick thread dump.

jcmd <pid> Thread.print > /tmp/dump-1.txt
sleep 5
jcmd <pid> Thread.print > /tmp/dump-2.txt

Two dumps a few seconds apart tell you what's stuck versus what's just busy. If the same 140 threads sit on the same line in both dumps, you've found your bottleneck.

Minute 13 to 15. Mitigate first, fix later

This is the part a lot of engineers get wrong, including me for years. The goal of the first 15 minutes is not to find the root cause. It's to stop the bleeding.

So I ask what the fastest safe action is.

• Roll back the last deploy.
• Turn off the feature flag.
• Scale up if it's a capacity problem.
• Shed load or fail over to another region.

We rolled back the payment service. Errors dropped from 18% to 0.3% within four minutes. The real fix, an index on the new query, went out the next afternoon after a proper review.

A rollback isn't admitting defeat. It's buying time to think while customers can pay again.

What I don't do in the first 15 minutes

Equally important.

• I don't read code line by line hoping to spot the bug.
• I don't restart everything just to see what happens. That destroys evidence like heap state and thread dumps.
• I don't push a hotfix without review unless rollback is truly impossible.
• I don't debug alone in silence.

After the fire is out

Once things are stable, I write down the timeline while it's fresh. When it started, when we noticed, what we tried, what worked. Memory fades very fast after a 2 AM page.

Then we run a blameless postmortem within a few days. The checkout one produced three action items. A slow query check in CI, a pool saturation alert, and trace IDs in two services that were missing them.

Key takeaways

• Confirm the alert and find the exact start time before anything else.
• Communicate early, even if all you have is "investigating."
• Look for what changed, because something usually did.
• Follow one real failing request through traces.
• Mitigate first with rollbacks or flags, and find the root cause later.

What's the first thing you check when you get paged? Let me know in the comments.