We Shipped a Bug That Charged Customers Twice. Here's the Postmortem
A retry, a missing idempotency key, and 47 minutes that taught our team more than any design review ever did.
At 6.42 on a Friday evening, our support queue went from 3 open tickets to 140 in about ten minutes. Every one of them said roughly the same thing. I was charged twice.
I was the on-call engineer. I still remember the feeling in my stomach when I opened the payments dashboard and saw the capture count running at almost double the order count.
This is the postmortem we wrote, cleaned up a bit and shared with permission. Blameless, as always, though I'll admit some of the blame is mine.
What Happened
Between 6.31 PM and 7.18 PM, 2,214 customers were charged twice for a single order. The total duplicate amount was about $186,000. All of it was refunded within 36 hours, but some customers saw overdraft fees from their banks, and we covered those too.
No data was lost. No orders were duplicated. Only the payment captures.
The Setup
Our checkout service is a Spring Boot app that calls our payment provider through an internal PaymentClient. When a customer places an order, we create the order in PostgreSQL, then call the provider to capture the charge, then publish an OrderPaid event to Kafka.
Two weeks before the incident, we'd added a Resilience4j retry around the capture call. The provider had a few flaky timeouts, and retrying seemed like the obvious fix.
@Retry(name = "paymentCapture")
public CaptureResult capture(Order order) {
return paymentApi.capture(new CaptureRequest(
order.getPaymentToken(),
order.getTotalCents(),
order.getCurrency()));
}resilience4j.retry.instances.paymentCapture:
max-attempts: 3
wait-duration: 500ms
retry-exceptions:
- java.net.SocketTimeoutException
Look at the request. No idempotency key.
The Trigger
At 6.31 PM, the provider started having latency problems on their side. Captures that normally took 300ms were taking 4 to 6 seconds. Our client timeout was 3 seconds.
So our client gave up waiting and threw SocketTimeoutException. The retry kicked in and sent the capture again. But the original request hadn't failed. It was just slow. The provider eventually processed both.
From our side, the first attempt timed out and the second one succeeded. From the provider's side, two valid capture requests came in, and they did exactly what we asked.
A timeout tells you that you stopped waiting. It doesn't tell you the other side stopped working.
Why We Didn't Catch It Sooner
Three things delayed detection.
- Our alerts watched payment failures, not payment duplicates. Everything looked healthy because everything was succeeding.
- The retry metric existed in Micrometer, but nobody had put it on a dashboard or an alert.
- It was Friday evening and order volume was high, so the jump in captures blended in with normal traffic for the first few minutes.
Support saw it before monitoring did. That's always a bad sign.
The Response
Here's the rough timeline.
- 6.42 PM. Support escalates. I get paged.
- 6.51 PM. I spot the capture to order ratio at 1.9 on the dashboard.
- 6.58 PM. We find the retry logs and connect it to provider latency.
- 7.04 PM. We flip a feature flag to disable retries on the capture path.
- 7.18 PM. Duplicate captures stop as in-flight retries drain.
- 7.40 PM. We start building the list of affected customers.
The 13 minutes between spotting the ratio and disabling retries is the part I'd most like back. I spent too long trying to understand the root cause before stopping the damage. Stop the bleeding first, investigate second. I knew that and still didn't do it.
Root Cause
The direct cause was retrying a non-idempotent operation.
Get Naveen Metta’s stories in your inbox
Join Medium for free to get updates from this writer.
But the deeper cause was that our design treated a payment capture like any other HTTP call. We'd added retries the same way we'd added them to our inventory lookup, which is safe to repeat. Capturing money isn't.
Nobody reviewing the PR asked whether the operation was safe to retry. I was one of the reviewers. I approved it.
The Fixes
Idempotency keys on every money movement
Our provider supports an idempotency key header. If they see the same key twice, they return the original result instead of charging again. We now generate one from the order ID and attempt type, and store it with the order.
String key = "capture-" + order.getId();
return paymentApi.capture(request, Map.of("Idempotency-Key", key));Because the key is derived from the order, retries from any pod or any restart produce the same key. That matters. A random UUID per attempt would've been useless.
Timeouts that match reality
We raised the capture timeout to 10 seconds and added a separate reconciliation job. If a capture times out, we don't retry blindly. We mark the order as pending and query the provider for the true status a minute later.
Alerting on the right signal
We now alert when the capture to order ratio goes above 1.02 for five minutes. We also alert on retry rate for any payment endpoint. Both would've fired within three minutes of the incident starting.
A review checklist question
Every PR that adds a retry now has to answer one question in the description. Is this operation safe to execute twice? If the answer isn't an obvious yes, it needs an idempotency key or no retry at all.
Cleaning Up the Mess
The technical fix took a day. The cleanup took most of a week.
We had to figure out exactly which captures were duplicates. That sounds easy, but some customers had genuinely placed two orders for the same amount within minutes. Refunding those would've been its own incident.
We matched captures to orders by provider transaction ID and only refunded captures with no matching order row. We ran the script in dry run mode first, had a second engineer check a random sample of 50, and only then ran it for real.
Support sent personal emails to every affected customer. Our finance team reimbursed 31 overdraft fees. It cost real money and a lot of goodwill, and no amount of clever code gets that back.
What I Took Away Personally
I'd read about idempotency a dozen times. I'd explained it in interviews. And I still approved a PR that retried a payment without one.
Knowing a concept and applying it under deadline pressure are two different skills. The checklist question exists because I don't trust myself, or anyone, to remember every time.
Key Takeaways
- Never retry a non-idempotent call without an idempotency key.
- A timeout means you stopped waiting, not that the request failed.
- Alert on business ratios like captures per order, not just error rates.
- During an incident, stop the damage first and find the root cause second.
- Turn hard lessons into checklist questions so they survive the next deadline.
Have you been bitten by a retry that should never have happened? Share your story in the comments.
