Idempotency in Distributed Systems — The Pattern That Prevents Disasters

How to ensure your APIs handle retries gracefully, prevent double charges, and avoid duplicate orders — with full Spring Boot implementation examples.

The $4.2 Million Bug

One flaky network timeout cost a fintech company $4.2 million in a single weekend — and the code that caused it looked completely normal. In 2019, a company I was consulting for had a payment gateway that occasionally timed out. Their mobile app retried the request. The gateway processed it again. Two charges. Sometimes three.

In one weekend, they double-charged 1,847 customers totaling $4.2 million in duplicate transactions. Their customer support team worked 72 hours straight processing refunds while their CTO drafted an incident report for the regulators.

The fix? A 200-line implementation of idempotency keys. Something that should have been there from day one.

This is not an edge case. In distributed systems, the network is unreliable — timeouts happen, retries happen, messages get delivered more than once. If your system isn’t idempotent, it’s just a matter of time before something goes wrong.