Breaking Prod #24: AWS Went Down Three Times. Three Different Ways.
AWS Went Down Three Times. Three Different Ways.
The easy read on AWS this year is that it is having a rough stretch. One provider, a few bad days, it happens. That read is comforting and it is wrong.
Look at the three outages next to each other and they have almost nothing in common. A different layer of the stack failed each time. A different failure mode each time. This was not one bug that kept resurfacing. It was three independent failures in three different parts of the system, which is a far more unsettling picture than a single recurring bug, because you cannot patch your way out of it with one fix.
The Timeline
May 2026, the physical layer. Chiller units in a Northern Virginia data hall fail. Temperatures climb, and automated thermal-safety systems shut hardware down to protect it. EC2 instances and EBS volumes go with it. The failure is not code at all. It is metal, heat, and airflow.
July 16, 2026, the network layer. CloudFront cannot load its configuration. A failure in the VPC Origins feature means routing config cannot be applied correctly, and CloudFront serves 5xx errors instead of websites for about three and a half hours. It cascades far beyond AWS, taking identity providers, AI platforms, and collaboration tools with it. Nothing is physically broken. The routing brain just cannot get the right instructions to the right place.
July 2026, the service layer. An 80-minute outage in the service layer takes down a wide swath of the consumer internet at once, Apple Pay, Reddit, and Hulu among them. Not a data hall, not the network fabric, but the services running on top.
Three outages. A cooling failure, a config-distribution failure, and a service-layer failure. If you went looking for the pattern, the pattern is that there is no single pattern.
What Would You Have Done?
Your product just started throwing errors. You are not sure yet whether it is you or your cloud provider. In the first 30 minutes, do you:
A) Immediately dig into your own code, dashboards, and recent deploys to rule yourself out first.
B) Check provider status and your upstream dependencies in parallel with a fast sanity check of your own system, then move to comms and failover once it points upstream.
C) Wait for the provider to confirm an outage before doing anything, since there is no point acting until you know.
Answer: B. The slowest, most expensive path is A, because on an upstream outage your own system is healthy, so you can burn the entire window proving a negative. C is worse, because providers often confirm an outage long after your users feel it. The teams that handle these well figure out fast that the cause is not theirs, and spend the saved time on communication and failover instead of self-diagnosis.
What Actually Broke
Three different layers, three different failures, and that is the whole point.
The physical layer broke in May. Cooling is the layer nobody thinks about until it takes down compute and storage across a data hall. No software fix would have prevented it, because nothing in software was wrong.
The network layer broke in July. CloudFront was not physically damaged and its services were not down. The system that distributes routing configuration simply could not apply the update, so healthy infrastructure served errors because it did not know where to route.
The service layer broke in July as well. The infrastructure underneath was fine. The services running on top dropped out, and everything depending on them dropped with them.
Get Ayushmore’s stories in your inbox
Join Medium for free to get updates from this writer.
Three separate systems, each failing on its own terms. Fixing the cooling does nothing for config distribution. Fixing config distribution does nothing for the service layer.
Impact and Outcome
Across the three events, the blast radius reached well past AWS itself. The CloudFront failure alone rippled into identity, AI, and collaboration tools that most of their own users did not realize sat on AWS. The service-layer outage took down household-name consumer apps for over an hour. And the physical failure in May proved that a single region is still a single point of failure, no matter how modern the stack on top of it is.
AWS recovered each one and will publish its own analyses. But for the thousands of companies built on top, the outcome was the same every time: hours of degraded or failed service they did not cause and could not fix, and a scramble to figure out whether it was even theirs to fix.
Learnings
Design for layered failure, because the layers fail independently. Do not treat AWS goes down as one scenario. Treat physical, network, and service failures as three separate scenarios, because each hits differently and needs a different defense.
Stop betting on a single region. The May cooling failure is the reminder that the physical layer still exists and still fails. Multi-region is not paranoia, it is the defense for the one layer no software can save you from.
Degrade gracefully when a dependency is unreachable. The CloudFront failure was healthy infrastructure that could not route. If an upstream service being unreachable takes your whole product offline, that fragility is yours, not theirs.
Map your blast radius per layer. For each AWS layer you depend on, know what goes dark when it fails. If you cannot answer quickly, you do not yet know how exposed you are.
Instrument the is-it-us-or-them question. The single most valuable thing you can shorten is the time to know a problem is upstream. Build the signals that answer it in minutes, not half an hour.
The Deeper Pattern
Resilience is not one property. It is a stack of them.
When people call a system resilient, they usually picture it never failing. That is not what resilience looks like under real stress. Under stress, a complex system fails in whatever layer is weakest that day, and the layers are independent of each other. The thing that took you down in May has no relationship to the thing that took you down in July.
This is why hunting for a single root cause across the three is a trap. There is no shared bug. There are three separate systems, each of which failed on its own, and each of which needs its own defense. Every layer can betray you, and every layer betrays you differently.
Macro Learnings
You did not get hacked, and you did not get taken down. Someone you depend on did.
That is the hardest kind of failure to defend against, because you cannot fix it. You cannot re-architect AWS’s cooling. You cannot patch its config distribution. You can only decide how much of your reliability you are willing to hand to a single provider, and design for the day it fails in a way you did not predict.
Detection is not prevention. Knowing AWS is down does not keep your users online. The only real defense is deciding, in advance, what your system does when a layer you do not control gives out.
What This Means for the Rest of Us
You do not need to wait for the next outage to prepare for it.
Pick your most critical service. List every AWS layer it depends on, physical, network, and service. For each one, answer honestly: what happens to us when this fails, and how fast would we even know it was this and not us. If you cannot answer, that is your first piece of work, not more redundancy you have never tested.
Because the next AWS outage is already coming, and the reporting will call it unprecedented, and it will be a layer nobody was watching. The failure that hurts is never the one you planned for. It is the one you assumed someone else was handling.
On the journey to unbreakable software. Until next week, keep building robustly.
