What caused the large AWS outage?
Hey inquisitive minds,
This article is a simplified version of many detailed reports and technical posts I’ve read across the internet — about one of the most surprising cloud failures in recent times.
On the night of October 19–20, 2025, Amazon Web Services (AWS) faced a major outage in its US-East-1 region (Northern Virginia). The problem lasted for more than 12 hours and affected many other AWS services and customers around the world.
The surprising part?
It all started because of a small bug in the automation code that managed DNS for Amazon DynamoDB, a popular NoSQL database service.
What Actually Happened
DNS — The Basics
DNS (Domain Name System) works like the phone book of the internet.
When a computer wants to connect to “dynamodb.us-east-1.amazonaws.com”, DNS translates that name into an IP address.
If DNS stops working, services cannot connect — even if the servers themselves are healthy.
How DynamoDB’s DNS System Worked
DynamoDB used a system made up of:
- A DNS Planner — this part watched server health and traffic load, then created a plan showing which IPs should be active.
- Three DNS Enactors — these followed the planner’s instructions and updated AWS’s DNS records. Each Enactor ran in a different Availability Zone for safety.
The Core Bug — A Race Condition
Here’s the simple timeline:
- Enactor A started applying an old DNS plan but got delayed.
- Meanwhile, the Planner made a new plan since Enactor A was slow.
- Enactor B quickly applied this new plan and then deleted old ones (cleanup).
- Later, Enactor A finally finished its old plan — but its data was outdated, so it accidentally overwrote the new DNS data.
- The cleanup process then removed the plan Enactor A had just applied, leaving no DNS records for DynamoDB in that region.
Result:
All requests to DynamoDB in US-East-1 failed because the endpoint name pointed to nothing.
Get Aarthi_Honguthi’s stories in your inbox
Join Medium for free to get updates from this writer.
This is called a race condition — two parts of a system working at the same time and interfering with each other.
The Chain Reaction — How One Bug Broke Many Services
Because DynamoDB is used by many other AWS services, the failure caused a domino effect:
- New EC2 instances couldn’t start because the control system that tracks instance details depends on DynamoDB.
- AWS Console and IAM (login system) were affected — users couldn’t log in or manage their resources.
- Lambda and Load Balancers failed because they rely on DynamoDB data for scaling and health checks.
- Even after DNS was fixed, a long queue of pending operations slowed down recovery.
So a single DNS issue turned into a regional meltdown, proving how deeply connected cloud systems are.
Why This Incident Matters
- Even the biggest cloud company can have bugs that cause massive failures.
- The cloud is not magic — it’s still made of software that can break.
- Many systems depend on others in ways we don’t always realize.
- The US-East-1 region is the oldest and largest AWS region, so its outage affected customers everywhere.
What stood out most to me is how one small automation bug could affect giants like Netflix, Reddit, and even AWS itself. It shows that understanding system design, dependencies, and failure handling is just as important as learning syntax or algorithms.
So as beginners, our goal shouldn’t just be to build fast — but to build thoughtfully. Every line of code we write contributes to something bigger, and with the right mindset, we can create systems that don’t just run — they endure.
Thank you for reading!
If you enjoyed this breakdown, subscribe or follow for more simplified tech stories and system design insights. 🚀
