Postmortem: E-commerce Website Flash Sale Outage (Fictional)
Issue Summary
In the weeks leading up to the widely anticipated flash sale, we built enormous hype, playfully claiming we would “break the internet.” When the clock hit 2:00 PM PST on February 25, 2025, people swarmed the site as if it were their final chance to shop, leading to unanticipated technological issues. And our site went down in flames. For 90 agonizing minutes, until 3:30 PM PST, most users couldn’t shop, add to cart, or check out. They were stuck staring at spinning wheels while we freaked out. The servers were swamped, and the database? It shut down under the chaos.
Timeline
We’re at the office, drinking coffee, feeling proud, and then all this hell breaks open. Our phones start vibrating like crazy with notifications, and the dashboards light up red like a Christmas tree gone mad.
Here’s how it went down:
- 2:00 PM PST: Alerts kick off and server loads high; the site’s getting slower than my grandma on a Sunday drive.
- 2:05 PM PST: We’re jumping out of our chairs, rushing to figure out what’s blowing up.Someone’s coffee hits the floor in the chaos.
- 2:10 PM PST: “Is this a DDoS?” The traffic was insane, we worry, sweating over the enormous load.
- 2:20 PM PST: Nope, simply a horde of deal-hungry shoppers. Way more than we imagined. Oops
- 2:30 PM PST: We put additional web servers into the fight, but the database is still gasping.
- 2:45 PM PST: Our DBA exclaims, “It’s the database, you idiots!” — facepalm moment.
- 3:00 PM PST: We run to spin up examine clones, holding our breath.
- 3:15 PM PST: Response times creep back to decent — we’re quietly praying it holds.
- 3:30 PM PST: The site’s alive! We’re exhausted but cheering.
Root Cause and Resolution
Root Cause
We did not expect a huge circus erupting from this flash sale.Our equipment wasn’t prepared for that sort of pounding, thus the traffic attacked us like a runaway truck. The web servers were wheezing, but the database? Holy crap, that was the real disaster — folded faster than a lawn chair in a windstorm.
Resolution
We jumped into panic mode. Tossed numerous additional web servers into the turmoil to hold things together, but the database was still choking. Then Sarah — she’s our database whisperer — yelled, “Read replicas, now!” We got them up, and they took the heat off like a refreshing drink on a hot day. By 3:30 PM PST, we were back online. Barely. I’m still nervous from it.
Corrective and Preventative Measures
Corrective Measures
We’re juicing up our gear to endure insane traffic and building better traps to catch problems before they explode in our faces.
Preventive Stuff to Do
- Auto-scaling web servers: They’ll grow with the crowd — let ’em expand when the crowd comes up.
- Better alerts: Real warnings when the CPU’s crying or the database is about to blow.
- Load tests: We’ll stress-test before big sales — no more guessing games.
- Database tweaks: Queries and indexes get a tune-up for speed under pressure.
- CDN hookup: Cache the easy stuff and lighten the load.
A Note to Our Users
We’re very sorry you had to cope with this disaster. We can envision you cursing at your displays, and we feel bad about it. Nothing sucks more than a crash mid-shopping frenzy.We’re working our butts out to make sure this doesn’t repeat — promise! Thanks for sticking with us.
Get Gemechis Chala’s stories in your inbox
Join Medium for free to get updates from this writer.
By: Gemechis Chala, Feb 2025
