Breaking Prod #23: The Firewall Upgrade That Blocked 000
Subtitle: A routine change at an Australian telco cut off emergency services for 13 hours. The bug was not the firewall.
Routine is the word that shows up right before every serious outage.
The change is routine. The upgrade is routine. The rollout is routine. Everyone signs off. The ticket closes. Nothing feels risky, until it is.
In September 2025, a routine firewall upgrade at Optus quietly cut off emergency service calls across Australia for 13 hours. People could not dial Triple Zero. No ambulance. No fire. No police. Nobody at Optus knew until customers started calling in to say the calls were not going through.
Nothing broke visibly. Every dashboard stayed green. The change ticket closed clean. Underneath all of that, one of the most safety-critical routes in the country was silently dropping calls.
What Actually Happened
At 3 AM local time, Optus network engineers kicked off a scheduled firewall upgrade. Standard maintenance. On the calendar. Through review.
The upgrade finished. Every check passed. The firewall was on the new version. Standard tests were green.
What nobody caught was that the new config handled certain traffic types differently than the old one. Regular voice and data still routed fine. But the rules that handled emergency service calls, calls to 000, had been quietly regrouped in a way that dropped them at the firewall boundary.
For 13 hours, anyone in Australia who dialed 000 through affected Optus infrastructure got dead air or a fast busy tone. No warnings. No alerts. No page.
The failure was found the way silent failures usually get found. Customers complained. Someone tried to call an ambulance. It did not go through. Someone at Optus finally checked. The emergency routes had been down since the upgrade.
Why It Broke
Four layers should have caught this. None of them did.
The change validation tested standard traffic. Emergency traffic was assumed to behave the same way. It did not.
The monitoring watched the firewall. Was it up? Yes. Was it processing traffic? Yes. Was anyone checking whether emergency calls were actually reaching their destination? No.
The change approval workflow treated the upgrade as normal maintenance. Emergency routing was not flagged as needing special review.
The rollback plan existed but assumed someone would notice. When the failure is silent and only affects one narrow slice of traffic, nobody notices until customers do.
What Would You Have Done?
Say you are the SRE lead at a telco. You find this failure mode a week before the upgrade. What is your fix?
A) Add a manual QA pass on emergency routes before every rollout. Someone checks the 000 path is working before the change goes live.
B) Add automated end-to-end tests for emergency service paths as part of the change gate. The pipeline cannot ship the upgrade unless the 000 test passes.
Get Ayushmore’s stories in your inbox
Join Medium for free to get updates from this writer.
C) Ban firewall upgrades to production without a live emergency-services traffic test as a gating requirement. Not a checkbox. Not a synthetic test. An actual live call through the actual production path, verified end to end, before the change is approved.
The correct answer is C.
Manual QA passes get skipped when the team is under pressure. Automated tests can pass without touching the exact rule that changed, because the test path is not always the production path. The only thing that reliably catches this class of failure is a rule that says the change cannot ship without a verified live test of the critical route.
That is a hard bar. It slows changes down. It costs coordination time. But 13 hours of dead emergency calls costs more.
The Bigger Pattern
Change management usually assumes uniform risk. Every change gets the same review, the same testing, the same approval.
That assumption is fine for most systems. It breaks the moment your service carries paths where failure has consequences beyond uptime.
Every telco has an emergency services path. Every hospital IT system has a life safety path. Every payment platform has a settlement path. Every trading system has a risk limit path. These paths do not just fail differently. They fail with real world consequences no dashboard can capture.
Most engineering organizations do not have separate change workflows for these paths. They should. The process for shipping a UI tweak is the same as the process for touching the 000 route. Same review, same testing, same approval speed.
But the risk profiles are not the same. Treating them the same means the safety-critical ones eventually get burned by the shortcuts that are fine for the low-stakes ones.
Not all traffic is equal. Some paths carry human life. Those paths need higher rigor, not the same rigor.
Micro Learnings
- Routine is the word engineers use for a change they have not fully thought through.
- Monitoring the system is not the same as monitoring the outcome. Your firewall can be up and healthy while the traffic it is supposed to route is dying.
- Test the exception paths as hard as the happy path. The failures live in the exceptions.
- Safety critical routes deserve their own change approval workflow. Same review as everything else means the same corners get cut.
Macro Learning
The failure at Optus was not the firewall. The firewall did exactly what it was configured to do. The failure was the assumption that all traffic is equal.
Every reliable system has paths with consequences beyond uptime. A dropped payment. A missed trade. A blocked emergency call. When those paths fail, the impact is not measured in service credits or downtime minutes. It is measured in the real world event that could not happen because the system said no.
If lives depend on a path, routine is not a word you get to use. If the failure of a path cannot be undone with a rollback, that path does not belong in the same change workflow as the ones that can.
Every team has a 000 path. A route through their system where quiet failure has consequences the dashboard will never show. Find yours before your customers do.
What is the critical path in your system that would fail silently if a routine upgrade broke it?
Follow along for more.
