Sitemap

Post post mortem

A short story on engineering blinkers

--

Last year, on a Friday, I had a presentation of the benefits assessment tool I was a Product Manager for. It was its third “official opening”, but hey you don’t complain when you get visibility 😉 I got two ministers there, along with all that ensues at that level: local officials and, obviously, journalists.

It just happened that the weekend right after I had to help friends to empty a house full of movie shooting gear, so I was about to leave for three days of cleaning up in the countryside.

Press enter or click to view image in full size
That’s the kind of stuff we had to clear. It did take a while!

I had given the usual presentation and shaken hands and given a couple interviews while the ministers were doing their job (bringing people in and congratulating officials for stuff they have just been briefed about).

The things you do to satisfy partners…

All of that was combined with the opening of a nationwide ad campaign. After 2 years of trying, we had finally nailed radio spots and sponsored posts on social networks. My team had to create a specific subdomain to add required third-party trackers without contradicting our privacy policy and it had been quite a bit of work but once again: you don’t complain.

So we’re moving all that heavy cinema equipment around and we have only two days to finish and I’m nosebleeding and obviously that’s the BEST MOMENT to get paged at 1pm on a weekend.

Two of my sensors are down, the app doesn’t seem to come back up on its own, I’m in a place with lousy internet. I try to load the website immediately, it does render. But a couple minutes later, I get a second text notification confirming that the computation component is still down… I immediately call another teammate and let him finish lunch while I take a first look, and we agree to rotate throughout the afternoon.

Get Matti Schneider’s stories in your inbox

Join Medium for free to get updates from this writer.

So there I was, spending 20 minutes foraging through logs while bringing the computation engine back up, writing a collaborative assessment and future postmortem amidst cardboard packaging and film equipment. The outage was about 15 minutes long total, in two sessions, and was invisible to most users as our caching strategy was valid and the UI was still served; some users only got an error telling them that the computation was not possible at that time. The logs showed that we had had as many hits in 10 minutes as we were used to in a day.

But for us, there was still no way to understand what happened and why there was a peak in activity, so it was likely to happen again anytime. I had the sensors dashboard full size on my screen for the whole afternoon and evening, checking it every time I was moving stuff in and out of the room, ready to manually restart the server anytime.

We managed to clean everything, there was no other outage over the weekend, and it was all fine even though it was stressful. My team was all up and available, we protected the one teammember who was off, our community manager did a great job at keeping users updated, and we made sure to have at least one person pageable with a reliable internet connection at any time of the day or night.

The After

I spent the best of Monday analysing logs to determine if we had suffered from a DoS attack, a DDoS, a major component failure… I measured packet frequency, payload origin, found no pattern, checked worst offenders against known blacklists… And then, mid afternoon, our series-watching, glitter-loving, TV shows-fan CM casually drops:

“Hey, I think I found it. We’ve been on national TV news.”

🤦‍♀️

I watch so little TV I did not even know one of the main channels does their noon news edition 15 minutes earlier than others. While I was raging out loud and looking for attacks and technical failures, my CM was watching the replays of all TV news of the day we had the outage inbetween replying to users, and he found the reason without any effort. We had just been hit by thousands of very legit users, drawn to our app by a mention on mass media.

So hey, that’s one small illustration of why it’s worth hiring for diversity. I don’t know shit about TV, and apparently neither did the other engineers in my team. I could have wasted even more time and energy trying to figure out what happened: I had eliminated that option as the hours did not match, and I was looking in the dimensions that were the most tangible for me and was expecting our press partners to let me know if we had major outlets appearances. They failed, but my team won.

That’s called “scaling”, apparently.

And that was the entirety of the necessary technical fix.
👈

--

--

Matti Schneider
Matti Schneider

Nomadic transdisciplinary engineer delivering public digital services @OpenFisca. Ex core @BetaGouv @MesAides @GovtNZ. 🇫🇷 ? → @matti_sg_fr.