Your Outage Will Happen Again Unless You Write the Postmortem Like This

Most teams write a postmortem after a bad outage. Fewer teams stop the same outage from coming back. The gap between those two things is not effort or goodwill. It is a set of specific choices about how the document is written, what counts as a fix, and what happens to the fixes after the meeting ends.

This article is about those choices. It looks at what the people who run some of the largest systems on the internet have published about writing postmortems, and at what their own public postmortems show when the same kind of failure returns. It turns that into ten rules, a template and a checklist.

The sources are primary and were all read for this piece: Google’s Site Reliability Engineering book (the “Postmortem Culture” chapter and its example postmortem), the Site Reliability Workbook chapter on postmortems, public incident reports from Cloudflare, Amazon Web Services and Google Cloud, GitLab’s 2017 database postmortem, John Allspaw’s “Blameless PostMortems and a Just Culture” from Etsy’s engineering blog, Richard Cook’s “How Complex Systems Fail”, and the postmortem guidance that Atlassian and PagerDuty publish. The full list is at the end.

One thing was run: a small Python script that lints postmortem action items for the problems the SRE Workbook calls out, such as…