<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Hart on Medium]]></title>
        <description><![CDATA[Stories by Michael Hart on Medium]]></description>
        <link>https://medium.com/@hart-michael?source=rss-c7ee58e90e72------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OG5gRH1AzDChgyigIeS4bA.jpeg</url>
            <title>Stories by Michael Hart on Medium</title>
            <link>https://medium.com/@hart-michael?source=rss-c7ee58e90e72------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:36:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hart-michael/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[On Canadian Patriotism and Remembrance Day]]></title>
            <link>https://hart-michael.medium.com/on-canadian-patriotism-and-remembrance-day-5756a7eb5f83?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/5756a7eb5f83</guid>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Tue, 11 Nov 2025 15:29:50 GMT</pubDate>
            <atom:updated>2025-11-11T15:30:05.010Z</atom:updated>
            <content:encoded><![CDATA[<p>On May 8, 1945, Canadian soldiers from <a href="https://en.wikipedia.org/wiki/Princess_Patricia%27s_Canadian_Light_Infantry">Princess Patricia’s Canadian Light Infantry</a> rolled through the city of Haarlem in the Netherlands, officially liberating the city from the German occupation. The Germans had surrendered three days earlier, and the <a href="https://www.youtube.com/watch?v=i0QeQRbq47M&amp;t=2s">Dutch citizens welcomed the Canadians with great rejoicing</a>. Somewhere in the thick of the celebration was a thirteen year old boy named Gerbrand. He and his family survived the war intact, physically at least, but so many of his neighbours and friends did not live to see the liberation. That young man is my Dad.</p><p>In 1951 my Dad immigrated to Canada, to the same country that liberated him, to a land full of opportunity that welcomed him and over 150,000 of his fellow countrymen. There he learned a new language, started a business, raised a family, and fully adopted Canada as his new country by becoming a Canadian citizen.</p><p>I knew all this growing up, but it took three separate events to make it really come clear and meaningful. The first was when hiking through the town of Jougne, France, we came upon a memorial to all the French soldiers from Jougne who had died in the First and Second World Wars. What caught my attention is that every town and city in Canada, including my tiny village of St. George, has similar plaques and memorials with all the names of those who went across the ocean to fight and die for freedom. Seeing the same memorial in a foreign country so far away was a shock, and made the distance suddenly seem a lot less.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oA4ol4hNf7cJ_YApTW1T0w.jpeg" /><figcaption>Memorial plaque in St. George, Canada. Photo by author.</figcaption></figure><p>The second event was when my wife and I, somewhat accidentally, came across the <a href="https://www.cwgc.org/visit-us/find-cemeteries-memorials/cemetery-details/2002900/bergen-op-zoom-war-cemetery/">Bergen-op-Zoom War Cemetery</a> in the Netherlands, where 1197 casualties from the two World Wars are buried, many of them Canadian. Walking through the rows of graves on a beautiful sunny day so far removed from war, reading the names of young men who weren’t even eighteen years old yet, and seeing the clustered graves of entire bomber crews, was indescribably emotional.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KFxgYd1tQuO0BbkgdNJEiA.jpeg" /><figcaption>War Memorial in Jougne, France. Photo by author.</figcaption></figure><p>It wasn’t until President Trump started threatening Canada with the 51st state rhetoric that I, along with millions of other Canadians, started to think deeply about why we love our country. Canadians are generally understated in their patriotism, in part to avoid acting like our southern neighbours. But when Canadians are threatened, even in supposed jest, we are roused from our laid back nature and start taking notice.</p><p>My patriotism is quite personal. Canada’s army, made up of ordinary citizens from all over the countryside, crossed the ocean and liberated my Dad’s family from the horrors of fascism. The same country welcomed my Dad, his brothers, and his fellow citizens a few years later and allowed them to make a new life for themselves, raise families, and live in peace. The men and women whose names are on the plaques in St. George and Jougne and so many other cities made Canada possible, and I, we, owe them a great debt.</p><p>Today is Remembrance Day where we remember the sacrifice our Canadian ancestors made to give us a free country, and that day has become a lot more meaningful in the past few years. Charlie Angus summarizes Remembrance Day in Canada in <a href="https://charlieangus.substack.com/p/canadas-100-days-a-remembrance-day">Canada’s 100 Days: A Remembrance Day Reflection</a> better than I can:</p><blockquote>The lesson … is that as the world becomes increasingly unstable and dangerous, Canadians need to remember the cost that other generations were forced to pay so we could live in peace and freedom.</blockquote><blockquote>Whether it was on the North Atlantic, bombing runs over Berlin, at Ortona, Falaise, the Scheldt, the Netherlands, Korea, Medak Pocket, or Kandahar, young Canadians always showed up and did their part.</blockquote><blockquote>And more.</blockquote><blockquote>They did it to free other people. But they also did it so that a new generation wouldn’t have to go through what they suffered. That is the meaning of Remembrance Day: to protect and preserve that legacy.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5756a7eb5f83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conway’s Law, DevOps, and Your Source Code]]></title>
            <link>https://hart-michael.medium.com/conways-law-devops-and-your-source-code-6b85cd89496f?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b85cd89496f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[conways-law]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Wed, 24 Jan 2024 01:41:04 GMT</pubDate>
            <atom:updated>2024-01-24T01:41:04.532Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fFn3kWVRFyturHaa37hIDA.png" /><figcaption>Conway’s Law, by ChatGPT</figcaption></figure><p>The layout of your source code repositories will affect your company’s org chart, and if you get the first wrong, your ability to deliver software will suffer.</p><p>Conway’s Law states that your software architecture will naturally match your company’s org chart. I expand the law and believe that your <em>source code repository structure </em>will naturally influence your org structure, and if you are practicing “DevOps”, the wrong structure will result in chaos.</p><p>The code that defines the infrastructure of your service, commonly known as Infrastructure-as-Code, belongs in the same source code repository as the rest of your service.</p><h3>Conway’s Law</h3><p>Let’s quickly recap what Conway’s Law is all about. Melvin Conway summarized the law in <a href="https://www.melconway.com/Home/Committees_Paper.html">his seminal paper</a> as:</p><blockquote>Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.</blockquote><p>Martin Fowler <a href="https://martinfowler.com/bliki/ConwaysLaw.html">expands on the law</a> with more detail:</p><blockquote>Conway’s Law is essentially the observation that the architectures of software systems look remarkably similar to the organization of the development team that built it. It was originally described to me by saying that if a single team writes a compiler, it will be a one-pass compiler, but if the team is divided into two, then it will be a two-pass compiler.</blockquote><p>Proving Conway’s Law is well outside the scope of this article, <a href="https://en.wikipedia.org/wiki/Conway%27s_law">the Wikipedia article</a> has a list of supporting evidence for anyone wishing to dig further. For today’s purposes we assume Conway’s Law is true.</p><h3>Reactions to Conway’s Law</h3><p>Martin Fowler explains three reactions to Conway’s Law, which matches what I’ve seen in practice.</p><p>The first reaction is to ignore it and assume the law doesn’t apply, which is much like ignoring gravity and then stepping off a cliff. A worse reaction than ignoring it is to fight against the law. I heard a senior leader once rant “We should <em>fight</em> Conway’s law!”. Instead of ignoring gravity you’re going to fight it and try to jump to leave earth’s orbit, with dysfunctional or even catastrophic consequences to your organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L-ZpoaHXXFf3TNVO" /><figcaption>Saturn V would like a word about the effort required to fight the law of gravity. Source: <a href="https://en.wikipedia.org/wiki/Saturn_V">Wikipedia</a>.</figcaption></figure><p>The second reaction is to accept the Law and ensure your org structure and architecture don’t clash. In Fowler’s words “<em>recognize the impact of Conway’s Law, and ensure your architecture doesn’t clash with designers’ communication patterns.</em>”</p><p>The third reaction is to use the <em>Inverse Conway Maneuver</em>. I really love this phrase. It was first coined by Johnny LeRoy and Matt Simons in <a href="https://jonnyleroy.com/2011/02/03/dealing-with-creaky-legacy-platforms/">this article</a> in December 2010. Their thinking was that, paraphrasing Einstein, “<em>you can’t fix a problem from within the same mindset that created it</em>” and if your software system is dysfunctional, first fix the org chart, and then fix the system. That is, your org chart is what caused the dysfunction in the system architecture, so don’t lean on the same org chart to fix the architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZvAP0P-tbPun7IkM" /><figcaption>ChatGPT’s drunk attempt at the Inverse Conway Maneuver.</figcaption></figure><p>Taking the Inverse Conway Maneuver a step further when building a new system is to first plan the desired architecture of your system, and then change your org structure to match the desired architecture. That way you’re not fighting the law but using it to your advantage, and your new architecture will be implemented with little friction. That sounds fairly naive at first glance, and the realities of your teams needs to be taken into consideration. Martin Fowler again has a great quote:</p><blockquote>I still remember one sharp technical leader, who was just made the architect of a large new project that consisted of six teams in different cities all over the world. “I made my first architectural decision” he told me. “There are going to be six major subsystems. I have no idea what they are going to be, but there are going to be six of them.”</blockquote><p>In the early days at Arctic Wolf we talked frequently about Conway’s Law, and purposely kept a monorepo (single repository with all the source code in it), a monolith application (a single binary), and a flat organization for as long as we could, and probably a bit longer than we should have. But in the early days of a startup, when product market fit is still questionable and big changes come fast and frequently, having a simple architecture and organization allowed us to be very flexible and move very quickly.</p><h3>DevOps</h3><p>Before I can talk about source code, I need to briefly discuss DevOps. While I’ve lived DevOps practices well before it was named, I dislike the term because it’s nebulous, not well defined, frequently completely misunderstood, and has been taken over by marketing campaigns to the point of almost meaninglessness. That said, AWS’ <a href="https://aws.amazon.com/devops/what-is-devops/">definition</a> is fairly succinct:</p><blockquote>DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity…</blockquote><p>The key is that DevOps is <em>philosophies</em> and <em>practices</em>. DevOps is <em>not</em> taking the old “Operations” team or “IT” team and re-labeling them as the “DevOps Team” and carrying on without any behavior change. Unfortunately many organizations do this, create a “DevOps Team”, and don’t understand why their velocity hasn’t increased, but exploring that is far outside the scope of this article.</p><p>The other way to describe a DevOps philosophy is to give a team the mandate “You build it, you run it” and give them the tools and responsibilities and guardrails to do that. The result is a team that can move quickly. I have <a href="https://hart-michael.medium.com/why-you-need-continuous-deployment-93d7b5936523">written previously</a> about my experience at Arctic Wolf where a DevOps philosophy led to hundreds of code deploys per day, and many other organizations have seen similar results.</p><h3>Infrastructure As Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xKMHs4QY4DozDaIE" /><figcaption>Infrastructure-as-Code, showing Terraform and CloudFormation. Screenshot by author.</figcaption></figure><p>The code used to define the deployment of your service, commonly called Infrastructure-as-Code (IaC), belongs in the same source code repository as the rest of your application code. If the IaC is kept separately, because of Conway’s Law your org will end up with a separate Operations-type team maintaining the IaC, and you will lose the velocity a DevOps practice promises.</p><p>“Hold on!”, the confused reader might be thinking, “How did we start with Conway’s Law, take a detour into DevOps, and then make a sharp dive into Infrastructure-as-Code?” The three are related, bear with me.</p><p>A software development team practicing DevOps “you build it, you run it” is responsible for a few different aspects of the service’s lifecycle:</p><ol><li>Writing the application code that makes up the service</li><li>Creating the automated tests, both unit and system/integration tests</li><li>Developing the IaC that makes up the service in production</li><li>Creating dashboards and alerts to observe the service.</li></ol><p>In many organizations the first two are stored in one repository, the third in a separate repository frequently managed by a different team, and the fourth is a manual afterthought.</p><p>A quick side note, by IaC I mean everything that is needed to deploy the service to development, test environments, and production. That’s Docker files, Helm charts, Cloudformation or CDK, Terraform, shell scripts, CI/CD definitions, SQL scripts, etc.</p><p>Let’s dig into a few more definitive reasons why the application code, automated test code, and IaC belong in the same repository.</p><ul><li>IaC <em>is</em> code. It’s not anything special or strange or weird that belongs elsewhere. In my experience some developers seem to be fine with maintaining complex application code and strange test constructs (have you <em>seen</em> TDD frameworks?) but don’t want to touch IaC code like Terraform, leaving it for a DevOps team.</li><li>The service is incomplete without the deployment definitions living alongside it. I’ll go as far as to say that the application code is mostly useless without the IaC, it can’t actually function as intended. If your service is hosted in Kubernetes on AWS and has, for example, a DynamoDB table — the application code by itself is useless without the Docker file to package it, the Helm chart to deploy it, and the CloudFormation or Terraform to define the DynamoDB table and the IAM artifacts required to grant permissions.</li><li>It is far easier for a developer to reason about a service when all the code lives in one repository. It’s simpler to make changes to a complex service when the IaC lives in the same repository, especially when those changes span application and infrastructure code, like when changing a DynamoDB definition, or adding a new API call that requires an IAM policy change.</li><li>Versioning is much simpler when all the artifacts live in one repository, especially when deploying continuously and from the main branch.</li><li>The line between “application” or “service” code and IaC is increasingly becoming blurred. A traditional service from a decade ago might be a single binary that’s deployed by hand on VM’s or even physical hardware. A modern service could be a serverless application that cannot run outside of the cloud execution environment, with IAM roles giving permission to access AWS’s DynamoDB or Google’s BigTable, the structure of which is defined in IaC. The definition of the DynamoDB or BigTable is really no different from a SQL database definition which commonly lives beside the application code. Or the service could be made up of step functions where some of the application workflow is now defined in IaC. The blurring of those lines make it imperative to keep all the code in one repository.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/0*NWafUwou6Of_GJ2V" /><figcaption>Diagram of the Immelmann Turn, a WWI maneuver. Source: <a href="https://en.wikipedia.org/wiki/Immelmann_turn">Wikipedia</a></figcaption></figure><p>The best reason to keep all the code in one repository is what happens if you don’t. Leaning on the Inverse Conway Maneuver, if you separate the application code from the IaC needed to deploy it, you <em>will</em> end up with a separate team owning the IaC, and be right back in the old world of having a separate Development and Operations teams, with all the dysfunction associated with that. Conway’s Law states that ending up with two teams is inevitable.</p><p>A brief tangent about my definition of “source code repository” — I’m really using that term as a proxy for a logical grouping of code, with a set of permissions wrapped around it. Typically this is implemented as a single GitHub or GitLab repository, but it might be, if you’re particularly unlucky, a subdirectory in something like Perforce.</p><p>What about shared services and the likes of a Platform Engineering group? There is definitely a need for IaC defining services like shared networking, shared Kubernetes clusters, security guardrails, etc, and that typically falls on the shoulders of specialist teams. As a company grows, splitting out a Platform Engineering and Security team is a natural evolution and keeping the IaC for shared services in separate repositories is usually the right thing to do. As long as those teams don’t become the DevOps team, and those specialist teams can still treat their own source code in a DevOps fashion.</p><h3>Conclusion</h3><p>Keeping Conway’s Law top of mind when making architectural decisions, planning source code repository layouts, and making organizational changes is key to setting up your organization for success. Ignoring or fighting against Conway’s Law will result in a dysfunctional organization and architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b85cd89496f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why You Need Continuous Deployment]]></title>
            <link>https://hart-michael.medium.com/why-you-need-continuous-deployment-93d7b5936523?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/93d7b5936523</guid>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Tue, 19 Dec 2023 14:15:01 GMT</pubDate>
            <atom:updated>2023-12-19T14:15:01.607Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FpPz9UUOeZ6cwyOj" /><figcaption>Manual labor vs automation, by ChatGPT</figcaption></figure><p>A few weeks ago a colleague described a software development team that maintained manual testing and deployment practices. Deployment was so complex that the original developers wore completing the difficult deployment as a badge of honor. The question was: how can that team move toward a Continuous Deployment model? I confess to a shudder of horror when I read that, having lived that scenario in previous organizations. While moving from a manual complex deployment scenario to an automated one is a large technical challenge, the cultural shift required for a successful change is far more important.</p><p>First we should cover some terms:</p><ul><li><strong>Continuous Integration</strong>, the CI part of what’s commonly referred as “CI/CD”: CI is the practice where developers merge code to the main branch frequently, and automated tools build and test the code after every merge.</li><li><strong>Continuous Delivery</strong>: Changes are available to be deployed to customers but the act of deployment may be a manual step, often requiring approval.</li><li><strong>Continuous Deployment</strong>, traditionally the CD part of CI/CD: practice where every change that passes automated tests is deployed to be available to your customers.</li></ul><p>There is a lot of writing on <em>how</em> to implement Continuous Deployment using various technologies, but far less on the <em>why</em>, which I’ll explore in the remainder of this article.</p><p>Two publications stand out as recommended reading, and I’ll build on them. The first is <a href="https://itrevolution.com/product/the-phoenix-project/">The Phoenix Project</a>, a fantastic fictional story that introduces the DevOps concept, which includes Continuous Deployment. The second is the <a href="https://continuousdelivery.com/">Continuous Delivery</a> website, a great resource linking to peer-reviewed studies and further reading.</p><h3>Examples</h3><p>I’ll share three examples of services I’ve worked on, each with a very different state of Continuous Deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wcEAzBnV3aT_15Fv" /><figcaption>My career in three pictures, by ChatGPT</figcaption></figure><p>I’m not sharing these to shame or blame anyone, on the contrary each team was full of smart individuals trying to do the right thing with the tools and limitations we had at the time. We have collectively learned a lot about how to deliver software in the past two decades, and we can use these examples to explore why you want Continuous Deployment for your services.</p><p>Early on in my career I supported a development team which had scheduled deployments on a prescribed day of the week, inevitable rollbacks later that day, and a hotfix deploy later in the week. That sounds like a critique but it’s not. This team was trying to deliver business value on a weekly cadence — almost unheard of in the era of quarterly or even yearly releases. This took place in the early 2000’s, automated testing frameworks were almost non-existent, and the Subversion source control tool had just been released. The team had no automated builds or tests, and executables were copied by hand (usually by me) to the target servers. As I write that I can’t help but be impressed that the system worked at all. It’s easy to pick out what’s wrong, but what did they get right?</p><ol><li>Change sets were small, less than a week’s worth of changes by a few developers.</li><li>Feedback from the customers was instantaneous.</li><li>Rollbacks were possible and tested prior to deployment.</li></ol><p>A service I worked on at BlackBerry, which is no longer in use, had quarterly releases with the traditional dev-complete code lockdown, a long testing phase, and a very complex and manual deployment procedure in the terrible 0200–0400 weekend change window. In spite of the thousands of hours of testing for each release, almost every deployment resulted in a rollback or multiple hotfixes to address issues that came up. The biggest challenge with this service was the sheer size of the change sets, containing three months of work for twenty or more developers. No one could possibly understand the scope of the changes, and that made every deployment risky in spite of a large and skilled QA team.</p><p>Compare that to my latest experience at Arctic Wolf, where we implemented Continuous Deployment, and some services were re-deployed almost hourly with the latest changes. The system wasn’t perfect, we definitely had bugs escape into production, but with a few exceptions we found it was almost always easier to fix the bug and deploy that change rather than try to roll back a deployment.</p><h3>What Continuous Deployment Gives You</h3><p>I could write an entire book on the advantages of implementing Continuous Deployment, but for the reader’s sake let’s explore Risk, Quality, Cost, Speed, Happiness and Security.</p><h4>Less Risk</h4><p>The change management philosophy a few decades ago, which is still pervasive in some older organizations and in use at BlackBerry in the early 2000’s, was that change is the enemy of stability. To maintain a service with five nines of uptime, changes had to be limited and carefully gated. Unfortunately that came in direct conflict with the Product &amp; Dev team’s need to ship new functionality or fix bugs. In my second example that conflict was in clear view — the Operations team, driven by strict SLA’s, had a mandate to provide extremely stable services. The Product &amp; Dev teams had to ship new functionality, and the result was infrequent but very large changes, and inevitable conflict between the two groups.</p><p>With Continuous Deployment, the change sets should be small but frequent, and new functionality is introduced in small well-understood batches. When something does inevitably go wrong, the teams can very quickly narrow down the exact change causing the issue. Smaller change sets reduce the risk of large outages.</p><p>To achieve Continuous Deployment there is an underlying assumption that your software can be deployed in a rolling manner with blue/green deployment and zero downtime. That also implies your infrastructure is defined with Infrastructure-as-Code and doesn’t require manual intervention to deploy. That also reduces risk, as deployments are automated, reproducible, and testable.</p><h4>Higher Quality</h4><p>A solid dependable suite of automated tests are a key requirement to implement Continuous Deployment. If your teams are committed to keeping those tests up to date, regressions should be caught before they are deployed, resulting in higher quality software. I once overheard a team state “we don’t trust our tests” and had reverted back to manual testing, and deployed once a week or less. Because they tested manually, they struggled with both velocity and quality. Once they had their test suite back under control their velocity and quality came back up.</p><h4>Lower Cost</h4><p>Multiple studies show that bugs become exponentially more expensive to fix the further they make it from developer to customer. There is definitely an upfront cost to developing automated testing required for Continuous Deployment, but bugs will be caught earlier in the development cycle, resulting in lower overall cost in the long run.</p><h4>Faster Time to Market</h4><p>Compare my BlackBerry example with the other two, where it could take up to six months for a customer to see a new feature or bug fix. In the other two examples the teams could provide new features far more quickly, in weeks or even hours.</p><p>One of the four <a href="https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance">DORA</a> metrics (see <em>Measuring Success</em> below for more details) is “Lead Time for Changes”, or the time it takes from committing code to it being deployed to production. The lower that number is, the better, assuming the other metrics are also in line. An organization that can deploy a change to customers in hours is far more successful than one that takes six months.</p><h4>Happier Teams</h4><p>One of my favorite Slack reactions was a “#YOLO” icon that my colleagues would use to indicate they had reviewed and approved a GitHub pull request. It was used partially in jest because everyone cared deeply about the services we were deploying, but it was also in part a nod to the CI/CD pipeline and the testing infrastructure, and the trust we had in it. My stated goal as a leader was that a developer should be able to merge their code change and then step away for lunch, trusting that the CI/CD pipeline would build, test and deploy the change correctly, and if things went badly wrong either rollback the change or call them via PagerDuty. That’s a lofty goal and I don’t think we ever got all the way there. In reality my colleagues and I typically tracked changes as they were rolled out, but I wanted us to have the confidence that the automation would just work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*aEIew7RAp5AU0_iZD9dYYg.png" /><figcaption>Happy teams #yolo, by ChatGPT</figcaption></figure><p>Let’s be realistic — producing happy teams is not the reason your company exists. However, happy teams will be far more productive than unhappy ones, with less burnout and turnover, so it is in your best interest as a leader to make your teams happy. Giving teams the ability to deploy frequently without intervention means they can focus on delivering value to customers, and in their spare time they will create awesome Slack reactions.</p><h4>More Secure</h4><p>While all the other reasons above are important, this one is my favorite and not frequently talked about. Keeping software dependencies and operating systems up to date is <em>hard</em>. It can take a huge amount of time and can be very disruptive when a new patch for a critical bug is released. Continuous Deployment can remove a lot of that pain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*woRBv0PwDZ8bs2l8.png" /><figcaption>The Log4Shell logo. Source: <a href="https://commons.wikimedia.org/wiki/File:Log4Shell_logo.png">Wikipedia</a>.</figcaption></figure><p>When the <a href="https://en.wikipedia.org/wiki/Log4Shell">Log4Shell</a> vulnerability was announced, I led the response for patching Arctic Wolf’s systems. It was an incredibly stressful day, but we managed to patch thousands of AWS EC2 instances throughout the day running hundreds of thousands of containers, all while keeping the services running so the customer-facing security teams could look for Indicators of Compromise in the incoming data to protect the customers. Let me restate that — the search load on the massive data store doubled that day, and we patched thousands of systems while handling that load. The only way we could accomplish that was using the Continuous Deployment pipeline we’d built, I can’t imagine an alternative scenario.</p><p>That’s an extreme example, but the same pattern was used for any new vulnerability — commit the new version to whatever dependency management system is in use, depend on the tests to catch any regressions, and depend on the automation to roll it out.</p><h3>Objections</h3><p>Let’s address some objections to implementing Continuous Deployment.</p><h4>Not Google</h4><p>A common objection is “But we’re not Netflix or Google or ____, we don’t need that”. You do not need a large team to implement CI/CD. The entire R&amp;D team at Arctic Wolf was less than a hundred developers when we moved from weekly releases to Continuous Deployment in 2018. If I was starting a new service or company today, the first thing I would do is implement CI/CD with a Hello World example, and then start iterating from there. Even a small team of a few developers will see all of the benefits listed above.</p><h4>Separation of Duties</h4><p>In some regulated environments, auditors are frequently looking for separation of duties between developers and staff who have access to production systems. I can’t speak to all types of certifications, but I had good success in implementing Continuous Deployment while keeping SOC2 and ISO27001 certifications. The rationale we used was that a developer couldn’t make a change by themselves, they submitted a GitHub Pull Request that required a mandatory review by someone else, and all changes to production systems followed the same flow. The CI/CD pipeline configuration was also stored in GitHub with the same workflow, deployed using CD, and couldn’t be modified by a single developer. That setup had enough separation of duty to satisfy everyone.</p><h4>Badge of Honor</h4><p>Occasionally, as in the situation in my introduction, a team or an individual wears the complexity of a deploy (or builds or tests) as a badge of honor. I do understand the mindset, in the early part of my career I was a Unix SysAdmin, <em>twiddling bits</em> as a friend pointed out, and I was good at taking on large complex tasks by hand, with artisanal handcrafted servers. Taking that away from someone can be scary, and in some cases the individuals use their knowledge of the complexity as misguided job security. A key point is to reassure the individuals that their job security is not at stake with the change. How you approach this will depend heavily on the character of the teams. If someone refuses to change, they become a risk to the health of the organization, and then you can deal with it as a performance issue. In practice if you start small, by implementing Continuous Deployment with one service and then expand from there, this objection will usually quietly go away by itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8y5T7U_8E724Doga" /><figcaption>Me, hand crafting artisanal servers, by ChatGPT</figcaption></figure><h4>Not a SaaS</h4><p>When the software you are delivering is not a typical SaaS, but packaged software or an appliance, deploying continuously may not be possible. However in my experience, Continuous Delivery is still possible and desirable. In this scenario you should still follow CI/CD processes, building and testing the software continually, but the actual deployment is gated by a business process triggered (or blocked) by external processes, not triggered (or blocked) by the dev team. The mechanism may differ, but the end result is the same — the product team enables a new feature for a customer. In the SaaS world this is usually implemented by enabling a feature flag in software that’s already been deployed, while for an app it may be deploying a new version in the App Store, and for more traditional packaged software it’s releasing a new version on a website. The same principles of Continuous Integration and Continuous Delivery apply to both situations.</p><h3>Measuring Success</h3><p>Google’s DevOps Research and Assessment team has put together four commonly accepted metrics to measure software delivery performance. Measuring these, even anecdotally at first, before you implement CD will mean you have a baseline to measure against during the implementation. The full “State of DevOps 2022” with the metrics <a href="https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-out">is posted here</a>, and the metrics are:</p><ol><li>Lead time for changes: measuring time from code commit to release in production.</li><li>Deployment frequency: how often organizations deploy code to production.</li><li>Time to restore service: the average time it takes to restore service when a service incident or a defect impacts users.</li><li>Change failure rate: percentage of changes to production, resulting in degraded service subsequently requiring remediation.</li></ol><p>Several companies are building services to measure these metrics which are worth looking at. I have no affiliation with either: <a href="https://www.codegem.app/">CodeGem</a> and <a href="https://jellyfish.co/">JellyFish</a> are two that stand out.</p><h3>Next Steps</h3><p>If you’ve made it this far, there are lots of great talks and writing on <em>how</em> to implement Continuous Deployment using different technology, and a plethora of vendors selling great products to help. I recommend first starting to track the four metrics listed above, and then starting with a small service that’s not critical to the line of business. Once you have a service or two deploying continually the benefits will become clear to your organization.</p><h3>Further Reading</h3><ul><li><a href="https://continuousdelivery.com/">The Continuous Delivery site</a></li><li>WeaveWorks’ blog post: <a href="https://www.weave.works/blog/what-is-continuous-delivery-and-why-do-software-teams-need-it">What is Continuous Delivery and why do Software Teams Need it?</a></li><li>Google’s <a href="https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-out">State of DevOps 2022</a></li><li><a href="https://itrevolution.com/product/the-phoenix-project/">The Phoenix Project</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93d7b5936523" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Little Village That Could]]></title>
            <link>https://hart-michael.medium.com/the-little-village-that-could-ed0bb38c4a42?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed0bb38c4a42</guid>
            <category><![CDATA[village-life]]></category>
            <category><![CDATA[history]]></category>
            <category><![CDATA[village-stories]]></category>
            <category><![CDATA[trains]]></category>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Thu, 14 Dec 2023 22:03:07 GMT</pubDate>
            <atom:updated>2023-12-14T22:03:07.768Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*Y8L1lf660MGJLnl7yxtL2g.png" /><figcaption>The Little Village That Could, by ChatGPT</figcaption></figure><p>The windows rattled, that one picture hanging on the wall tilted again, our conversation stopped. We did not notice.</p><p>Growing up beside a busy Canadian National railway, trains were a part of life. The engineers blew their whistles as they approached the road crossing, and the roar of the diesel engines as they made their slow way up the Niagara escarpment drowned out everything, making conversation on the deck impossible. Our brains are magical and can tune out almost anything. A guest once noted how loud the train that had just passed was and our response was “what train…”. We would continue the conversations where we had left off, my Mom would straighten the picture as she walked by, and our day would go on as if nothing had happened.</p><p>We had a healthy respect for the iron monsters that inhabited the tracks beside the house. We knew from a young age that the freight trains would take kilometers to stop, and the trains were sometimes two kilometers long. My Dad would let us stand beside the crossing barriers on the road as they hurtled by and my brothers and I would quickly step back. The noise and wind assaulted our minds and our primal instincts told us to stay far away. Well, far away from the moving trains, but not far away from the tracks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDFIKyzLaKJNnZxYj3byuQ.jpeg" /><figcaption>Freshly washed CN engine 3171 crossing Orkney Road, December 14, 2023. Photo by author.</figcaption></figure><p>In the 1880’s trains were the primary method of transportation in Southern Ontario. Roads between towns were mud tracks, sometimes reinforced with logs and optimistically called “corduroy” roads. What I found shocking is how many railway accidents occurred. The Grand Trunk Railway, the dominant railway in Canada in the late 1800’s, regularly saw multiple fatalities and injuries <em>per day</em>. Some were horrific accidents involving an entire train, but others involved individuals falling off trains, and in one case getting snagged by telegraph cables while walking on top of the train. One regular pattern was injuries or death due to someone sleeping on the tracks.</p><p>We did not sleep on the tracks, nor did we ever linger. We did on occasion put copper pennies on the track and then crouched far away trying to track the penny as the heavy engine flattened it. Only a few flat pennies were ever recovered, they had a tendency to get flung away at high speeds.</p><p>Country living comes with lore and rumors and stories. One extremely foggy morning a few seconds after a train crossed the road I heard a crash and squealing brakes. Fog plays games with sounds and I didn’t think much of it. The story came out later by word of mouth — a truck carrying manure was crossing the tracks from one field to another on an uncontrolled farm crossing, a common occurrence in farming life. The driver stopped his engine, listened carefully and heard nothing in the thick fog, started his truck and began crossing. To his horror a VIA Rail locomotive loomed in his window. The train struck his truck just behind the cab, sending the cab and driver into the field, where he crawled out miraculously unharmed. The train slid to a stop a kilometer away, covered in manure.</p><p>There were tunnels under the railway crossing, decades old made of hand hewn stone, ostensibly for water drainage but to us they were paths to buried treasure full of danger and monsters complete with a caved-in ceiling. Rumor has it a younger brother managed to crawl through the smaller one with the collapsed ceiling, making it from one side of the tracks to the other. No treasure was found but bragging rights were established. Don’t tell our mother.</p><p>Much to the amusement of my daughters, a young boy lives deep inside every Dad, and the love of trains has not left me. When I moved to St. George, Ontario, I couldn’t help but notice remnants of train tracks everywhere. There is an old tunnel along Highway 5 under the remains of a bridge, built in the same style and era as our childhood tunnels. Two giant concrete abutments stand on either side of Main Street just south of town, slowly being reclaimed by the trees. Town lore, just as wild as country lore, talks about a railway accident involving that bridge, or maybe two accidents, a hundred years ago or more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W44lrNyzjic0SbfNs1O9jw.jpeg" /><figcaption>Remains of the concrete bridge pier, looking East from Main Street. Photo by author.</figcaption></figure><p>Small towns are funny things. There is never a shortage of drama, rumors and arguments about the most trivial things. But when someone is in need, all of that is forgotten and the villagers come together to help in whatever way they can. In early October 2015 a young boy was dying of cancer and was not going to live long enough to see Christmas. His mother asked her family if they could celebrate Christmas early, and the residents of St. George enthusiastically answered “Yes!”. The deathly Halloween displays were quickly replaced with Christmas decorations, and a parade with a hundred floats was organized in a few days. On the evening of October 24, 2015, Evan rode in Santa’s sleigh in the longest parade the town had ever seen as fake snow was blown across his lawn, mingling with the real snow and tears.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0mlG9k6MTlljvO6sC6iaA.jpeg" /><figcaption>Ethan riding Santa’s sleigh, October 2015. Photo by author.</figcaption></figure><p>Generations earlier the townsfolk came together to aid complete strangers in a much different scene. The descendants of those living in St. George in 1889 still live in the area today. There are still Howell’s on Howell Road, at least one mailbox on Burt Road proudly displays the last name of Burt, and the Roseborough family still farms much of the surrounding land. Brantford’s Daily Expositor wrote in 1889 “The people of St. George did noble, heroic work.” and Hamilton’s Spectator stated “A noble array of heroic women turned out to minister to the dying and wounded.”</p><p>On February 27, 1889 just before 6pm, the St. Louis express, a regular passenger train with service from Windsor to Toronto, had a catastrophic wheel failure just before the St. George bridge. The steam engine, tender and smoking car, flying along at 50 mph (80 km/h), derailed but made it across the bridge bumping across the ties, tearing apart the bridge deck behind them. The passenger car fell through the hole and by some accounts flipped end over end before landing right side up in the valley some 60 feet (18 meters) below. The dining car tipped into the valley, landing on one end and resting vertically up against one of the bridge piers. The last car remained on the bridge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uF-7l6h4k7K43WgB.png" /><figcaption>The train wreck, camera looking North West. The St. George station is visible at the top left. <a href="https://www.thebtown.ca/blogs/strange-brantford/the-great-st-george-trainwreck">Source</a>.</figcaption></figure><p>Town lore says that the Grand Trunk Railway was laid a mile south of town, because the surveyor planning the route was overcharged for a book of matches in the village, and out of spite moved the track south instead of through the village. Whether true or not, the bridge was close enough to the village that the residents would have been accustomed to the sounds of the passing St. Louis Express just before dinner and had known immediately that something went badly wrong. They rushed out with every manner of conveyance they had and did what they could to help the dying and wounded. The dining car caught fire, with early diners and staff pinned under chairs and tables and the burning cook stove. Uninjured passengers and villagers put out the fire without any fire fighting equipment and began the rescue. At least two wounded were recovered from where they had fallen off the train into the snow.</p><p>The Expositor and Spectator newspapers described how the villagers turned the Mechanics Institute (an early library) into a makeshift morgue, and the thirty badly wounded were spread between Dr. Kitchen’s house, Beemer’s Hotel, and the Commercial Hotel. Dr. Kitchen, the village doctor, must have been overwhelmed with the horrific scene, but no mention is made of it. Later that evening special trains from Hamilton and Woodstock arrived on the scene with more doctors to aid the wounded, and workmen to begin the cleanup. The same trains took the wounded and dead back to their home towns. In the end eleven people died and more than thirty were badly wounded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AO5w1dYyMGRo8vp7.png" /><figcaption>Another view of the accident in St. George. <a href="https://www.thebtown.ca/blogs/strange-brantford/the-great-st-george-trainwreck">Source</a>.</figcaption></figure><p>The 1880’s were a dangerous time for rail travel. If a passenger train was involved in an accident the results were frequently tragic and received heavy newspaper coverage. Freight trains regularly had accidents as well with fatal results, but received little or no press. One of the wounded recovered in the snow in St. George was Henry Angles, who died the next day. The Daily Expositor published this paragraph March 2, 1889 that caught my attention:</p><blockquote><strong><em>Death of Fireman Angles, the Eleventh Victim — The Latest Particulars</em></strong></blockquote><blockquote><em>LONDON, Ont., March 1 — Fireman Henry Angles, who was injured in the late accident to the St. Louis express at St. George, died at his home here at 1.30 this morning. Angles has been in two serious accidents before and escaped. He was firing for Engineer Cox when their engine struck an open bridge at Merritton. He jumped and was saved while Cox was killed. When the Port Stanley excursion train dashed into an oil train at St. Thomas in July, 1887, and many were burnt and killed, Angles was firing for Engineer Donnelly. Angles escaped by jumping but his mate was killed. In Wednesday’s disaster the engineer stuck to his post and escaped injury while Angles again jumped, his head striking an abutment of the bridge in his descent.</em></blockquote><p>The St. Thomas disaster was horrific, seventeen people died in that accident. Among the dead were Engineer Harry Donnelly, an engineer with fifty years experience, passengers trapped in the wreckage, and rescuers burned by an oil tank explosion shortly after the accident.</p><p>I could not find a record of the first accident Henry Angles was involved in that matched what was written in the Expositor. That wreck likely involved the swing bridge crossing the second Welland Canal just east of Merritton, which today is a council ward of the city of St. Catherines. There were at least three accidents in four years involving the swing bridge that resulted in the train going into the river:</p><ul><li>1872 — A freight train derailed into the Welland canal, killing Engineer Gaddy. (<a href="https://heritagecollections.elgin.ca/link/archives160686">Source</a>)</li><li>August 18, 1873 — An accident occurred with a passenger train and the bridge, where Engineer Thomas Cox jumped at the last minute and the unnamed fireman stayed with the engine that went over the bridge but survived. No lives were lost.</li><li>April 25 or 26, 1876 — From the Simcoe Reformer published April 27, 1876: A freight train on the Canada Southern Railway on Monday night went into the canal at Welland, the bridge being open. The engineer and a brakeman were killed, the fireman survived reportedly because he was taking lunch in the caboose while the brakemen was covering fireman duties. (<a href="https://events.wellandhistory.ca/2013/03/10/train-on-csr-went-into-the-old-canal/">Source</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hozWXpyzHI9MlJG6.png" /><figcaption>Freight train in the Welland Canal, 1872. <a href="https://heritagecollections.elgin.ca/link/archives160686">Source</a></figcaption></figure><p>Humans are adept at becoming accustomed to living with dangers or threats that would shock others, and rail travel in the 1880’s was dangerous but with an acceptable risk. Henry Angles was born in New York City, the son of Irish parents who likely immigrated during the Irish Potato Famine with 900,000 other Irish citizens. It’s not clear how he ended up in London, Ontario, where he married Ida Green. Work was a blessing for those who escaped starvation, and holding a well-paying albeit dangerous job on the railway was certainly better than not eating. The tragedy is that Henry survived twice by jumping from a moving train heading to inevitable wrecks, but died doing the same in the third accident.</p><p>It is not the steel and noise and speed of trains, exhilarating as that can be, that interests me. Trains bring people together, obviously quite literally by moving people, but it’s the journey and shared experience that builds the connections. Traveling through Europe by train last summer was fascinating, the conversations and memories even in languages I couldn’t understand were so much fun: watching a family with excited young children heading to the beach, a conductor letting a young girl make the announcement for the next station, seeing an incredible Swiss mountain looming through the clouds. Sharing those with complete strangers brought us closer together, even if just for a few minutes.</p><p>As horrific as some of the accidents of the past were, the common thread in all of them was the response of the residents of the surrounding towns and villages, and the praise in the newspapers for the random strangers rushing to help the unfortunate. I’m now part of the community who 134 years earlier came to the aid of the passengers of the St. Louis Express. Working together to celebrate Christmas two months early for Evan was a very public highlight of what this village can do.</p><p>As horrific as some of the events of today are, there is still a common thread of desire to help each other through it all. The willingness to aid complete strangers is what makes us humans, helping our community is what keeps us from anarchy. Every community pursues this either intentionally or subconsciously, and it’s definitely worth the effort. I hope we never lose the desire to help each other.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed0bb38c4a42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Code is the Same Thing as Writing Prose]]></title>
            <link>https://hart-michael.medium.com/writing-code-is-the-same-thing-as-writing-prose-ba8d04ed6917?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba8d04ed6917</guid>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[prose]]></category>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Wed, 06 Dec 2023 16:10:14 GMT</pubDate>
            <atom:updated>2023-12-06T16:10:14.912Z</atom:updated>
            <content:encoded><![CDATA[<p>The act of writing Code and writing Prose is the same. They both use the same mental processes, have language constructs and audiences, and require significant focus to be effective. I’ve written code and countless technical docs and articles for well over two decades, and lately started writing fiction. This article has been percolating in my head for years. Let’s dig into how they are the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*HV6y01T2tL5xl-KQ6cGkuw.png" /><figcaption>The Same Thing, by ChatGPT</figcaption></figure><h3>Language</h3><p>Both Code and Prose are written in a language. My prose is in English, and while I’ve written code in C, Java and Perl, I’m most familiar with Python.</p><p>Counting the number of words in English is not simple, as <a href="https://www.merriam-webster.com/help/faq-how-many-english-words">this article</a> by Merriam-Webster explains, but there are roughly 470,000 English words. A typical English speaker knows between 20,000 and 35,000 words, depending on their level of education.</p><p>Python, like most programming languages, has a very small set of keywords and constructs, and the real value is in the shared libraries, or pips in the Python world. <a href="https://pypi.org/">PyPi</a>, the repository of those libraries, has 498,650 projects, or pips, listed as of December 5, 2023. A recent small project I was working on pulled in 68 of those.</p><p>It’s a bit nonsensical to compare the number of words in English to the number of packages in PyPi, but we can agree that both English and Python are very rich languages, and no single person can reasonably understand the full scope of either language.</p><h3>Compilers</h3><p>The concept of a compiler is fairly simple: translate human-readable code into machine-readable instructions. In this example below, the python code is supposed to print “Hello World!” ten times, with the assembly code generated on the right. I can’t read assembly code, it’s a completely foreign language to me, but that’s not important because the computer can read and execute it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mi5XyQvMHuTpPu5w" /><figcaption>Screenshot from godbolt.org</figcaption></figure><p>When writing prose, the reader’s brain becomes a compiler, translating the words from screen or paper into tangible thoughts and images, even if the words are not strictly correct. A great author can describe a scene so vividly that you can see it in your head even though the words are just pixels or ink.</p><p>My favorite line of prose is by Stephen Leacock in <a href="https://www.gutenberg.org/cache/epub/4682/pg4682-images.html#chap04"><em>Gertrude the Governess</em></a> “He flung himself from the room, flung himself upon his horse and rode madly off in all directions.” This is where admittedly my argument falls apart, since the last part of that line does not strictly make sense, yet our brains can see past it and conjure up with a vivid image that’s captured the imagination of many readers. A compiler would throw an error like:</p><blockquote>HorseError: Horse can only gallop in one direction</blockquote><p>ChatGPT also struggles to make sense of it, this is the best it could do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*hWrRyrxDE6Fr0ZCg6P8AuQ.png" /><figcaption>Galloping Wildy, by ChatGPT</figcaption></figure><h3>Audience</h3><p>Every writer has an audience, even if they aren’t aware of it or think explicitly about it.</p><p>A writer of prose might be writing a journal that will never be seen by anyone, and the audience is themselves. Or they may be writing a draft of a book that will hopefully catch a publisher’s eyes. Or, like so much of my writing in my professional life, the audience for technical docs is made up of my peers who need to understand the reasons why something is built the way it is.</p><p>Software Developers have two audiences. The first is the compiler and runtime, after all a piece of software that won’t compile or run, or worse do the wrong thing when running, is useless. But the far more important audience is <em>the next developer who has to read or edit the code</em>. It’s very easy to write unreadable code, it’s far more difficult to write code that another developer can easily understand. And beware, that next developer might be you. Countless times I’ve thought, <em>who wrote this crap</em> and a `git blame` (a whodunnit in software development) showed that I was the author a few years ago. Oof. If you want to see some terrible and beautiful examples of functional but completely unreadable code check out the <a href="https://ioccc.org/">Obfuscated C Code Contest</a>. Once you’ve written code, someone has to maintain it, and that future developer is your audience.</p><h3>Generative AI as an Aide</h3><p>I recently started writing fiction, which is the quality of <a href="https://en.wikipedia.org/wiki/Vogon">Vogon</a> poetry and should never see the light of day, but that aside I found that ChatGPT made an excellent research assistant. My story took place in Ireland around the <a href="https://en.wikipedia.org/wiki/Great_Famine_(Ireland)">Great Famine</a> in the 1840’s, and much as I want to research my story by exploring Ireland for a few months, a more reasonable approach was to ask ChatGPT questions like:</p><ul><li>what are some Irish first names in 1840’s</li><li>how much salary did a farm laborer earn</li><li>how much did a train ticket cost</li><li>what’s the name of a ship that went from dublin to quebec in 1847</li></ul><p>It happily gave me answers. They may have been wrong answers, but ChatGPT responded fast enough for me to be able to continue writing without interruption or expensive overseas trips, knowing full well that if the Vogons ever decide to publish the writing they would have to do some serious fact checking.</p><p>On the code side I subscribed to GitHub Copilot, and it is very, very good. It definitely hallucinates at times, and occasionally is completely wrong. Where it shines is that it understands the context of what you are working on. In a real “duh” moment I forgot how to do multi-line strings in Python, and asked it “how do make the string in line 43 multiline so it’s not more than 120 chars long” and it rewrote the line for me. I also asked it for an AWS CloudFormation template with some specific features, partially as a test and partially because I was lazy, and while there were a few errors, 95% of the template was correct. GitHub Copilot is like having a very senior developer sitting beside you, always available for questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*Y7jx2XYIDeRXH7OqNnsuNA.png" /><figcaption>A Code Wizard, by ChatGPT</figcaption></figure><p>Generative AI has far too much hype today, and it can be wildly wrong, but it’s an extra tool in the writer’s toolbox that’s worth spending some time to learn.</p><h3>Environment and Tools</h3><p>The best writing and coding happens when the author can get into the flow, or into the zone, into the Deep Work space that Cal Newport writes about in his <a href="https://calnewport.com/writing/">excellent book</a>. Deep Work requires a quiet space without interruptions, and it takes practice to get into that mode. Unfortunately most software shops do not recognize this, distractions abound in the form of Slack/Teams/Email notifications and meetings, and software quality suffers as a result.</p><p>Tools for authors and developers are abundant and varied. For authors this may be pen and paper, Microsoft Word, or Google Docs or something else entirely. I’m using Google Docs for this article, because for the most part it Just Works™ and stays out of my way. For software developers it’s typically an IDE like IntelliJ or VSCode. I strongly believe that a specific tool should not be forced on writers, because that will wreck the writer’s flow. If you’re a leader employing writers of prose or code, those writers are your most expensive asset, and spending a few extra dollars on the tool of their choice will make them far more productive.</p><h3>Writing is an Art</h3><p>Writing prose is definitely an art, even potentially dry technical documentation can be artfully written. Many developers, including yours truly, will argue that writing code is also an art. There is a lot of very bad art, so much so that <a href="https://museumofbadart.org/">The Museum of Bad Art</a> exists, and it’s well worth a visit. The art of writing has also produced some spectacularly bad prose. If writing code is an art, the same can be said that there is some horrendously bad code out in the wild. The code submitted to the <a href="https://ioccc.org/">Obfuscated C Code Contest</a> is purposely horrible, but most software developers have run across code so bad that it’s better to throw it out and rewrite it rather than try to understand the mess and fix it. I’m sure some of my code falls into that category as well. But occasionally you stumble across a piece of code that is a work of art, it’s easy to read and understand, it’s efficient and well-documented, and masterful in all ways. When you see that and need to make a change to it, you’re like an artist restoring a famous painting, trying desperately not to make it worse.</p><h3>Failure</h3><p>One of the tips in the <a href="https://welearncode.com/25-tips-new-devs/">25 Tips for New Developers</a> is “Get comfortable with failure”. That applies to both code and prose. Writing code is hard, understanding the execution runtime can have a huge learning curve. Writing prose, especially fiction, can also be extremely hard. As I’ve proven to myself with my Vogon-esque story, inventing dialogue and creating characters from nothing can be extremely challenging, and failure is inevitable. I’ve ripped out entire sections of code and entire chapters of prose, starting over in both cases with knowledge learned in the failing, making me a better writer in the end.</p><h3>Learning from each other</h3><p>Medium has a lot of information on how to become a better writer. If writing code and prose is the same, then a lot of the advice targeting writers should also apply to software developers. <a href="https://medium.com/@kevinnokiawriting/improve-your-writing-with-stephen-kings-routine-26912dd8a195">This article on Stephen King’s writing routine</a> could easily be applied to writing software. Admittedly it’s more difficult to create his environment in a paid software developer position, but it is possible. In the early stages of Arctic Wolf we talked extensively about how to create a good distraction-free environment, with large blocks of time that were meeting-free, and even entire meeting-free days. I had some of my most productive writing times in that era.</p><p>If you read the <a href="https://welearncode.com/25-tips-new-devs/">25 Tips for New Developers</a> article and substitute writing prose with code, much of the advice applies to new writers. Things like “Find a Community”, “Find a Niche”, “Focus on the fundamentals first”, and “Build projects you love” all can be applied to writing of all kinds.</p><h3>Now go and write!</h3><p>Werner Vogels, CTO of AWS, always ends his keynote speeches with “Now go and build!” It doesn’t matter if you’re writing code or prose… Now go and write!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba8d04ed6917" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to be on-call]]></title>
            <link>https://hart-michael.medium.com/how-to-be-on-call-034e3a202729?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/034e3a202729</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[on-call]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[blameless]]></category>
            <category><![CDATA[blameless-culture]]></category>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 16:21:43 GMT</pubDate>
            <atom:updated>2023-10-18T21:12:09.707Z</atom:updated>
            <content:encoded><![CDATA[<p>A few years ago at Arctic Wolf I put together a talk titled “How to be on-call”, in response to the rapid growth of the organization and increasing number of on-call schedules. The talk turned out to be very popular and the recording became part of the onboarding process for new employees. While some of the talk was company-specific, much of it is applicable to the broader software industry, and I’ll share some of the highlights here.</p><p>I have been on-call for most of my career and led teams with on-call rotations, and have a lot of experience with the negative impact of on-call to my personal life and the lives of my colleagues. I’ve missed Christmas dinner (years later my Mom still brings it up), worked through weekends and nights, missed many kids’ events, and once juggled a fussy baby and an incident call at the same time. My goal is to make being on-call as sane as possible, balancing what the business needs with our collective personal lives.</p><p>“#oncallsucks” was trending circa 2020, and in response Charity Majors summarized the on-call responsibilities in <a href="https://charity.wtf/2020/10/03/on-call-shouldnt-suck-a-guide-for-managers/">On Call Shouldn’t Suck: A Guide for Managers</a> “<em>It is engineering’s responsibility to be on call and own their code. It is management’s responsibility to make sure that on call does not suck. This is a handshake, it goes both ways, and if you do not hold up your end they should quit and leave you.</em>”</p><p>A few ways that #oncallsucks:</p><ul><li>Alert Fatigue — there are <em>still</em> too many horror stories of alerts constantly going off that are not actionable. Dan Ravenstone had a great talk at Monitorama “<a href="https://vimeo.com/843998561">Thinking Critically About Alerting</a>” about how to deal with this.</li><li>Onblame Oncall — quoted from a colleague — quite the opposite of blameless, where the on-call responder is blamed for whatever broke.</li><li>Always On — this was the culture at a former company, and you were always on-call even when on vacation. This was horrible in so many ways and led to quick burnout and high turnover.</li><li>There Can Only Be One: you are the only person on-call, there’s no escalation process to get help, and no team to back you up.</li></ul><p>On-call does not have to suck, but it’s hard work for everyone to remove the suck and even harder to continually keep it from sucking.</p><p>My original talk was focussed heavily on the individual, but I’m going to focus here more on how a team can support each other, and the responsibilities of the leaders.</p><h3><strong>Why on-call?</strong></h3><p>The first question to ask especially as a leader is: do we need an on-call rotation? If your service has an actual SLA (with penalties etc) or is a true global service, or is supporting say a 24x7 manufacturing process, the answer is likely yes. There are many cases where the easy answer is yes, but under scrutiny that may not hold up. I have two examples were a 24 hour schedule could be reduced:</p><ul><li>If the AWS Trust and Safety team contacts you about potential abuse on your account, you must <a href="https://repost.aws/knowledge-center/aws-abuse-report">respond within twenty four hours</a>. If that happens on say early Saturday morning, a response cannot wait until Monday, but it also doesn’t mean you need to wake someone up. In that particular scenario we configured the paging software to only callout during daytime hours, seven days a week.</li><li>In the case of office network infrastructure, the temptation is to setup a 24x7 alerting process. In many cases a blip in the night is acceptable as long as the network is functioning by the time the first individuals arrive on site. Some creativity with scheduling means the IT staff can sleep most of the night before getting called out.</li></ul><p>Don’t take it as gospel truth that you actually need a 24x7 schedule, be creative to reduce the impact to the on-call staff if at all possible.</p><h3><strong>So you are on-call…</strong></h3><p>Before you start your on-call rotation, it should be obvious that you should know how you’re going to be called, what you might get called for, and expectations on how to respond and escalate. I say it <em>should</em> be obvious, but it’s shocking how frequently this is all taken for granted.</p><p>At a minimum you should know:</p><ul><li>How am I going to be called — PagerDuty or VictorOps or X? Any setup required?</li><li>Who gets to wake me up?</li><li>Am I on the escalation path for anything?</li><li>What are the expectations on response time when I do get called?</li><li>Where are the automated alerts defined, and where are the runbooks? (There <em>are</em> runbooks, right?)</li><li>How do I escalate to get help?</li></ul><p>When you do get called, Don’t Panic! In my first few calls in the late 90’s I fought panic, because I didn’t have answers to the questions above and had no real escalation path. If you are the first responder, your job is to triage the alert and determine next steps. Those steps are highly contextual to your situation, but should always include the question “was it worth waking up for this alert”, and if the answer is no, tune the alert the next day.</p><p>Take care of yourself. Waking up in the middle of the night for a call always wrecks me — the adrenaline boost of the call means I won’t go back to sleep — anecdotally others struggle with the same. Do not hesitate to ask the team to cover the next night so you can get a night’s sleep.</p><h3><strong>Your team is on-call…</strong></h3><p>The best way for an on-call shift to be less stressful for me is to know that my team has my back, that I’m not in it alone. When I’m not on-call, the responsibility of supporting my colleagues falls on me. What does that actually mean in practice? A few suggestions:</p><ul><li>Write good runbooks. At Arctic Wolf alerts were defined in a GitHub repository, and runbooks were attached and enforced at commit time — you could not commit an alert without a runbook. This brought some discipline to the process of creating alerts, encouraged careful thought on how to debug situations, and meant that when called in the middle of the night you had a really good starting point. Put a slightly different way — if you can’t write a runbook which contains actions on how to resolve the alert, the alert shouldn’t exist. There are a lot of great resources on how to write runbooks, <a href="https://medium.com/@magstherdev/runbooks-3a37434141ac">this article</a> is a great example.</li><li>Keep an eye on the on-call team member’s work load. If they’ve been up all night, organize a swap so they can take the night off. Some individuals (like me) have a tough time asking for help, but recognizing that in others I’ve had to almost forcibly take someone off a schedule to give them a break. The team’s leader can do this but it’s best if the team organically takes care of each other.</li><li>Fostering a blameless culture means creating a psychological safety net, which is incredibly important when responding to incidents, especially when there’s significant pressure and a legitimate fear of “what if I make it worse”. I wrote an entire article on that: <a href="https://www.gybe.ca/a-few-words-about-blameless-culture/">A few words about blameless culture</a>.</li><li>Create a good on-call onboarding process that answers the questions above in the “So you are on-call…” section. The first few calls can be extremely stressful. I’ve seen successful onboarding including call shadowing, where a new team member was on the same rotation as a more senior member, and they answered calls together for a few weeks.</li></ul><p>The short summary: take care of each other, treat your team members as you want to be treated.</p><h3><strong>You are a leader with an on-call team…</strong></h3><p>The best, albeit most controversial, way to support your on-call team is to pay them for the extra time. You are asking your team to work 168 hours per week, impacting their personal lives and their nights and weekends, and you are adding significant stress to their jobs. Paying an employee for that impact sends a strong clear message that you recognize and appreciate the extra hours. There are many ways to do this and the actual amount may be affected by the specific country’s legislation, but my favorite was eight hours of pay for a 24x7 shift and a minimum two hours of overtime for a callout.</p><p>The added benefit of paying for on-call is that you will bring visibility of the cost of running service to the business, with a budget line item containing the cost. The rest of the business understandably doesn’t know what it takes to be on-call but they certainly grok budgets, and having a common language to discuss the actual cost becomes very important when planning for hiring, discussing adding more services or products that will require 24x7 support, and setting SLA’s for products.</p><p>A leader must measure the number of callouts, and take immediate action if the number starts to increase. I have seen three common reasons why the number of callouts can increase:</p><ol><li>Scale — Both of my previous two organizations scaled more than 2x per year for multiple years in a row, and at some point a service would naturally start hitting scaling limits. While we tracked a myriad of metrics showing <a href="https://sre.google/sre-book/monitoring-distributed-systems/">the four golden signals</a>, an increased callout rate was definitive proof that the service was suffering, and new feature development needed to stop so the team could address scaling.</li><li>Services change over time, and alerts need tuning. Alert fatigue, where an alert firing frequently is seen as “normal” and then ignored, is a real thing and should be squashed as fast as possible. Tune those alerts!</li><li>Along with rapid development comes the possibility of a service accidentally becoming less resilient. This can easily happen even to a service that isn’t scaling and taking on extra load, and needs to be addressed.</li></ol><p>How you measure and track this will depend on your org, but I highly recommend your team has visibility into your tracking and decision making process so they can see that you are looking out for them, and the actions you take to slow the callout rate will speak volumes.</p><p>A team needs to know that their leader has their backs and supports them. Again that’s pretty obvious, but a few ways you can do this:</p><ul><li>Run interference for them. In a previous job I had an awesome manager who would stand in the path leading to our cluster of cubicles, preventing irate users and other leaders from interrupting us while we were resolving an incident. Modern equivalents to someone rushing your desk are out-of-band questions in Slack/Teams/Zoom/etc directly to your team members, and your team needs to feel comfortable to escalate those directly to you instead of trying to answer them.</li><li>Create a good escalation process. PagerDuty’s internal policy, <a href="https://goingoncall.pagerduty.com/people/">documented here</a>, is “Never Hesitate to Escalate” — which first means your team needs someone to escalate to, and then a Blameless way to do so without any fear of retribution regardless of the time of day.</li><li>I strongly recommend not scheduling project work for the on-call team member for the duration of their on-call shift, and build that into your project planning and estimation process. The individual can spend their time on bug hunts or research or annoyance-driven-development, and be ready for interrupt-driven work. A few of my teams implemented very successfully, it takes a lot of pressure off the individual and removes timeline risk from any planned project work.</li><li>If a response team is physically in the office, send in food for lunch or dinner, or if the team is remote, hand out coupons to food delivery services where possible. This seems like a small gesture, but I fondly remember running large incidents out of a conference room over the dinner hour, and a stack of pizzas appearing without warning, sending a strong signal that our leaders were watching out for us.</li></ul><p>One of the harder things I’ve had to do with on-call teams is to formalize response times and expectations. With smaller teams it’s easy to build trust and have an informal “best effort” policy but as teams grow so does the expectation of a more formal policy. I know first hand the impact that on-call has on personal lives and the difficulty of adhering to any policy. That said these were the guidelines I’ve recommended in the past:</p><ul><li>Be able to acknowledge an alert within 15 minutes.</li><li>Best effort to get online within 30 minutes. If, when you acknowledge the alert, you know you can’t hit the 30 minute response time, escalate. I had a long commute and I frequently would either find a local coffee shop or escalate and join the incident call when I arrived at home.</li><li>This can be a thorny one fraught with legal issues — be sober enough to be able to answer the call and work the problem to resolution. My personal rule of thumb is when on-call I should be legally able to drive, but that of course will depend on your local laws.</li><li>You cannot be on vacation (PTO) <em>and</em> on-call. I was surprised I had to even state this but ran into it a few times. Taking time off is incompatible with being on-call, that’s not negotiable.</li></ul><p>The last way to make sure your on-call rotation is healthy is to have enough people on it. The absolute minimum for a 24x7 on-call rotation, assuming weekly shifts, is four individuals. Any less than that and you will burn out your team, and I’m hesitant to even give the number four because in most cases it’s too low. Being on-call one week out of four puts a significant burden on a team member and will cause them to leave. Having six people is better, and eight is good. If your team has less than four or even six, join forces with another team and share a rotation until you can grow the team. A few of my teams implemented that as they grew and split, and the positive side effect was the teams wrote really good runbooks to support each other.</p><p>I’m convinced that on-call doesn’t have to suck, and while getting called in the middle of the night will never be fun, a good on-call culture can make it bearable.</p><h3><strong>Resources</strong>:</h3><ul><li>Charity Major’s article “<a href="https://charity.wtf/2020/10/03/on-call-shouldnt-suck-a-guide-for-managers/">On Call Shouldn’t Suck: A Guide for Manager</a>”</li><li>Dan Ravenstone’s talk at Monitorama PDX 2023: <a href="https://vimeo.com/843998561">Thinking Critically About Alerting</a></li><li>Writing good runbooks: <a href="https://medium.com/@magstherdev/runbooks-3a37434141ac">Runbooks: An On-Call person best friend</a></li><li><a href="https://www.gybe.ca/a-few-words-about-blameless-culture/">A few words about Blameless culture</a></li><li>The Four Golden Signals is a section in the <a href="https://sre.google/sre-book/monitoring-distributed-systems/">Monitoring Distributed Systems</a> chapter of Google’s <a href="https://sre.google/sre-book/table-of-contents/">SRE Book</a>.</li><li>PagerDuty’s docs “<a href="https://goingoncall.pagerduty.com/">Best Practices for On Call Teams</a>” are excellent.</li><li>This is a great article on the mechanics of scheduling: <a href="https://hackernoon.com/how-to-build-an-effective-and-sustainable-on-call-schedule-for-your-team-3p11a3txy">How to build an Effective and Sustainable On-Call Schedule For Your Team</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=034e3a202729" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Few Words About Blameless Culture]]></title>
            <link>https://hart-michael.medium.com/a-few-words-about-blameless-culture-316c3a4ea8ab?source=rss-c7ee58e90e72------2</link>
            <guid isPermaLink="false">https://medium.com/p/316c3a4ea8ab</guid>
            <category><![CDATA[blameless]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[blameless-culture]]></category>
            <dc:creator><![CDATA[Michael Hart]]></dc:creator>
            <pubDate>Tue, 17 Oct 2023 16:36:43 GMT</pubDate>
            <atom:updated>2023-10-17T16:36:43.802Z</atom:updated>
            <content:encoded><![CDATA[<p>The concept of blameless culture has been around for a long time in other industries, and while the history isn’t clear, you could argue that it became an “official” part of the tech industry with the publication of the definitive book <a href="https://sre.google/workbook/table-of-contents/">Site Reliability Engineering</a> in 2016.</p><p>My summary of blameless culture is: when there is an outage, incident, or escaped bug in your service, assume the individuals involved had the best of intentions, and either they did not have the correct information to make a better decision, or the tools allowed them to make a mistake.</p><p>During my tenure at <a href="http://www.arcticwolf.com/">Arctic Wolf</a> I embraced and evangelized blameless culture, and brought it into every part of our organization that I could, including of course the post-mortem process. That culture took on a life of its own, as they do, and in some cases it grew in directions I hadn’t expected. Before I expand on what my definition of blameless is, I’ll share a personal story that’s somewhat embarrassing but frames my journey.</p><p>Once upon a time, when I was young and cocky, I was a Unix system administrator at a large manufacturing company. I responded to a call one night where the power for a data center was cut during maintenance of a fire suppression system in the building, causing immediate and expensive downtime to the manufacturing plant. At the beginning of the subsequent root cause meeting (the modern post-mortem language wasn’t common in the industry yet) I blurted out something to the effect that the meeting was a waste of time because we knew that the fire department tripped the power. Awkward silence ensued, and I discovered the gentleman sitting beside me was the fire chief. He had every right to express his anger at my blameful comment, but instead he quietly explained the process of determining root cause without assigning blame. As it turned out I was wildly wrong and the cause was, as is so often the case, cascading failure with a side dish of unexpected behavior, and no one could have predicted the end result.</p><p>That episode stuck with me for a few reasons, in large part because it was an embarrassing moment, but resulted in a few key lessons which took some time to percolate. That moment was a clear example of how to be blameful, not blameless. A more important reason this sticks in my memory is the fantastic leadership example the fire chief set for me, where a senior leader took the opportunity to mentor a newer employee through a potentially bad situation, and I’ve tried to emulate that wherever I can. He very likely has no idea how much impact he had on my career, but I’m very thankful for his patience with me that day.</p><p>My favorite public example of blameless behavior is when AWS S3 went down in the us-east-1 region on February 28, 2017, taking down a good portion of the Internet. AWS wrote in <a href="https://aws.amazon.com/message/41926/">the public post mortem</a> “<em>an authorized S3 team member using an established playbook</em>” … performed some maintenance, and the team member inadvertently introduced a typo which resulted in the outage. AWS’s response could have been “an employee made a mistake and we fired them” but instead my brief summary of the response was: the tool did not have enough safeguards and the service’s recovery was not fast enough. AWS addressed the situation by building in more safeguards, and implemented improvements to the service recovery time.</p><p>One unexpected direction the blameless culture took, which in hindsight I’m happy about, is to not name individuals beyond the boundaries of the team. This can result in the perception by senior leadership, who may not fully understand blameless culture, that their teams were not taking responsibility for their work. Talented developers in small teams are typically quick to point out their own mistakes, but I insist on not stopping post-mortems when someone points out their own mistake, and continuing investigating how to improve tools and information. The individual’s name is not important, almost immaterial to the postmortem, but the steps needed to improve the systems are the key parts to communicate to company leaders.</p><p>Finding the balance between taking responsibility for an issue and not naming names is difficult, but increasingly important as an organization grows. While I fully expect my team to own up to any mistakes, typos, assumptions etc during a post-mortem or in private, their names should be kept out of any docs published to the broader organization. Removing names provides psychological safety for team members, and prevents the temptation by leadership or others to blame individuals and take action. When presented with names I’ve seen leaders take immediate vindictive action, like termination, but also delayed action affecting compensation or promotion decisions later. In my ideal world the team as a whole, and their respective leadership chain takes responsibility for the events.</p><p>In my definition of blameless I “assume … best of intentions”. But. I’ve always made it clear to my teams there is an unwritten exception clause: if an incident/outage/bug was caused by either deliberate action, or by mistake <em>and then</em> covered up, there would be immediate consequences that most likely would result in the individuals finding new employment. Deliberate action causing damage is easy to understand and will likely involve law enforcement, but covering up mistakes deserves a bit more attention. I have two examples, one negative and one positive:</p><ol><li>I was part of the response team in situation where an individual made a mistake, and then attempted to cover their tracks. That resulted in a spectacular 48-hour outage and shortly thereafter termination of their employment. The org’s leadership was clear that the individual was not fired for making the mistake, but for the attempted coverup.</li><li>Back in my sysadmin days a colleague accidentally deleted a production database. They immediately took their hands off their keyboard and said “hey team, I just messed up and need help”. We rallied together and came up with a solution without causing an expensive outage. My colleague could have covered it up, and I’m not sure we would ever have found why the database had disappeared, but because we had a blameless culture and great team, the individual had the psychological safety net to immediately ask for help.</li></ol><p>Early in my career a leader told me, well before chaos engineering was a thing, “If you’re not breaking things once in a while you’re not working hard enough.” Anyone who runs production services lives this every day and knows there is a risk of making a mistake that will cause an outage. The best possible action when making a mistake is immediately escalate to your team and leadership and ask for help to help remediate the issue. At Arctic Wolf we had a cultural tenet “Bad News Fast” which we applied to projects and incidents alike, and contributed heavily to the company’s success. Team members were rewarded and praised for raising bad news fast, whether it was an issue in production, or an upcoming capacity planning threshold that would cause months of rework, or finding an issue in a project that would significantly delay the delivery.</p><p>Blameless culture cannot be reserved only for postmortems, it needs to be lived and promoted every day. <a href="https://www.linkedin.com/in/robzuber/">Rob Zuber</a> of <a href="https://circleci.com/">CircleCI</a> summarized it really well in <a href="https://circleci.com/blog/value-of-blameless-culture/">The value of blameless culture</a>: “<em>Incidents are the microscope under which we tend to examine blameless culture</em>” and “… <em>the culture that shows up in your incident response is going to be a direct reflection of the culture that you build every day, with every action, under the most mundane circumstances.</em>”</p><p>Building a blameless culture is a complex topic, but one well worth investing a lot of time in, and will result in a healthy and high performing organization. Lots of great books and articles have been written on how to implement it, with a few key ones here:</p><ul><li><a href="https://sre.google/sre-book/table-of-contents/">Site Reliability Engineering</a> — <a href="https://sre.google/sre-book/postmortem-culture/">Chapter 15 on Postmortem Culture</a> is most applicable for this topic but the rest of the book and its companion <a href="https://sre.google/workbook/table-of-contents/">The Site Reliability Workbook</a> are great resources.</li><li><a href="https://response.pagerduty.com/">Pagerduty’s incident response</a> docs are fantastic, with the <a href="https://response.pagerduty.com/after/post_mortem_process/">Postmortem Process</a> page going into the mechanics.</li><li><a href="https://circleci.com/blog/value-of-blameless-culture/">The value of blameless culture — from IC to C-Suite</a> is also a great summary.</li></ul><p>Want to learn more? Ping me via LinkedIn, I’d be happy to chat.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=316c3a4ea8ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>