<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eddie Pettis on Medium]]></title>
        <description><![CDATA[Stories by Eddie Pettis on Medium]]></description>
        <link>https://medium.com/@epettis?source=rss-16431268d714------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IIblOVaCRzjIIObPIiPeWQ@2x.jpeg</url>
            <title>Stories by Eddie Pettis on Medium</title>
            <link>https://medium.com/@epettis?source=rss-16431268d714------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:58:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@epettis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sometimes you just have to act]]></title>
            <link>https://epettis.medium.com/sometimes-you-just-have-to-act-5a961c8ddb27?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a961c8ddb27</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Sun, 02 Nov 2025 01:23:03 GMT</pubDate>
            <atom:updated>2025-11-02T04:03:56.001Z</atom:updated>
            <content:encoded><![CDATA[<p>I attended Google’s EDGE leadership training in 2012 to learn more about how to function within and lead a high performing team. I definitely learned something, but I’m not sure everyone would agree with it.</p><p><strong><em>Warning: Major spoilers for Google’s EDGE leadership training.</em></strong></p><p>EDGE invites 20 leaders from assorted roles to a summer camp in the Santa Cruz mountains to play a series of games and that teach leadership skills.</p><p>We were bad. That’s not an opinion. The games are scored, and we were near the bottom in every one of them but one.</p><h3>The sorting game</h3><p>The first game gave each of us a number, blindfolded us, and asked us to sort ourselves without talking.</p><p>Someone quickly suggested writing the number on the other person’s back. Then, someone invented IEEE-754 floating point using their fingers. Despite the simplicity of just writing it on the back, the team insisted that IEEE-754 was more expressive and, therefore, better.</p><p><em>Fine.</em> We’re a team. I can get behind the plan, even if it’s so complicated that I don’t really understand it.</p><p>As soon as I looked at my number, I sighed and realized that the team blew it. My number was a googol (10¹⁰⁰). It turned out that the floating point system only went up to an exponent of 10 because we only have 10 fingers.</p><p>I got in line and just started swapping to the back because that’s a pretty big number. At some point, people started refusing and tried to feel my fingers. I rejected the plan, turned them around, and wrote 10¹⁰⁰ on their back. They all swapped until I got to the guy at the end who turned me around and wrote ∞ on my back. (He just got out of line and walked all the way to the back.)</p><p>Everyone laughed during the debrief. They hadn’t anticipated numbers like <em>e</em>, π, and so on. I pointed out that the simple solution worked well and would have been better. Better luck next time, I guess.</p><h3>The marble game</h3><p>Another game scored us based upon how many marbles passed through four independent structures built by four distributed teams.</p><p>Someone came by, took all of our building materials back to a central hub, and didn’t return them. As a distributed team, we needed a special pass to “fly to HQ”, which we didn’t have. We just idled for an entire quarter of the game.</p><p>We had an “all hands” at the halfway point where I pointed out that we were making exactly zero progress while they over-optimized for something we didn’t understand because they hadn’t communicated it to us at the remote site. They explained a bunch of scoring rules associated with color and aesthetics.</p><p>“Great! But can we have literally anything to build with?”</p><p>In the second half, my team tried to build anything meeting the constraints with the little time we had left. The facilitators gifted the teams a fifth quarter because the other teams hadn’t built much of anything. Ours was technically on time, but it jammed up because the twists were too tight, which we probably would have figured out sooner if only we had some parts to test with in the first half.</p><p>Again, everyone was laughing it off, but <em>we were failing</em>. That evening, a facilitator told us exactly how bad we were.</p><p>“Most groups are good at something, but you’re bad at everything.”</p><h3>The tennis ball game</h3><p>After two full days of suggesting changes to our methods, they voted me the leader as punishment.</p><p>“If you’re so smart, why don’t you do it?”</p><p>The rules were straightforward. We threw tennis balls to each other at a minimum distance of six feet. Every person had to touch the ball. The ball had to pass through all four quadrants of the game area. The score was the minimum of the balls cycled clockwise and the balls cycled counterclockwise within the time limit minus the number of dropped balls.</p><p>I told everyone to break into groups and practice tossing and catching tennis balls while we figured out what to do. I asked a timekeeper to let me know when we had 15 minutes left.</p><p>One team quickly figured out how to throw the balls back and forth in a zig-zag pattern where they could see the balls coming and going without pivoting.</p><p>Then, a neighboring team tried to catch my attention with a complicated circular motion. I asked for a demo. A minute later, they still hadn’t organized themselves.</p><p>“We’re not doing this because you can’t explain it to yourselves,” I said. They were visibly upset as I walked away.</p><p>At 15 minutes, I called everyone in for an all hands to announce the throwing strategy. People at the back weren’t paying attention, so I gave a loud “HEY!” to get their attention.</p><p>“We all have to do this. You need to know what to do, and I need to know you know what to do. Now, let’s practice it.”</p><p>We spent a lot of time practicing. Along the way, someone realized that we could pocket the tennis balls. We would fill the pipeline in both directions, pause, pocket the ball on one side, continue for a while, flush the pipeline, remove the ball from the pocket, and continue in the other direction until time ran out.</p><p>We got the second highest score of all time. We also dropped the fewest balls of all time.</p><p><strong>Nobody was happy.</strong></p><p>“I felt like I wasn’t being heard.”</p><p>“Eddie was everywhere and nowhere at once.”</p><p>“Eddie was a drill sergeant that wouldn’t let his marines get fragged.”</p><p>After 20 minutes of debrief, they asked me how I felt.</p><p>I said, “I just want everyone here to realize that this is the only time we were successful and nobody is happy.” I left them to think about that.</p><h3>What I learned</h3><p>This whole experience was incredibly revealing to me.</p><p><strong>I learned that many people don’t actually care whether it works or not.</strong> They just want to see themselves in the solution, which is a really dangerous quality in a leader. I take the accountability upon myself to ensure that it actually works. I do my best to reach out broadly and solicit input. Then, I point out changes so they see themselves in the final product and accept it.</p><p><strong>I learned not to allow myself to be trapped inside a box of my own making.</strong> I originally submitted to the team dynamic because everyone values group harmony. As the games played out, I became increasingly outspoken and direct with my feedback. Now, I jump straight to direct feedback because we don’t usually have time to pretend it’s working when it clearly isn’t.</p><p><strong>I learned that it’s often better to just play with the problem a bit while theorizing about its solution.</strong> We always discovered some way that theory broke down, allowing us to adapt our solution to match the ugly reality. A quick and dirty experiment usually pays for itself, even if you completely throw it away.</p><p><strong>I learned that somebody has to close discussion and make the decision.</strong> We kept failing because we spent too much time theorizing and designing and not enough time doing. People hated this because they were having too much fun theorizing about the problem. I usually end up being the bad guy who insists that we act.</p><p>I carry these lessons with me everywhere. I’m typically the person to call the end of discussion. I tend to summarize the discussion and stack rank the solutions so far to simplify the solution space. When time starts running out, I start acting even if everybody else wants to keep talking.</p><p>And <strong>I’ve learned that some people get unhappy about that because they haven’t learned the lessons that I have</strong>.</p><p>You may be unhappy with the solution’s elegance. You may feel that we didn’t incorporate all your brilliant ideas.</p><p>However, the real world is messy and complex. You do the best with what you have. You need to accept the ugliness, learn from the experience, and do better next time.</p><p>Done is better than perfect.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a961c8ddb27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wrangling complexity with breadth first problem solving]]></title>
            <link>https://epettis.medium.com/wrangling-complexity-with-breadth-first-problem-solving-19e838f4f6dc?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/19e838f4f6dc</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Sun, 26 Oct 2025 21:50:28 GMT</pubDate>
            <atom:updated>2025-10-26T21:50:28.137Z</atom:updated>
            <content:encoded><![CDATA[<p>Parachuting into a messy situation? Having trouble getting your bearings? Consider a breadth first problem solving approach instead.</p><h3>The manager who kept asking questions</h3><p>Back in 2018, I worked for a guy named Steve Webster who asks a <em>lot </em>of questions. He followed each question with another deeper question until I couldn’t answer anymore. We would laugh about how he was uniquely capable of pushing my buttons. I’d tell him, “That’s not important.” Then, he would proceed to explain why it was important, and I would politely disagree with him as he interrupted me with more questions.</p><p>It scared him that none of the engineers on my project could answer his questions either. They said, “Eddie said not to worry about that stuff.” It was hard to argue otherwise because everything was running smoothly. A few breakfast conversations later, we discovered that I solve problems differently from most people.</p><p>In my experience, most people (including Steve) take a depth first approach to problem solving: ask a question, uncover more questions while learning the answer, and repeat. This seems natural and works well for many problems. You might even say that it promotes mastery.</p><p>Unfortunately, purely depth first problem solving is impossible in a sufficiently complex ecosystem because you’ll never reach the bottom. At some point, you just have to stop, but the more junior you are, the harder it seems to be to stop. I’ve seen junior (and some senior) engineers get lost in a bottomless pit of frameworks, configurations, and documentation. As they go deeper, they become less informed about how the details work, leading to progressively worse decisions.</p><p>In contrast, I take a breadth first problem solving approach. Given a big question, I ask what smaller questions are necessary and sufficient to answer the big question. Once I get the answers to those questions, I stop digging.</p><p>On the surface, this seems to promote willful ignorance. I mean, why wouldn’t you want to know the details? Aren’t they important? Not really. It takes a lot of people to work on large, messy ecosystems. I can delegate those questions because we have experts who are paid to know the answers to those questions, and if I ever need the details, I can ask them.</p><h3>How to solve a problem in a depth first manner</h3><p>Stop me if you’ve heard this one: Someone shows up at your desk and asks a direct question about a nuanced internal system that they don’t work on. <strong>What do you do next?</strong></p><p>Before answering this question, consider the circumstances that might have led this person to you. They don’t work on this system, so something led them here: documentation, source code, or maybe someone else. They’ve spent a lot of time thinking about this question. They’re being respectful of your time because they’ve tried to find the answer themselves, similar to <a href="http://www.catb.org/~esr/faqs/smart-questions.html">how Eric S. Raymond recommends asking a good question</a>. They really are trying to do the right thing.</p><p>I usually answer their question with a question: “<em>What problem are you trying to solve?</em>”</p><p>Consider another example: Have you been in a review where another technical lead asks a nuanced internal question and proceeds re-designs your solution on the fly? These designs rarely work because they aren’t solving the same problem as you. Remember, <a href="https://epettis.medium.com/how-to-lead-large-programs-like-eddie-a203ebba4e0a">people solve problems based on their experience with the problem</a>. Put another way, their solution overfits for their use case. I’m trying to understand their perspective by asking them to describe the problem.</p><p>Next, I map their understanding of the problem to my understanding of the problem. For example, Google’s operations team once asked why the ML control system turned on chillers in the winter. It’s a perfectly valid question because that’s obviously a bad idea. However, reinforcement learning systems sometimes just do things to see what would happen. After some discussion, the problem was less about this particular example and more about <strong>how do I know that the ML control is working as intended?</strong></p><p>Now that we have identified the real question, we can start understanding the necessary and sufficient sub-questions we need to answer to answer that question.</p><ol><li>What was the controller’s decision?</li><li>Was it exploiting (minimizing energy) or exploring (learning new behaviors)?</li><li>What was the model’s confidence in the decision?</li><li>Is the confidence well calibrated?</li></ol><p>In practice, it’s impossible to determine if a statistical decision was “correct” or not based on a single data point because the model is predicting behaviors often outside its control: weather, user workloads, maintenance, and so on. We need to look at a lot of decisions to determine if their distribution matches the distribution the model expected.</p><p>These questions address a large class of problems about ML control decisions. Surprisingly, none of these sub-questions require us to understand the model’s underlying implementation. It could be monkeys with typewriters. All we need to know are the decision, the reinforcement learning mode, the statistical variance, and the actual results. This allows a bunch of software engineers with a rudimentary understanding of ML to tackle this problem without engaging the Deepmind ML experts.</p><p>In summary:</p><ol><li>Ask what problem is actually being solved.</li><li>Map the actual problem to your own understanding of the problem.</li><li>Identify which questions are necessary and sufficient to address the problem.</li><li>Answer those questions.</li></ol><h3>Other benefits of breadth first problem solving</h3><p>I’ve already explained how breadth first bounds the size of the problem. However, it’s useful in other ways.</p><p><strong>Subproblems naturally lend themselves to service oriented architectures.</strong> Consider an ecosystem where each question is answered by an API endpoint. High level questions hide implementation details about how the answers are decided. The answers are relatively sparse and general, leading to simplified data types to share between services.</p><p><strong>Breadth first takes benefits from specialization.</strong> This is the fundamental principle behind <a href="https://en.wikipedia.org/wiki/Division_of_labour">division of labor</a>. Think about all that time spent searching for and consuming low level details to harness even a fraction of your peers’ knowledge. Trusting your peers and leveraging their expertise saves your time and gets to a much better conclusion faster.</p><p><strong>Breadth first problem solving scales better for horizontal programs.</strong> Suppose you’re trying to solve a large problem across dozens of services. The number of sub-questions scales roughly O(1), whereas the required depth first knowledge tends to scale O(n).</p><h3>Criticisms of breadth first problem solving</h3><p>Breadth first problem solving is not without its own criticisms.</p><p><strong>Breadth first problem solving is just depth first with a bounded depth.</strong> This is entirely true. The key to breadth first problem solving is that it forces you to think about the depth as you’re solving the problem. You may have a few sub-questions with a depth of 1 and a lot at depth 2. You will never make it to depth 3 because you will realize that you have your hands full and start delegating.</p><p><strong>This description skips over the ability to ask the right sub-questions.</strong> This is also true. It’s also not unique to breadth first problem solving. I would argue it is easier to ask the right top level sub-questions than to ask the right questions many layers deep in an area in which you are unfamiliar.</p><p><strong>Breadth first problem solving unsettles depth first solvers.</strong> This was the heart of Steve’s concerns. You need to be very clear about your problem solving methodology to calm the herd. My original document on this subject (published at Google) was written to demonstrate that I really did know what I was doing.</p><p><strong>Breadth first problem solving encourages ignorance.</strong> Honestly, I just disagree. It requires a lot of knowledge to understand what it takes to definitively answer a question. It’s just different knowledge.</p><h3>Conclusion</h3><blockquote>“At some point in a Noogler’s progression, they realize that they have explored this large room and look under the rug only to find a trap door opening into an even bigger room.” — Andrew G. Morgan, my friend and carpooler</blockquote><p>To be very clear, breadth first problem solving is not strictly better than depth first problem solving. Done right, they’re both great ways to attack engineering problems. In my experience, it’s just too easy to get stuck trying to solve problems breadth first because you’re learning so much that you don’t realize you’re not progressing towards solving your initial problem.</p><p>Steve came to accept my way of solving problems. As I grew in my career, I found breadth first problem solving to be invaluable at addressing large horizontal problems, like reliability, tech excellence, and new product introduction. It helped me get my bearings quickly in messy areas like cloud capacity planning and inventory efficiency. Hopefully, it works for you as well as it has worked for me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19e838f4f6dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How DCSW created an amazing culture]]></title>
            <link>https://epettis.medium.com/how-dcsw-created-an-amazing-culture-25e22f7fcbcc?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/25e22f7fcbcc</guid>
            <category><![CDATA[organizational-culture]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Tue, 14 Oct 2025 09:48:29 GMT</pubDate>
            <atom:updated>2025-10-14T09:48:29.145Z</atom:updated>
            <content:encoded><![CDATA[<p>I would have never taken the call with Equinix in 2021 because I couldn’t imagine a happier situation. The work was hard, messy, and often thankless, but Data Center Software (DCSW) had a truly amazing culture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7sVwgPWneYUszSOaAESqiw.png" /><figcaption>The Brains versus Brawn Olympics (2013)</figcaption></figure><p>One of my former DCSW teammates recently reached out to me to ask how we managed to do that because he wants to start his own company and recreate that environment. Given that conversation, I thought it would be helpful to share my personal account of how we accomplished turning just another infrastructure team into The Data Center Software® team.</p><h3>Lead with metrics</h3><p>A lot of infrastructure teams spend their time fighting for more headcount, a messy refactor, a performance optimization, or any of a dozen things because infrastructure teams are generally cost centers. Resources flow to where the money is made.</p><p>We addressed this problem by systematically brainwashing the entire organization to ask “why”. We invested in rules of thumb to quantify things: how much a watt of power was worth, how much a software engineer’s time was worth, how much a technician’s time was worth, and so on. We taught people about <a href="https://en.wikipedia.org/wiki/Opportunity_cost">opportunity cost</a>. We taught people how to perform <a href="https://en.wikipedia.org/wiki/Fermi_problem">Fermi estimation</a> before even writing a line of code. We created a local design doc template whose first section was literally titled “Why?” and required approval before even describing “what” and “how”.</p><p>Everyone could explain <strong>exactly</strong> why their project was important, giving their work meaning and providing an advantage in promotion committees that directly led to more pay. People were less attached to “what” they were doing than “why” they were doing it, which gave everyone the mental flexibility to change their minds and adapt to changing priorities.</p><p>We were also great at filtering ideas. We innovated in ways that actually mattered. We self-initiated disk crushing robots, cloud-controlled cooling systems, sophisticated repair correlation, ILP-driven cluster planning, workforce planning, and power capping because they jumped off the page when we ran the numbers. We dropped several others when the numbers didn’t sort themselves out. That’s all working as intended.</p><p>Metrics gave us the ability to speak truth to power. They let us to <a href="https://epettis.medium.com/how-to-lead-large-programs-like-eddie-a203ebba4e0a">make our executives work for us</a>. We got what we needed because we could explain it to leadership. It was never <em>quite</em> enough to be comfortable, but it was never so low that we were overwhelmed. I consider that a win for an infrastructure team.</p><p>It’s an amazing feeling for your work to have purpose and be supported by your leadership.</p><h3>How can I help?</h3><p>The metrics sometimes don’t go your way. The problem is important, but the quality metric keeps going down and to the right. It sucks to be accountable when things aren’t going well. We solved this problem by simply asking, “How can I help?”</p><p>Adam Grant’s book “<a href="https://adamgrant.net/book/give-and-take/">Give and Take</a>” highlights that Givers are statistically overrepresented at the top and the bottom of the performance distribution across a variety of applications, whereas Takers cluster near the middle. Givers near the bottom are taken advantage of by the Takers because they give indiscriminately. Givers near the top are more selective and build communities of like-minded Givers, shutting out the Takers.</p><p>DCSW’s leadership shunned Takers. We had one manager who engaged in empire-building. We ran the math to show that their team was costing more than it was saving and the attrition was preventing us from hiring elsewhere. That manager was soon gone. We had numerous individual contributors who were more interested in writing a promotion project than solving the problem. They were encouraged to transfer elsewhere.</p><p>In contrast, leaders modeled the Giver mentality. We showed up to everyone’s engineering reviews to offer suggestions and draw connections between teams even when they didn’t really affect our day jobs. We offered constructive feedback on design documents in far flung areas. We stood up in support when a postmortem backlog showed leadership systematically underinvested in sustainable operations. We had one-on-ones with lower level engineers to understand the view from the “boots on the ground” and escalate perspective.</p><p>That attitude was infectious. Tech leads did the same for their teams. Teams helped each other. Everyone was in it together, and there was strength and wisdom in numbers.</p><h3>No assholes</h3><p>Bill Coughran once said, “Engineering is easy. People are hard.”</p><p>The “brilliant asshole” is a fallacy. They are the worst kind of Taker. Sure, there are people out there who are massively more productive than others at individual tasks, but it’s about solving the problem, not about getting things done quickly.</p><p>At scale, problems are too large to be solved by an individual. The complexity is too simply large to wrap your head around. There are too many moving pieces, byzantine processes, and interests. If your co-workers are reasonably smart people, <a href="https://fs.blog/chestertons-fence/">Chesterton’s Fence</a> suggests that all those are there for a reason. Nobody wants to work with an asshole, so an asshole will never build the relationships necessary to understand the true nature of the problem, let alone bring everyone along to solve it.</p><p>DCSW erased assholes. Anybody detected in the interview stage was quickly rejected. One guy would lose his temper with people in meetings or calls from his desk. We walked in one day to find that his desk had been replaced with a couch. (“Turned into a couch” became part of the DCSW slang for the next decade.) At least two more were put on performance plans and made clear that a turnaround was required, which they thankfully demonstrated and maintained.</p><h3>Having fun while doing serious work</h3><p>The best part: We had a lot of fun.</p><p>We gave the interns a chance to demonstrate leadership by having them organize the summer activity — a Brains versus Brawn olympics with events like a 400m relay where you had to recite the alphabet backwards to pass the baton. We created a “radio station” over Google Meet during the pandemic. We dressed up like each other for Halloween. We ran a white elephant party that was a full-on game show with an announcer, games of skill, and a host that we then adapted with drop shipping and a chat bot during the pandemic.</p><p>Culture isn’t dictated by your VPs. It isn’t an organized offsite. It isn’t a one-time event. Your culture is what you do every day.</p><p>We found a way to laugh every day. We were annoyingly loud so you really did need headphones to work in the office. (Sorry.) We made jokes in serious meetings. DCSW chat was a running gag reel, even after becoming the “Place where DCSW chat used to be” after DCSW was broken up in 2023 (in reference to <a href="https://99percentinvisible.org/episode/u-t-b-a-p-h/">Yinzers defining directions based on where Pizza Huts used to be</a>). We called out people who did a great job in front of their boss’s boss’s boss so everyone knew why.</p><p>And it was universally perceived as amazing.</p><p>You can’t just create a culture. You have to plant a garden, tend it, watch it grow, and blossom to create the next generation. You do the little things right every day and, eventually, you get something special.</p><p>I wish I knew a faster way, but that’s how we did it. With 5–10 years to spare, maybe you can, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25e22f7fcbcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interviewing with the eyes of a child]]></title>
            <link>https://epettis.medium.com/interviewing-with-the-eyes-of-a-child-058c08eef547?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/058c08eef547</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Sat, 05 Apr 2025 16:36:25 GMT</pubDate>
            <atom:updated>2025-04-05T16:36:25.587Z</atom:updated>
            <content:encoded><![CDATA[<p>Yesterday, I decided to play along when one of my Product Manager co-workers said her nine year old wanted to be interviewed. Her daughter (let’s call her Alice) brought a resume and everything. I dumbed down my typical question to a 4th grade vocabulary. Then, we conducted an interview that legitimately surprised me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MsvH6dOn3NKVIYmc97h87A.jpeg" /><figcaption>Image generated by Imagen 3.</figcaption></figure><p>My question consists of a few key parts:</p><ul><li>What should this thing do?</li><li>How does it work?</li><li>How do you know if it’s working?</li><li>What is your decision-making process for trading off between conflicting requirements?</li></ul><p><strong>What should this thing do?</strong> Most candidates describe exactly what a modern system does. Then, we take our time exploring other use cases the modern system doesn’t handle, which leads into the rest of the question. Alice described the futuristic use cases right from the start without any prompting. Those use cases subsumed the modern use cases.</p><p><strong>How does it work?</strong> Most candidates describe exactly how modern systems work without any modification. Despite being only nine years old, Alice described a credible, simple solution that addresses modern and futuristic use cases.</p><p><strong>How do you know if it’s working?</strong> Almost everybody knows how to do this. Alice didn’t know the technical terms, but she understood intuitively and described a test to measure it.</p><p><strong>What is your decision-making process for trading off between conflicting requirements?</strong> This question is what really separates hires from no-hires. The no-hires rely upon intuition. The hires use math and data to reveal that intuition is wrong. 90% of candidates cannot move beyond intuition. I needed to walk Alice through the process a bit, but she was able to reason through the problem by estimating a few parameters and come to the right balance of requirements.</p><p>I am legitimately shocked that a nine year old outperformed a majority of PM candidates. I‘ve spent the last 24 hours reasoning through how this could happen. I openly acknowledge that it’s not a completely fair comparison because I dumbed it down, guided her a bit more, and was likely biased a bit by how excited she was to even be doing this. However, I have come to a conclusion.</p><p><strong>Alice wasn’t constrained by experience.</strong> She had a fresh perspective with the problem and could only think about the problem from first principles, which naturally led her to a better solution.</p><p>Looking across my career, I’ve seen this happen before:</p><ul><li>Roboticists wanted contractors to program robots for each building. A bunch of software engineers scaled the process with a compiler for the teach pendant’s instruction set.</li><li>Controls engineers insisted upon five nines of availability for cloud-based cooling control systems. A software engineer and mechanical engineer developed the most efficient control system they could with only three.</li></ul><p>The lesson from this whole experience is that we would all benefit from spending more time looking at problems like a child to find opportunities that most people would miss because they’re blinded by their experience. This is also an argument in favor of diverse, interdisciplinary teams because they bring different experience to the problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=058c08eef547" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The biggest technical mistake I ever made: Encoding data in a string]]></title>
            <link>https://epettis.medium.com/the-biggest-technical-mistake-i-ever-made-encoding-data-in-a-string-f18dd40165f1?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/f18dd40165f1</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[naming]]></category>
            <category><![CDATA[naming-conventions]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Fri, 17 Jan 2025 01:08:25 GMT</pubDate>
            <atom:updated>2025-01-17T01:08:25.797Z</atom:updated>
            <content:encoded><![CDATA[<p>They say the two hardest problems in computer science are naming, cache invalidation, and off-by-one errors. I screwed up naming sensors.</p><p>To my great shame, the naming scheme for Google’s power monitoring systems has endured for 15 years because it is the single most infectious piece of technical debt in all data center software.</p><p>If you take nothing else away from this post, <strong><em>do not encode data in a string</em></strong>.</p><p><em>Disclaimer: These examples are contrived to illustrate the point. Don’t @ me about real power hierarchies.</em></p><p>The simplest model for power hierarchy is a tree, rooted at the utility substation as shown in the picture below. Everything is nice and orderly. Transformers A and B are connected to the utility substation. Transformers A-1 and A-2 are connected to Transformer A. Transformers B-1 and B-2 are connected to Transformer B.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/0*yXCYGFa6J7I1sZxb" /><figcaption>Simple power hierarchy</figcaption></figure><p>These names are perfect. You can tell exactly where you are in the power hierarchy strictly by the name. <em>Humans and computers alike rejoice!</em> It’s the perfect MVP: simple, elegant, and quick to iterate upon. I get chills just thinking about it.</p><p>Unfortunately, data centers use generators to provide backup power. The figure below illustrates this situation. An automatic transfer switch (ATS) will switch between the utility source and generator source, depending on power availability. This is a little more complicated, but it’s still pretty easy to tokenize the name into the power hierarchy. <em>Launch and iterate, they say!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*ScJGb_iKQtGmuQGD" /><figcaption>Power hierarchy with generators</figcaption></figure><p>Now suppose that a generalist software engineer building capacity management software asks you what power is being consumed on A-1. They don’t understand the physical aspects, so you just decide to return the value from the ATS to save a lecture on power hierarchy.</p><p>The refactors started happening a few years later. Data centers are too expensive to rebuild from scratch, so it’s common to refresh the data center equipment in place.</p><p>Now consider the example depicted below where Transformer B-2 and Generator B-2 are being replaced. The loads on ATS B-2 are low enough to be powered from a newly upsized ATS A-1 without exceeding its breaker limit, so the design team decided to supply power to ATS B-2 from ATS A-1. This allows the machines nominally powered by ATS B-2 to continue functioning during the upgrades on Transformer B-2 and Generator B-2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*4J4JbvBj2Y_dnTiA" /><figcaption>Refactor powering ATS B-2 from ATS A-1</figcaption></figure><p>You start to realize your mistake when that engineer comes back to ask how much power is being consumed by A-1.</p><p>You must distinguish whether the question is about Transformer A-1, Generator A-1, ATS A-1’s nominal loads, or the output of ATS A-1. This is hard because all the data consumers have been asking for the simpler name A-1, which you have been translating on the backend to be the output of ATS A-1. However, that is not the load your users want because capacity software needs to calculate both (a) how much total electrical current is flowing through the breaker A-1 and (b) how much current is flowing into only the nominal A-1 loads because they need to calculate the thermal load in that space.</p><p><em>It is impossible to answer both questions correctly using the existing solution.</em> The only way to determine both answers is by asking a more complex topological question, but you have encoded everything into the name, which has been broadly adopted during the previous few years. You are stuck. The only answer is to change the API to ask a more specific question, which imposes a greater burden on all the consumers who must also migrate to it.</p><p>There are so many more examples. Sensor naming was on the critical path for every new data center architecture. Some sites enumerated electrical panels rather than naming them hierarchically. Nameplates on the floor didn’t match the names in the software. Refactors often involved manual configuration changes. It became easier to silence meter health alerts for complex cases, rather than fix the software. Renames required extra care and broke all dashboards using those values.</p><p>And it’s all because we encoded data in a string.</p><p>The migration started in 2018. To my knowledge, they still have not switched to a topological model for 100% of applications.</p><p>There are lessons to be learned through this cautionary tale:</p><ol><li><strong>A uuid is better than a structured name.</strong> <a href="https://www.hyrumslaw.com/">Hyrum’s Law</a> states that users will depend upon all observable behaviors of your system. Unsurprisingly, all the consumers tokenized the string and used whatever local interpretation made sense instead of a common understanding of power hierarchy. A uuid for each sensor would have required consulting a hierarchy that gave stable answers.</li><li><strong>Translation to a human-readable name should always be the last step.</strong> At some point, a human needs to know what they’re looking at and map it to the physical world, but computers don’t care. All APIs except a human-readable-to-uuid mapping call should accept only uuids to enforce this requirement.</li><li><strong>Beware oversimplification.</strong> Electrical hierarchy is complicated. Cooling hierarchy is even more complicated. Neither is truly static. Also, they aren’t technically hierarchies in their most general form. You need to accept that some degree of knowledge is required to perform scientific and computing tasks. Lean into this complexity with good documentation and training.</li><li><strong>You cannot go to space with a better hot air balloon.</strong> Agile philosophies have created this illusion that we can do anything through small iterations. At some point, you need to stop and rethink your approach. This will rarely seem like the correct answer in the moment because the greedy decision to add one more hack is always cheaper than refactoring and migrating to enable more advanced use cases.</li></ol><p>I wish I could say this problem was limited to power hierarchy. I have seen the same problems appear parsing DNS names, location names, and spatial mapping. Don’t let it happen to you. Just say “no” to structured names.</p><p><em>My condolences to Anton Savelyev, who has faithfully spent years undoing the mess I created as an ignorant, newly minted software engineer.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f18dd40165f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breaking things down for executives and engineers]]></title>
            <link>https://epettis.medium.com/breaking-things-down-for-executives-and-engineers-2006a8b14338?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/2006a8b14338</guid>
            <category><![CDATA[program-management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 23:14:40 GMT</pubDate>
            <atom:updated>2025-01-16T23:14:40.479Z</atom:updated>
            <content:encoded><![CDATA[<p>Engineering presenters often use the wrong level of abstraction when communicating with executives.</p><p>Without guidance, presenters speak about the familiar. Since engineers spend their days working on a series of tasks, they provide a lot of detailed information devoid of business context. In contrast, executives want to see a business outcome — increase the value from X to Y. This usually results in a derailed presentation as executives try to make sense of the details.</p><p>You can address this discrepancy by creating a three layer hierarchy:</p><ol><li><strong>Outcomes:</strong> Things your VPs care about, usually changing a metric from X to Y. You could justify a party when completing an outcome.</li><li><strong>Milestones:</strong> The necessary and sufficient deliverables required to achieve an outcome. You would assign a milestone to a program track.</li><li><strong>Tasks:</strong> The necessary and sufficient engineering deliverables required to complete a milestone. You would assign a task to an engineering tech lead.</li></ol><p>Your presentations to VPs should focus on outcomes. A roadmap shows how the metric will move every quarter by completing a set of milestones. Progress reports show how the metric has changed based upon completed milestones. The focus is the outcome, whereas the milestones are merely in support of the metrics.</p><p>You can often coordinate your programs by grouping milestones into tracks. The track lead can be a product manager or a strong technical lead and paired with a technical program manager to roll up milestone statuses into outcomes. Milestones complete on approximately a monthly cadence.</p><p>Engineering teams focus on completing tasks. They can break the tasks down further as needed with whatever granularity is needed to get things done. Just mark the tasks as completed so the milestones can be closed out.</p><p><em>Special thanks to my friend and former colleague Luiz Mendes for reminding me about this topic. Glad it has helped!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2006a8b14338" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transitioning from a cog to a VP]]></title>
            <link>https://epettis.medium.com/transitioning-from-a-cog-to-a-vp-2a747eb8df24?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a747eb8df24</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Sun, 22 Dec 2024 22:59:35 GMT</pubDate>
            <atom:updated>2025-03-05T17:15:52.786Z</atom:updated>
            <content:encoded><![CDATA[<p>One year ago, I transitioned from a cog in the machine at Google to a new job as a VP-level individual contributor at Equinix. On one hand, it was a great opportunity to transform a Fortune 500 company’s physical infrastructure. On the other, the responsibility was terrifying.</p><p>Given the uncertainty in my own move, I thought I’d share the most poignant lessons from my first year at Equinix.</p><p><strong>Doing the right thing is the only important thing once you hit the level cap.</strong> I literally can’t be promoted without becoming a manager. It’s liberating to focus on the best thing for the user without concern about whether the problem shape is good enough for an anonymous promotion committee. I never realized how much I wanted that feeling.</p><p><strong>Being a VP comes with a radical increase in responsibility.</strong> My bonus is linked to company goals, rather than personal goals. This incentivizes me to do what’s best for the company every day even when it’s uncomfortable or unpleasant. Just a few of these things include partnering with other VPs to resolve team personality conflicts, setting aside my own work for executive staff priorities, and LARPing as a chief of staff to ensure the Office of the CTO roadmap set the standard for other organizations.</p><p><strong>People may not like you.</strong> My friend Steve Webster once said, “At some point in your career, you need to choose between being liked and being effective.” My role requires me to agitate for change, which upsets some people whose areas are transformed by my recommendations. Since I am accountable to company-wide goals, I must accept that those people may not be happy with the decision and move on to the next problem.</p><p><strong>Change requires quantifiable accountability.</strong> Galileo said, “Measure what is measurable; make measurable what is not.” Measuring outcomes is essential because the default state is to focus on effort, rather than outcomes. As we have quantified accountability, teams have re-focused on the tasks that are necessary to make these numbers go up and to the right. The magic words to start the conversation are, “How do you quantify success?”</p><p><strong>Define the architectural boundaries before the organizational boundaries.</strong> <a href="https://en.wikipedia.org/wiki/Conway%27s_law">Conway’s Law</a> states that organizations are destined to recreate systems matching their org charts. Therefore, the obvious solution is to make the org chart match the architectural boundaries. These boundaries should have quantifiable accountability measures as described above. This will upset some people whose organizations do not align with the new boundaries. You just need to accept that being disliked is part of the job.</p><p><strong>The C-suite is full of approachable people.</strong> I was scared about being in a role where I would present to the C-suite. Now that I’m on a first name basis with them, I realize that they’re people, too. Our current CEO remembers everyone’s name and backstory because she used to be a high school teacher. Our CTO loves his native French wine, but he begrudgingly admits that Testarossa makes good Californian wines. Our Chief Business Officer is educating me on the history of hip-hop. I trash talk college sports with our Chief Digital Officer because our schools are old rivals. They just seem unapproachable because they have a lot going on.</p><p><strong>You can move the entire company if you have the data.</strong> We have transformed how Operations thinks about data center management by measuring the ineffectiveness of manual configuration. We straightened out Operations’ AI application priorities by building financial rules of thumb and sorting them by impact. These efforts landed me on the company-wide value stream prioritization group, and I think I made a positive difference there.</p><p><strong>Corollary: Don’t approach executive staff until you have data.</strong> The Office of the CTO made a proposal to E-staff using rough estimates. The rough estimates struggled against scrutiny, so we made a more detailed pass. Even though the revisions are more convincing, we are playing from behind because we lost credibility. We would have been better off if we had taken more time to do the hard work up front because we wouldn’t need to rebuild trust for the program.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a747eb8df24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You only get one chance to ask everyone to do something]]></title>
            <link>https://epettis.medium.com/you-only-get-one-chance-to-ask-everyone-to-do-something-86527abf9963?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/86527abf9963</guid>
            <category><![CDATA[technical-leadership]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Sun, 26 May 2024 22:31:28 GMT</pubDate>
            <atom:updated>2024-05-26T22:31:28.212Z</atom:updated>
            <content:encoded><![CDATA[<p>Most horizontal software programs die because they are organizationally difficult to prioritize and technically hard to execute.</p><p>This guide describes how I have learned to run horizontals. So far, I have an 80% success rate using this technique. To briefly summarize:</p><ol><li>Know exactly what you will do with it.</li><li>Generate a sample output to verify it is useful.</li><li>Convince leadership that it is really important.</li><li>Create user-centric instructions.</li><li>Pilot the instructions with a trusted group</li><li>Support everyone as they do it.</li></ol><h3>Context: The Reliability Reset disruption</h3><p>Google Cloud leadership sent out an email in July 2019 requiring all services to immediately stop feature development and resolve multi-zonal reliability issues immediately. In the end, I think this was one of the best things to ever happen to Cloud, but describing it as “bumpy” would be an understatement.</p><p>The Reliability Reset was announced days after I started my role as Uber Tech Lead for Data Center Software (DCSW), so I volunteered to lead DCSW’s response. I immediately faced a mountain of problems including:</p><ul><li>What exactly is a reliability risk?</li><li>Should every small concern be filed as a risk or should we bundle them into larger risks?</li><li>“Zone” is a Cloud-specific term. What does a “zone” mean for non-Cloud?</li><li>What about global services that aren’t on the serving path?</li></ul><p>Every team in Cloud was left to make these decisions themselves. It was abject chaos. Teams who diligently identified risks were vilified by leadership dashboards, whereas teams that phoned it in had fewer hassles because leadership didn’t account for differently sized risks. The horizontal leadership thrashed service owners by changing their guidance as they learned more about the edge cases.</p><p>I made quite a few unilateral decisions inside DCSW to help people make progress, but none of them were corroborated by the people running Reliability Reset. It took months to answer these questions, leading to confusion and wasted time. Once things settled down, my technical program manager and I spent some time writing a postmortem to determine what we should have done instead. Since then, I have run multiple horizontal programs using those lessons. This post describes those lessons.</p><h3>You only get one chance to ask everyone to do something</h3><p>If you take nothing else away from this post, remember this principle: <strong>you only get one chance to ask everyone to do something</strong>.</p><p>If you don’t get it right the first time, teams will be universally angry and skeptical that it will work the second time. It is highly unlikely you will get a second chance. In the unlikely event you do get a second chance, it will be excruciatingly painful to execute.</p><p>Even if you get it right the first time, you will still have challenges. It may be <em>your</em> most important task, but every team out there has their own goals and priorities. You are asking them to preempt their normal priorities, which has a high bar. You need to convince their leadership to prioritize your request over their own and provide them with clear enough instructions that they don’t regret it.</p><p>You need to do a lot of homework before even making the request to ensure the request is actionable. A horizontal request will hit lots of edge cases. You need to understand most of those edge cases before reaching out to keep teams from weaseling out of your project because it doesn’t make sense for them.</p><h3>Running example: Capacity reassignment latency</h3><p>I will use an example throughout all these steps to illustrate the process.</p><p>I once worked on a capacity management project that needed to know how long it took various capacity operations to complete so we could build a roadmap to improve them. The operations were dependent upon the amount of capacity being reassigned. Each step took a highly variable amount of time.</p><h3>Step 1: Know exactly what you will do with it</h3><p>Imagine what the desired outcome looks like. What specific thing can you do with it? How does that fit into the broader ecosystem? This outcome should be impactful enough to justify asking everyone to do the work required to achieve it. If you can’t clearly articulate the outcome, do not proceed because you will never get everyone to do it anyway.</p><h4>Example</h4><p>We wanted the capacity management roadmap to be data-driven. Since capacity management was really important, it was pretty easy to get people on board with it.</p><h3>Step 2: Generate a sample output to verify it is useful</h3><p>Fake the outcome you expect from each component to determine if it is sufficient to generate the desired overall outcome. You should write a paragraph or two explaining how to interpret the output to ensure it makes sense.</p><h4>Example</h4><p>Capacity operation latency is complicated to model and visualize because the latency is dependent on the size of the operation and the time required in each step varies randomly. Each step can generate parallel sub-steps, each of which also varies randomly and can generate sub-steps.</p><p>The output needs to visualize percentiles to account for each step’s variability. This is complicated because the percentiles of individual stages are greater than the percentiles of the whole. We came up with a visualization that looked like the following picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/0*34qsarmDKt-j3L4e" /><figcaption>Sketch for measuring latency for capacity operations with parallel steps</figcaption></figure><p>I’ll spare you the (many) details. This visualization enables us to identify what steps contribute the most latency at each percentile, empowering us to create the improvement roadmap for the program. Each figure illustrates a different operation size.</p><p>The sample data required to generate this image represents each service’s sample output. Each step needs to provide its total execution time and parent-child relationships to its subtasks so we can separate how much time is spent by the parent versus waiting for the children to complete.</p><h3>Step 3: Convince leadership that this is really important</h3><p>You don’t need to convince every team to do something. You need to convince their leadership to have their teams do something. Then, you need to ensure there’s no credible excuse for their teams weaseling out of it (see steps 4–6).</p><p>You accomplish this by converting the output into an impact metric and gaining leadership support for actualizing that impact by disrupting every team’s roadmap. Ideally, you can convert this into financial impact. This step may require some quick and dirty hacks to measure a proxy metric.</p><h4>Example</h4><p>We wanted to build a roadmap, but we couldn’t estimate the opportunity without the result. Instead, we measured the end-to-end latency without the detailed breakdown using the low-fidelity data available, planted a flag in the ground with a credible target, and measured the financial impact of moving from the current state to the target state. We would then revise the target once we had the data and the roadmap.</p><h3>Step 4: Create user-centric instructions</h3><p>You need to clearly explain what you want users to do in a way that minimizes errors because you want to minimize friction. You also want to make it so easy that the teams can’t convince their leadership to de-commit from the request.</p><p>This process is somewhat unintuitive. The key is to remember that users just want this request to go away so they get back to their work. Some tips include:</p><ul><li><strong>Design documents are not instructions.</strong> Users don’t need the background, alternative implementations, or system designs. They just want this request to go away, so create instructions that are exclusively instructions.</li><li><strong>Push all extraneous information to the appendix or links.</strong> This is not the time to show off. Your readers do not need to understand why. They don’t need to know what you’re going to do with it. They just want this request to go away.</li><li><strong>Put the instructions first.</strong> People don’t want to read eight pages of flavor text before the instructions. They just want this request to go away.</li><li><strong>Link to sample implementations.</strong> Ideally, you should provide examples that are easily copied and pasted. The faster the user can make this request go away, the more likely it is to be completed.</li><li><strong>Provide an escalation process if users encounter problems.</strong> Email and Slack channels work moderately well. Office hours are amazing. You can use the escalations as an indicator that your instructions may need to be improved.</li></ul><h4>Example</h4><p>We defined a structured schema for logging operations, wrote instructions to apply them, and provided examples of existing implementations. Questions were directed to a mailing list. The questions were usually root caused to not reading the instructions.</p><h3>Step 5: Pilot the instructions with a trusted group</h3><p>You only get one chance to ask <em>everyone</em> to do something. However, you may have some allies that enable you to iterate. Use these trusted allies to test the instructions, highlight edge cases, and iterate until the instructions are crisp.</p><h4>Example</h4><p>We applied the logging to our machine disruption manager first. This gave us the ability to measure some important operations without other services’ help and refine our roadmap. We identified interesting edge cases that we incorporated into the schema. This pilot also revealed that our visualization needed to account for cases where capacity assignments were blocked by two parallel services.</p><h3>Step 6: Support everyone as they do it</h3><p>Finally, you ask everyone to do it.</p><p>In this step, you vigorously address every escalation your users identify to create a positive experience. Over time, you may discover that the instructions can be simplified further based on user feedback. You may elect to broadcast improvements to an announcement email alias as they are discovered.</p><p>You also need to track completion using some form of coverage metric to identify stragglers. These stragglers should be escalated to their leadership.</p><h4>Example</h4><p>Thanks to our pilot, we knew how long the coarse steps required, allowing us to determine what fraction of the time was unaccounted for. Then, we escalated stragglers to leadership.</p><h3>Why hasn’t this been 100% effective?</h3><p>I mentioned earlier that I have an 80% success rate with this process. We failed to successfully execute a technical debt horizontal using this process because the difficulty increased to the point where leadership support dropped.</p><p>DCSW developed a process for measuring and managing <em>new </em>technical debt that was working quite well. Since we were having success with it, we decided to give an engineering review describing our approach. Our second-level VP loved the success story and wanted to see it across his entire organization. He nominated some collaborators, and we were off to the races.</p><p>Then, his nominees wanted to go even further by cataloguing <em>existing</em> technical debt.</p><p>DCSW only cared about new technical debt because it was easy to track as we created it. Existing technical debt requires surveying all the existing code, which is extremely time consuming and useful solely for building a baseline metric. I brought this up very early. However, the nominees insisted, which increased the horizontal’s difficulty and enabled their teams to weasel out of the request after 1.5 calendar years of effort.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86527abf9963" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a large system diagram]]></title>
            <link>https://epettis.medium.com/how-to-build-a-large-system-diagram-85b8e4e79859?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/85b8e4e79859</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 20:01:57 GMT</pubDate>
            <atom:updated>2024-02-05T20:01:57.241Z</atom:updated>
            <content:encoded><![CDATA[<p>You have a problematic ecosystem and decide to diagram the entire thing. Now you have two problems. These suggestions can help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dUVaEOP59DoeXxZmpxtMCQ.png" /><figcaption>Image generated by Dall-E 2 AI. Most system diagrams are as non-sensical as this one.</figcaption></figure><p>Stop me if you’ve heard this story before. A new principal engineer shows up and asks how everything works. Since nobody can explain it, the principal engineer recommends that we all draw a system diagram so future hires can reference it.</p><p>Odds are, this effort will silently fade into the background after about 2–3 months because it is way harder than everyone thinks. Believe it or not, it is actually possible to draw a system diagram successfully, but you need to set some expectations up front. This post describes how to avoid common pitfalls of naïve system diagramming based on my involvement in several failed attempts.</p><h3>Focus on a single important user journey</h3><p>Most people want to draw the entire system diagram. This will fail because the scope is too large. There are just too many edges coming out of each service for every possible use case, leading to too many nodes and infinite scope. The sheer number of people involved will become too expensive, and the effort will die.</p><p>You can justify the effort by focusing on a single important user journey. This diagram helps optimize latency, debugging, complexity, etc. for something valuable. It’s even more effective if you can approximate that value. For example, we drew a system diagram for cluster deployments because each day of delay was worth a ton of money. Understanding that process was critical to optimizing latency.</p><h3>Decide if you want a system or sequence diagram</h3><p>Everyone defaults to a system diagram. I think this is a mistake.</p><p>If you want to know how something works, you almost certainly want a sequence diagram because it shows how data propagates through the ecosystem to accomplish the task. You start at the top and read to the bottom, illustrating how information flows from start to finish.</p><p>If you want to understand dependencies, you almost certainly want a system diagram because it shows the relationships between services. You can point at a single service and know exactly what can break it.</p><p>I point out this distinction because a system diagram rarely explains how things work. System diagrams often have back edges. They are often unclear about where the initial request begins or ends. A system diagram requires additional description that is self-explanatory in a sequence diagram.</p><p>Furthermore, sequence diagrams can often be delegated by asking a service tech lead, “What happens after they call your RPC?” It was easy for me to create a detailed end-to-end sequence diagram for capacity assignment operations by starting at the beginning and talking to the tech lead associated with each RPC called by the previous person I spoke with. The problems became clear, demonstrating what was needed to accelerate the process.</p><h3>Prefer high-level nodes over low-level nodes</h3><p>I used to run a service called the Eye of Sauron that collected and archived power and cooling data [1]. As a software team, we distinguished the Eye from our visualizations, health pipeline, and control systems. The operations teams referred to everything our team produced as “The Eye”.</p><p>Now imagine that we’re both drawing a system diagram. The Eye of Sauron team produces a detailed diagram that the operations team considers a single node. Each team has a valid view of the world, but they can’t be merged easily because someone needs to understand the detailed view well enough to understand what the edge on the high-level view means. This gets worse when the nodes do not neatly overlap.</p><p>Another problem occurs when one diagram defines a node by service boundaries and another diagram uses team boundaries. We need to reconcile the team boundaries to the service boundaries. This gets worse when the team is a virtual team managing a workflow involving several services.</p><p>You can only draw a system diagram if everyone agrees on a node’s granularity. My personal recommendation is to prefer higher-level nodes to lower-level nodes to hide complexity. When in doubt, group the nodes as external users view them, rather than service owners.</p><p>This may lead to conflict if external users consider two different services from two different teams to be part of the same node. If you’re senior enough to engage with it, this should lead to a conversation about why they’re two teams and services in the first place.</p><h3>Define an edge’s direction and label very carefully</h3><p>Suppose service A initiates an RPC to get data from service B. Which direction does the arrow point? What label do you place on the edge?</p><p>You have at least two choices for each. If you define the direction based on the RPC’s initiation, it points from A to B. If you define the direction based on the data flow, it points from B to A. The edge may be labeled by the RPC being called or the data being returned.</p><p>I argue that the selections are dependent. If you define the direction based on RPC initiation, the edge should be the RPC name. If you define the direction based on the data flow, the edge should be the data being returned. If you swap labels, it gets confusing because it looks like service A provides the data to service B or that service B calls service A’s RPC.</p><p>My recommendation differs based on whether you’re doing it by hand or through automation. If you’re drawing the diagram by hand, define the edges based on data flow because it is more natural. If you’re using automation, it’s too hard to determine the data flow’s direction, so use RPCs because it’s always correct.</p><h3>Create norms for joining subgraphs</h3><p>I have alluded to this a few times already. Individual tech leads will provide their view of how their service works. These drawings will differ from adjacent drawings. They will include more or fewer edges. The edges will have different labels. The edges will connect different nodes. Someone needs to reconcile the merges.</p><p>You can generally reduce the merge conflicts if you are using higher-level nodes. I generally ask the tech leads to clarify the relationship for each others’ clarification. This process requires you to set rules for whoever does the merge.</p><h3>Use Graphviz until late in the process</h3><p>You will drive yourself insane trying to draw the graph by hand. Please trust me on this.</p><p>Use the <a href="https://graphviz.org/doc/info/lang.html">Dot programming language</a> to draw the graph. It will redraw the layout every time you add a node or an edge. Representing the drawing in this language will reveal the merge conflicts at data entry. Sadly, no open source tool exists to merge two Dot diagraphs automatically, so you’ll need to do some copy-paste or write the tool.</p><p>Once you have a really good idea of what the drawing looks like, you can either adjust the Graphviz layout until you’re happy with it or create a final hand-optimized drawing using the Graphviz layout as a starting point.</p><h3>Automate discovery or accept obsolescence</h3><p>If you build a diagram by hand, it will be obsolete before you complete it because services are constantly evolving. That may be okay if your drawing is extremely high-level, but it will never work at lower levels. The only way to maintain a lower-level diagram is to automate its creation.</p><p>When you try this, you will immediately discover that all automated diagrams are terrible because the granularity is too fine to be useful. For example, a service replicated in each location should be one node, not hundreds of nodes. Similarly, everything behind the externally-facing API should be hidden inside the node. Also, it’s not terribly useful to know that a centralized monitoring platform is scraping your metrics or how your release automation calls your service.</p><p>At Google, another team wrote a service that grouped jobs and RPCs by their accounting users. I used the accounting user as the node granularity and wrote automation that aggregated all their data into hierarchical SVG drawings with hyperlinks between them [2]. The big drawing showed only the big nodes by accounting users. Clicking on the big nodes would show you a subgraph with all the internals for that service connected to the big nodes from adjacent services. This drawing could be generated automatically to keep everything up to date.</p><p>It was still too detailed out of the box, but you could make it useful by spending some time reviewing the graph. I limited the initial scope to a small set of services known to be part of the user journey. Then, I reviewed the output and expanded or pruned the scope based on chats with tech leads. It was pretty good for understanding how machine management operations worked until I switched to a sequence diagram.</p><p>You can use structured logging as another approach. If your logs include consistent entity ids, you can produce a topological sort that approximates a sequence diagram and go from there. Retries are annoying to deal with, but you can use the transition frequency as a heuristic. I never verified this approach, but it seemed reasonable.</p><h3>Accept usefulness over perfection</h3><p>Your diagram will not be perfect. The sooner you accept this, the sooner you can admit that usefulness is the goal. Define your use case early and solve only that problem.</p><p>The most useful system diagram we ever built had about 10 blocks in it for a team of 2,000. It was cartoonish and manually generated, but it explained to new hires the rough process for deploying clusters and who to ask about the steps. The drawing was useful three years later because the big blocks didn’t change enough to justify updates.</p><p>Another useful diagram was a cartoonish sequence diagram I used as a crash course for how deployments, repairs, upgrades, and decommissioning worked. It had only 15 blocks but was useful enough to train new team members about how our entire organization worked.</p><p>My capacity assignment sequence diagram was useful because it answered questions on the spot. I was the only one who used it directly, but I could say with certainty how things worked because each service’s tech lead had code reviewed my sequence diagram for accuracy. There was no need for another meeting to discuss questions because I already had the answer.</p><p>Each of these diagrams were successful because they were useful. I hope these lessons will help you find similar success. Good luck.</p><h3>Footnotes</h3><p>[1] The Eye of Sauron saw everything in the data center, and its gaze produced a focus ray of hatred on bigtable tablets during site turn ups. The name was funnier at the time, and I really wish we had randomized the row keys.</p><p>[2] Fun fact: I started writing this tool to merge Dot diagraphs because no open source tool existed. Unfortunately, it used a ton of internal Google libraries, so I couldn’t open source my solution. Sorry.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85b8e4e79859" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Eliminating normalized deviance from software teams]]></title>
            <link>https://epettis.medium.com/eliminating-normalized-deviance-from-software-teams-f38fd844a520?source=rss-16431268d714------2</link>
            <guid isPermaLink="false">https://medium.com/p/f38fd844a520</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Eddie Pettis]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 00:02:51 GMT</pubDate>
            <atom:updated>2024-02-01T14:51:24.586Z</atom:updated>
            <content:encoded><![CDATA[<p>In my experience, normalized deviance is the most common problem plaguing dysfunctional software teams. Its symptoms include technical debt, heroism, alert spam, and frequent outages. Too many leaders look for technical solutions to what is actually a cultural problem.</p><p><a href="https://en.wikipedia.org/wiki/Normalization_of_deviance">Normalized deviance</a> occurs when people become accustomed to disregarding best practices. For example, MRSA bacteria spreads in hospitals when clinicians do not wash their hands properly. Normalized deviance has been cited in the Challenger space shuttle disaster, airplane crashes where co-pilots defer to pilots in dangerous situations, and many other situations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7pEtKOlhlbX2-Niz5K7KZA.jpeg" /><figcaption>The Challenger space shuttle exploded because the solid rocket booster o-rings shrank in the extremely cold weather. Engineers predicted the problem prior to the launch but were overruled by management because previous launches had been successful, despite o-ring shrinkage (<a href="https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster">more info</a>).</figcaption></figure><p>When confronted with poor business outcomes, tech leads (TLs) will often propose technical solutions such as a deprecation or re-architecture. However, a year later, the team finds itself right where it started, often half-finished and now maintaining an even bigger mess.</p><p>When you really dig into it, the team usually has developed a culture that normalizes deviance. Maybe the team is overcommitted because they don’t feel empowered to say no, leading to technical debt and heroism. Maybe the team doesn’t understand how their system works, so they create alerts for too many things. Maybe the team just wants to get things done or get promoted, so they focus on building new things without regard for whether the existing things are still working.</p><p>No matter how much technical expertise you throw at the problem, it won’t get better until you fix the culture. As a senior individual contributor (IC) tasked with fixing the technical problem, you have no choice but to wade into the organizational morass and correct the culture. This post describes how I tackle such problems.</p><p><em>The rest of these instructions assume that you are a senior IC. However, you can apply many of these same techniques if you are a service TL or manager.</em></p><h3>Step 1: Admit you have a problem</h3><p>You will never solve the problem until the team accepts that they have a problem. This is hard because the team has normalized the current state. The team will always have a justification for their behavior. <em>We have so many half-finished migrations! We need three oncalls because our P0 bug volume is so high! Leadership just won’t listen!</em></p><p>You need to spend time with the team to identify what abnormal behaviors can be measured. Wherever possible, these should be aligned with business outcomes. A few recommendations include:</p><ol><li><strong>All published service level objectives (SLOs).</strong> A service should be able to meet its existing promises. This includes bug closure SLOs.</li><li><strong>Alert closure latency.</strong> A service should be able to keep up with its alert volume.</li><li><strong>Data accuracy.</strong> Any service publishing data should have a measure for its quality. There is little value (if not negative value) in publishing inaccurate data.</li><li><strong>Successful releases.</strong> A release is successful if it meets a quantifiable canary criteria and executes in production for at least a week or the next release, whichever is sooner.</li></ol><p>Chances are, the team is struggling in many, if not all, of these metrics. In fact, you will probably find that the team will claim an inability to even measure some of them. You should sit down with the TL and manager to review what they have and identify the gaps.</p><p><strong>This next moment is critical.</strong> You need to make it clear that you are going to help them but they need to trust you because the next steps will be uncomfortable for them. You must be willing to advocate for them with leadership to provide the support they need, and you must follow through on your promise until the team reaches steady state. If you can’t commit to that, find a new project because you’re sacrificing your credibility on future projects if you ask them to step out onto a limb and leave them there.</p><h3>Step 2: Figure out what the team really needs</h3><p>The team likely has an idea what to do. Thing is, it probably won’t work.</p><p>Teams in this situation are accustomed to not getting what they need to be successful. They address superficial problems because the hard problems are too difficult to be funded. In essence, they normalize the deviance so much that they can’t even imagine solving the real problem causing their misery. You must work with the team to identify the root cause that will actually bring the metrics back to a healthy state.</p><p>Since the team has normalized deviance, they’re likely to push back against your plan as overly broad and unnecessary. Keep pressing on the issue by asking tough (but constructive) questions focused on whether alternatives will actually address the metrics. Compare the efficacy of their approach to your approach and <a href="https://epettis.medium.com/how-to-lead-large-programs-like-eddie-a203ebba4e0a">selectively decide to defend or modify your plan</a>. Repeat this process until you have a concrete, actionable plan to bring the metrics to a healthy state.</p><p>This part is where the engineering comes in. It can’t be made formulaic because <a href="https://en.wikipedia.org/wiki/Anna_Karenina_principle">happy teams are all alike, but unhappy teams are all unhappy in their own way</a>. I’ve provided a case study below to illustrate how I did this for capacity forecasting.</p><h3>Step 3: Create accountability</h3><p>The metrics in Step 1 are great because leadership simply can’t argue against them without looking clownish. We must meet SLOs because we will be lying to our users if we don’t. We should respond to all alerts because we could be ignoring a user-facing outage if we don’t. We can’t publish inaccurate data because the business won’t work. If releases aren’t successful, we don’t have a service.</p><p>As the senior IC, you need to have a private conversation with leadership about the problem and everyone’s role in addressing it. Your role is to ensure that the team has a credible technical plan to address the problem. The team’s role is to execute that plan. Leadership’s role is to ensure the technical plan is prioritized and the team’s manager held accountable to the results. This conversation needs to happen before meeting with the team because a smooth initial meeting will demonstrate your ability to deliver leadership support, which enables the team to trust you.</p><p>Arrange a meeting between the team and leadership to review the metrics. The team should explain that they cannot meet the targets unless something changes. Since you’ve prepped them in advance, leadership should state that we must meet these metrics and ask how they can help. The team should present the plan that you have reviewed for addressing the plan and ask for it to be prioritized over other commitments. If you’ve prepared them in advance, leadership should agree, freeing up time to execute the plan.</p><p>In the end, the meeting itself is mostly performative. It brings formality to the plan, but you did 95% of the work before anybody stepped foot into the room. The team is taking accountability for fixing the metrics. Leadership is taking accountability to back the team’s plan. Doing this in a formal manner creates a <a href="https://en.wikipedia.org/wiki/Commitment_device">commitment device</a> for all parties that change the default option in future situations.</p><p>After this first meeting, maintain accountability through periodic metrics reviews. I prefer bi-weekly reviews in the beginning to keep the momentum going. Teams hate these reviews, but they’re necessary to reinforce accountability for both the team and leadership. Once the team has demonstrated an ability to make progress without being prodded, the review can switch to email updates with monthly or quarterly reviews.</p><h3>Step 4: Set proximal objectives</h3><p>Deviant teams are often a long way from meeting these metrics’ targets. You can drive progress and accountability by creating a <a href="https://www.efrontlearning.com/blog/2016/05/proximal-vs-distal-goals-elearning.html/">proximal objective</a>. A proximal objective is an objective that can be met in the short term. These are useful to create momentum and learn critical skills that are often missing in deviant teams: treating every alert seriously, writing good postmortems for every outage or rollback to understand how to prevent them, and so on.</p><p>In the beginning, you will probably choose a proximal objective arbitrarily because the team doesn’t have enough information until doing some initial steps. In the absence of evidence, I like to choose targets at 50%, 75%, 95%, and 99%. The numbers are round enough that people will nod their heads and agree, and they require progressively more rigor to achieve. Using alert response latency as an example:</p><ul><li>We can usually respond to 50% of alerts within SLO by deleting low precision alerts. This requires understanding why each alert exists and deleting the ones that are not specific enough to be useful. These alerts are often replaced by alerts measuring SLO burn directly.</li><li>We can usually reach 75% of alerts by building more oncall rigor. This means clear rotations, acknowledgement and escalation procedures, and simple playbooks.</li><li>95% often requires eliminating the problems causing the alerts. The team has likely been working on this all along, but now it’s gating progress because some alerts are simply hard to debug and address. The 5% out of SLO likely result in blameless postmortems to understand how to prevent recurrence.</li><li>99% requires an operational rigor where action is taken on every alert, every alert has a playbook, and the team has close relationships with adjacent services’ oncalls to address complex interactions between services quickly.</li></ul><p>As you progress through the proximal targets, the user experience and the team’s morale will visibly improve. This will build credibility with both the team and leadership, giving the team the leadership support necessary to proceed to the next proximal objective.</p><h3>Step 5: Do the little things right every day</h3><p>Now for the hardest part. The team must do the little things right every day to de-normalize deviance and create a culture that normalizes best practices. Failure to reset the culture will result in the team falling back into their old patterns and losing their progress.</p><p><strong>The single most important thing to do is communicate the consequences of leadership’s choices.</strong> There’s always more work to do and more customers to please. Someday, leadership will tell the team to change priorities. The team must politely but firmly state, “If we do X, then we will not be able to do Y, meaning we will miss goal Z. Do you still want to do X?” Since technical excellence is the easiest thing to deprioritize when deviance is normalized, a firm reminder that “we will not be able to complete our canary process, meaning that we will have more customer-facing outages” tends to turn conversation back towards customer impacts and existential risks.</p><p>Here’s a few tips for reinforcing best practices:</p><ul><li><strong>Always act like keeping the lights on is the top priority.</strong> This means responding to every alert, writing every postmortem, supervising every release, and inspecting every failure. The team is daring leadership to maintain their commitment from that formal meeting you had in Step 2. If leadership asks the team to change behavior, you remind leadership of the consequences of their choices.</li><li><strong>Be consistent.</strong> Inconsistency creates a <a href="https://www.verywellmind.com/what-is-a-variable-ratio-schedule-2796012">variable reward schedule</a> for external partners where they will keep asking until the team changes its mind. They may start at the manager or ping the oncall to work around the manager. The entire team must redirect new commitments back to the manager or TL. If the team is consistent long enough, partners will learn to stop asking.</li><li><strong>There are two answers to an external request: yes with consequences or no.</strong> Deviant teams tend to self-sabotage because they have <a href="https://en.wikipedia.org/wiki/Learned_helplessness">learned helplessness</a>. It emotionally easier to say yes and have them go away than firmly say no, fight it, and be told to do it anyway. This leads to oversubscribed teams and shuffling priorities whenever a stakeholder shows up insisting on the deliverable the team promised. If the request comes from inside the team’s reporting chain, the team must tell leadership the consequences of their choices. If the request comes from outside the reporting chain, they should escalate the priority and inform leadership of the consequences of their choices.</li><li><strong>Write a blameless postmortem for every service outage, no matter how small.</strong> Some people may consider this overkill. Writing a postmortem forces people to think deeply about why outages are happening in the first place. The team should add postmortem action items that detect and prevent recurrence to the plan and communicate these new tasks to leadership.</li></ul><p>Over time, the team will come to understand that the organization values doing things right and start believing in it themselves. Then, all those decisions you don’t have time to observe start getting made the right way.</p><h3>Failure modes</h3><p>This isn’t necessarily a full-proof strategy. I have seen it fail a few different ways:</p><ul><li><strong>Leadership changes their mind.</strong> If leadership loses their nerve, you should remind them about the long-term business consequences. A minor staffing correction when the plan is well underway is one thing, but if they systemically starve the plan, you might want to start looking for a new team because they’re knowingly incurring an existential business risk.</li><li><strong>The problem is bigger than the initial plan.</strong> Lots of teams will lose faith because they believe leadership will never support a change to the plan that dramatically increases the scope. Proximal objectives can mitigate this failure mode. If the jump from 75% to 95% requires a full refactor or something, the team can insert a new proximal objective and communicate why to leadership at the next review. The problem’s importance hasn’t changed, only the difficulty to fix it. You should support the team by explaining why they need that huge refactor.</li><li><strong>Manager self-sabotages the process.</strong> Old habits die hard. Once the commitment has been made in Step 2, leadership must measure the manager’s performance based on meeting those goals and escalating newly found tasks to meet the goals to ensure the manager is doing their part. You should point out the problem to the manager but escalate if it continues. In my experience, this is the most common failure mode.</li><li><strong>Selfish individuals sabotage the process for personal gain.</strong> Somebody on the team may need to pivot to working on the plan even though their promotion project is almost done. These people are most susceptible to side-channel requests because they want to say “yes” but somebody else is telling them to denormalize deviance. Ideally, managers should assign people in a way that is minimally disruptive to the team, but if that’s not possible, leaders and managers should try to find a way to recognize the engineer’s selflessness and impact on the team. You should contribute to the promotion case, if possible. If that still doesn’t work, then it comes down to accountability — the team needs X, and the engineer is accountable for delivering X even if they don’t like it.</li></ul><h3>Case study: Capacity forecasting</h3><p>This whole process is probably best illustrated with a real-life example.</p><p>Back at Google, I parachuted into a capacity forecasting team that was widely considered to be struggling. Their manager had a plan to fix it, if only leadership would fund it.</p><p>I asked how the team measured success and was pointed towards a dashboard showing forecast accuracy with a nice horizontal line indicating their SLO. The existing metrics were within SLO but nobody was happy. Either the metric was wrong or the business was wrong. This led me to doubt the team’s proposed plan.</p><p>No matter who or how I asked, the team couldn’t provide me with a convincing accuracy metric that illustrated consumers’ problems. Forecasts are always wrong, so accuracy is really a question of understanding how much error is acceptable. This led to a deep exploration of the accuracy metric and developing operational processes to manage outlier forecasts. We would use the outlier investigations to figure out what steps were necessary to fix the problem.</p><p>The TL named this program “quantitative excellence.” We created a bi-weekly review to maintain accountability for the outlier root cause investigations. This review demonstrated several problems with the forecast and how it was used by the business. The team figured out how to qualify new models to avoid regressions. Notably, many of the team’s original ideas were irrelevant to fixing the problems.</p><p>Along the way, we needed to correct several cultural issues:</p><ul><li>I advocated for quantitative excellence among senior leadership: our senior director, distinguished and principal engineers, planners, etc. They enthusiastically supported the program when they understood that this program would bridge the gap between the team’s metrics and their experiences. I worked with our organization’s leadership to ring fence headcount to work on quantitative excellence, free from outside interference.</li><li>We couldn’t get TL bandwidth from one sub-team because the sub-team was significantly oversubscribed, leading to lots of priority shuffling. I addressed this by escalating the concern to leadership, which put pressure on the manager to correct the problem by reinforcing accountability towards the goals.</li><li>One sub-team needed to decommit from a significant project and kept putting up bureaucratic roadblocks to our progress, which we dealt with by reinforcing accountability to meet the goals. I met with leadership to emphasize how they needed to reward the work prior to the shift to eliminate these behaviors in the future.</li><li>Nobody wanted to do the investigations, so we created a rotation where people were held accountable for reviewing outliers. Reviews took longer than expected, so we communicated how the plan would change. Leadership was happy to see the investigations would be happening and accepted the changes without drama.</li><li>Sub-teams treated each other as external partners, requiring business requirements, design documents, and so on. We reinforced that everyone on the program was one team with sufficient representation from sub-teams to translate program needs to their own domains. I brought the problem to the manager’s attention to take care of it and leadership’s attention to create accountability for the manager.</li></ul><p>In the end, capacity forecasting’s problems were relatively simple from a technical perspective and rooted in normalizing bad behaviors. TLs were able to execute once we addressed the culture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f38fd844a520" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>