<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Charlie Koster on Medium]]></title>
        <description><![CDATA[Stories by Charlie Koster on Medium]]></description>
        <link>https://medium.com/@ckoster22?source=rss-86e753fbc92a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MIhMxl88uo7fCu84SRfqsg.jpeg</url>
            <title>Stories by Charlie Koster on Medium</title>
            <link>https://medium.com/@ckoster22?source=rss-86e753fbc92a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:21:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ckoster22/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tech Industry Predictions for 2026]]></title>
            <link>https://ckoster22.medium.com/tech-industry-predictions-for-2026-087ef8ba7463?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/087ef8ba7463</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[2026]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sat, 27 Dec 2025 16:22:19 GMT</pubDate>
            <atom:updated>2025-12-27T17:05:39.870Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*enR8bHmhTN9VU0NDM2uyGQ.png" /></figure><h3>2026: What’s in?</h3><p>It is rapidly becoming 10x cheaper to build the 80% most used features of the leading software product in a given market, leading to <strong>accelerated product competition.</strong></p><p>Software moats aren’t as wide as they used to be. While proprietary data will continue to be a moat <em>strengthener</em>, many of your end users (and your internal talent) are increasingly likely to get poached by competitors who can operate more leanly and ship faster.</p><p>Agentic AI will assist different roles to varying degrees. Expect the anatomy of a <strong>product engineering team to shrink,</strong> as it may no longer make sense to staff 4+ developers within a given area of focus.</p><p>Adjustments are likely to ripple to the middle management layer as well, where coordination overhead becomes harder to justify when execution velocity is increasingly automated.</p><p><strong>Self-documenting and self-correcting knowledge bases</strong> will not only become possible, but also necessary as accommodations for <a href="https://ckoster22.medium.com/the-llm-incentive-ratchet-d12f337845c1">agentic LLMs ratchet</a> in one direction. Instead of static wikis that decay the moment they’re written, knowledge systems will need to continuously reconcile what was planned versus what was actually built, shipped, and the key decisions along the way. Contrary to the historical reputation of having always up-to-date documentation, this documentation serves as context for future AI-assisted tasks, creating a positively reinforcing feedback loop of efficiency.</p><p><strong>Measuring outcomes over outputs</strong> was always a true north tactic for high performing organizations. This will become even more clear in 2026 as vanity outputs - such as story point velocity and token usage - become factors that no longer matter. How much and how frequently user-facing value is delivered will remain king.</p><blockquote>Caveat: Continue to measure select secondary metrics to provide fuzzy signal when diagnosing organizational performance, but keep user-facing value as the key outcome worth paying attention to.</blockquote><h3>2026: What’s out?</h3><p><strong>Scrum is dying and story points are dead. </strong>While Scrum won’t completely disappear within a year - and if the death of Waterfall tells us anything there will be a long tail to this journey - there will be plenty of orgs still employing roles for enforcing that process. Those orgs will soon be viewed the same as Waterfall and Project Managers were perceived a decade ago: behind the times and slow to adapt.</p><p>Scrum, like any process, was only ever a means to an end. That end is a lighter weight, more Kanban-esque methodology for ruthlessly prioritizing and organizing work. AI-assisted teams who can competently use the technology will be able to move and iterate faster than what rigid scrum adherence allows.</p><blockquote>Tip: Avoid hiring the equivalent of “Kanban Masters”. Process is a means to an end, and a full-time process enforcer only increases the friction when process needs to drastically evolve alongside industry capabilities and competitive pressure.</blockquote><p>Notably, the confounding debate of<em> “What is a 3-point user story?</em>” is finally over and irrelevant now that agentic LLMs can complete the implementation of several user stories in the time it takes an entire team to run through a one-hour estimation meeting.</p><p>One possible domino effect of escaping Scrum’s ideological capture is <strong>fixed length sprints will begin to fall out of favor</strong> and (hopefully) be superseded by <a href="https://ckoster22.medium.com/when-sprints-undermine-agility-968180247f4c">narrative planning</a>. What’s the next story you can tell your customers that fits within a 1–3 week time box?</p><p>Narrative planning reframes work around outcomes and coherence rather than arbitrary capacity constraints. It encourages teams to think in terms of customer impact, sequencing, and momentum - what meaningfully advances the product story - rather than what neatly fits into a sprint board. Elevating the customer value narrative to a first-class concern also creates clearer alignment between product, engineering, and leadership.</p><p><strong>Tech stack specialization </strong>will be less sought out than in prior years, which will become apparent in both job postings and technical interviews. The agentic LLM - with autonomous access to mountains of knowledge and context - <em>is your specialis</em>t. What companies need now more than ever are the individuals who can think like an Engineer, an Architect, and an end user - and know how to effectively use and orchestrate these semi-autonomous specialists.</p><blockquote>Entry level candidates will feel a squeeze here. It’ll both be more difficult to land a new job and, paradoxically, more difficult to learn and grow. The temptation to lean on the LLM can rob early-career contributors of hard-earned, coachable moments - the kind that come from struggling through a problem yourself or with the guidance of a human mentor.</blockquote><p>Organizations and entrepreneurs alike will come to understand the dangers of <strong>reckless vibe coding</strong>.</p><p><a href="https://www.linkedin.com/posts/charlie-koster_vibe-coding-rejecting-most-of-your-agent-activity-7378227899587485696-j9OW?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAQilDQBZx500q2V0t3Q__1QlT9Lbj0cLtA">Vibe coding</a> isn’t inherently something worth avoiding. It’s great for whipping together ad hoc scripts for mundane tasks, or rapidly mocking up mostly working apps as conversational aids and demonstrations. However, at the end of the day you still need someone (and perhaps one day some<em>thing</em>) capable of intelligently reviewing, stress-testing, and iterating on the underlying implementation. Without that discipline, today’s vibes quickly become tomorrow’s technical debt.</p><p>How do these predictions stack up with your own expectations in 2026? Drop a comment and let me know!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=087ef8ba7463" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Divergent Outline Clarification in LLMs]]></title>
            <link>https://ckoster22.medium.com/divergent-outline-clarification-in-llms-6221dd6902fa?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6221dd6902fa</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[system-2]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[system-2-thinking]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 22:54:31 GMT</pubDate>
            <atom:updated>2025-07-29T12:40:52.871Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Technique for Clarifying Questions in Deep Research Applications</h4><p>When building my own deep research application, I found myself frustrated that, even though an agentic LLM asked clarifying questions early in the process, it would still produce reports that didn’t quite match what I needed.</p><p>I often reviewed the final report and thought, “I wish it had asked me whether I wanted to take the report in <em>that</em> direction.” This frustration led to an AHA moment: what if I structured the clarification stage to proactively identify potential differences in research directions?</p><p>To provide context, <strong>deep research</strong> refers to the iterative process of an LLM-driven workflow that refines research questions, gathers detailed information, and synthesizes findings into focused summaries or reports. Typically, this involves multiple specialized agents:</p><ul><li><strong>Clarifier agent</strong>: Asks targeted questions to refine the user’s intent.</li><li><strong>Research agent</strong>: Performs searches or queries to gather relevant knowledge.</li><li><strong>Summarizer agent</strong>: Turns gathered knowledge into structured reports tailored to the user’s needs.</li></ul><h3>Divergent Outline Clarification in the Clarifier Agent</h3><p>To address this challenge, I designed a <strong>Clarifier Sub-Agent</strong> specifically tasked with generating up to four clarifying questions for any given research topic. It performs this step by applying a <a href="https://www.linkedin.com/posts/charlie-koster_deep-dive-into-deep-research-with-qwen3-30b-a3b-activity-7346651724398436354-xMqa?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAQilDQBZx500q2V0t3Q__1QlT9Lbj0cLtA">System 2 thinking</a> technique that I refer to as <strong>divergent outline clarification</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QwamEyqiapjb2Wedh9RIyA.png" /><figcaption>Divergent Outline Clarification</figcaption></figure><p>The process begins by prompting the agent to pretend it has already gathered knowledge and its job is to now come up with a few hypothetical outlines for a summarization report.</p><blockquote>You are an AI assistant. Given a research topic (and optionally recent search context), your goal is to generate 1 to 4 clarifying questions to refine the research direction.<br>…<br>To do this, follow these internal steps:<br>1. <strong>Assume you have all necessary knowledge</strong> about the provided research topic. If recent search context is provided, incorporate this current information into your understanding.<br>2. Internally, <strong>brainstorm four distinct summarization outlines</strong> (as an array of strings) for a report on this topic. Think about different angles, scopes, or focuses. If search context is available, consider how recent developments might affect these outlines.</blockquote><p>This part of the prompt encourages the LLM to come up with a few outlines in the format of [&quot;Introduction&quot;, &quot;Section 1&quot;, &quot;Section N&quot;, &quot;References&quot;] . Importantly, the LLM will produce a variety of different outlines. For example, it may have academic, pragmatic, beginner’s guide, and executive summary flavored outlines.</p><p>Armed with these possible report summarization outlines, the LLM is then prompted to analyze the differences between the outlines.</p><blockquote>3. Next, analyze the key differences, ambiguities, or decision points between these outlines. Pay special attention to how recent information (if provided) creates new possibilities or clarifications needed.</blockquote><p>Lastly, the LLM uses this analysis to formulate precise, closed-ended clarifying questions:</p><blockquote>4. Finally, based on this internal analysis, formulate 1 to 4 CLOSED-ENDED clarifying questions <strong>that, if answered by the user, would help you choose the most appropriate outline or focus for the research</strong>.</blockquote><p>By generating clarifying questions in this structured way, the process injects intentional consideration into the early stages of research, significantly increasing the likelihood that the final output aligns closely with the user’s true intent.</p><h3>Example: “<em>Give me space discoveries within the past 30 days”</em></h3><p>What follows is example output from llama3.3 70b</p><blockquote>&lt;think&gt;<br>Given the topic of space discoveries within the past 30 days, along with the provided recent search context which includes various significant advancements in space exploration such as observations from the James Webb Space Telescope, new moons discovered around Jupiter and Saturn, findings on asteroid samples, water ice deposits on Mars, potentially habitable exoplanets, and large-scale cosmological structures, we can begin to refine our research direction.</blockquote><blockquote><strong>First, let’s outline potential summarization paths for a report on this topic:</strong></blockquote><blockquote>Focused on Exoplanetary Discoveries: This could delve into the specifics of the infant exoplanets observed by the James Webb Space Telescope, the potentially habitable super-Earth TOI-715 b, and what these discoveries mean for our understanding of planetary formation and the search for life. Outline: [&quot;Introduction to Exoplanets&quot;, &quot;Recent Discoveries with JWST&quot;, &quot;Implications for Life Search&quot;].</blockquote><blockquote>Solar System Exploration Advances: This path would explore the new moons of Jupiter and Saturn, asteroid sample analysis from OSIRIS-REx, water ice deposits on Mars, and what these findings reveal about our solar system’s dynamic nature and potential resources for future missions. Outline: [&quot;Introduction to Solar System&quot;, &quot;New Moons and Asteroid Discoveries&quot;, &quot;Mars Exploration Updates&quot;].</blockquote><blockquote>Cosmological Discoveries and Their Implications: Focusing on the larger scale, this could discuss the distant black hole merger detected by JWST, the massive ring-shaped structure known as the “Big Ring,” and how these discoveries challenge or affirm current cosmological models. Outline: [&quot;Cosmology Basics&quot;, &quot;Recent Large-Scale Discoveries&quot;, &quot;Implications for Cosmological Models&quot;].</blockquote><blockquote>Technological and Missional Advances: This outline would emphasize the technological achievements enabling these discoveries, such as the capabilities of the James Webb Space Telescope, advancements in Mars exploration, and how these missions are pushing the boundaries of space science. Outline: [&quot;Introduction to Space Technology&quot;, &quot;JWST Capabilities and Findings&quot;, &quot;Future Mission Possibilities&quot;].</blockquote><blockquote><strong>Analyzing the differences between these outlines highlights several decision points:</strong></blockquote><blockquote>The focus: exoplanets, solar system, cosmology, or technology/missional advances.</blockquote><blockquote>The scope: whether to include historical context, recent developments, or speculative future possibilities.</blockquote><blockquote>The depth of analysis: how much detail to go into on each discovery and its implications.</blockquote><blockquote><strong>Based on this analysis, clarifying questions that would help choose the most appropriate outline or focus for the research could be:</strong></blockquote><blockquote>Do you want the report to focus on discoveries related to exoplanets, solar system exploration, cosmological structures, or technological advancements in space science?</blockquote><blockquote>Should the analysis prioritize recent breakthroughs within the past 30 days or include a broader historical and future-oriented perspective?</blockquote><blockquote>Are you interested in a comparative study of different space missions and their findings, such as JWST versus other telescopes or Mars exploration versus asteroid sampling?</blockquote><blockquote>&lt;/think&gt;</blockquote><pre>{<br>  &quot;questions&quot;: [<br>    &quot;Do you want the report to focus on discoveries related to exoplanets, solar system exploration, cosmological structures, or technological advancements in space science?&quot;,<br>    &quot;Should the analysis prioritize recent breakthroughs within the past 30 days or include a broader historical and future-oriented perspective?&quot;,<br>    &quot;Are you interested in a comparative study of different space missions and their findings, such as JWST versus other telescopes or Mars exploration versus asteroid sampling?&quot;<br>  ]<br>}</pre><p>Notice how this differs significantly from simply requesting clarifying questions based on a provided topic. The questions aren’t generated arbitrarily. Instead, the LLM employs deliberate thought and intentional reasoning to pinpoint exactly what needs clarification.</p><p>With continued tuning, additional examples, and the adoption of increasingly intelligent models, this technique will further enhance the relevance of clarifying questions and ultimately produce deeper, more tailored research outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6221dd6902fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embracing System 2 Thinking in LLMs]]></title>
            <link>https://ckoster22.medium.com/embracing-system-2-thinking-in-llms-9cd9e4fdf7e1?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cd9e4fdf7e1</guid>
            <category><![CDATA[system-2-thinking]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sat, 25 Jan 2025 20:21:30 GMT</pubDate>
            <atom:updated>2025-01-25T20:21:30.123Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Better Prompts Aren’t Enough</h4><p>Imagine someone tosses a ball to you unexpectedly. Your System 1 response kicks in: you instinctively react. With this sudden visual input, electrical signals are sent through your brain and your hands automatically move to attempt to keep the ball from hitting you, if not catch it.</p><p>Now contrast that with someone saying, “Hey, I’m about to toss you this ball.” This time, your System 2 thinking takes over — you plan, anticipate, and position yourself for the catch. You visualize the arc of the throw and deliberately place your hands to maximize your chances of success. You are<em> thinking</em> in the traditional sense of the word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7ny5XvVJu2LbvvCi-fGTA.png" /></figure><p>When you send a prompt to an LLM you are invoking an analogous System 1 response. The LLM is taking your prompt and in a single shot it is giving you its spontaneous reaction.</p><p>Sometimes these responses are great, the LLM is able to catch the ball so-to-speak, however we’ve all seen instances where it goes less than great. Hallucinations, inaccuracies, logical inconsistencies. We’re left wondering, “What was this LLM <em>thinking</em>?”</p><p>The truth is, the LLM is thinking as little as you are a thinking when a random ball unexpectedly flies in your direction. Which is to say, there is no “thought”, only automatic reaction. A quick signal through a network of neurons, real or artificial, trained to react from past experiences.</p><p>But you can make LLMs think!</p><h3>System 1 Tunnel Vision</h3><p>An early anti-pattern I’m noticing with the usage of LLMs is unwitting over-focus on System 1 tactics when the problem calls for a System 2 approach.</p><p>We seem to be wired with a bias that a single invocation of an LLM should be sufficient to get us the results we want and we employ a few strategies to try and force that reality.</p><ul><li><strong>Prompt engineering</strong> has quickly become a necessary skill to improve LLM output, while prompt <em>over-engineering</em> quickly leads to frustrating results. LLMs can only handle so much nuance in their instructions.</li><li><strong>RAG (Retrieval-Augmented Generation)</strong> — a <a href="https://www.linkedin.com/posts/charlie-koster_llm-ai-activity-7243816807852056576-NU4T/?utm_source=share&amp;utm_medium=member_desktop">fancy term</a> that simply means “insert relevant contextual data” — is a must-have strategy when a bit of extra data compounds the efficacy of the output. Keep in mind that mutli-step, heavily nuanced problems may not be completely solved by RAG.</li><li><strong>Fine-tuning</strong> models can be a good last resort when the above two strategies are insufficient. Again, because we’re in System 1 territory, fine-tuning will be inferior for complex problem solving or reasoning.</li></ul><p>To be fair, all of these are valid strategies for improving results for<em> System 1 problems</em>. It is, however, important that you carefully determine whether the problem you’re attempting to solve with an LLM is better solved with a thoughtless, one-shot prompt. Or does it require concepts like reasoning, nuance, or mutli-step logic? If the latter, then System 2 thinking is likely to yield better results.</p><h3>System 2 Thinking</h3><p>System 2 thinking involves any combination of the following:</p><ul><li>Breaking the problem into distinct sequential or parallel steps</li><li>Iteration — piping output from an LLM as input to another LLM</li><li>Shared context across steps</li></ul><p>Notably, System 2 thinking is characterized by sending more than just a single prompt into an LLM.</p><h4>Example: Terms &amp; Privacy Analyzer</h4><p>An international company recently released a new model, <a href="https://www.deepseek.com/">Deepseek</a>, that is said to rival OpenAI’s state of the art models, however there are criticisms of Deepseek peppered throughout AI forums this week. Let’s dig deeper and analyze their <a href="https://chat.deepseek.com/downloads/DeepSeek%20Terms%20of%20Use.html">Terms</a> and <a href="https://chat.deepseek.com/downloads/DeepSeek%20Privacy%20Policy.html">Privacy policy</a> to understand why.</p><p>Rather than having an extended conversation with an LLM like ChatGPT — what would effectively be System 2 thinking with a human in the middle — let’s build a simple LLM agent that allows us to reuse this type of System 2 approach in the future.</p><p>I want the following <em>thought process</em> to occur:</p><ul><li>Extract the primary sections of a text document</li><li>Traverse through each section, extracting notable insights, and double checking nothing important was missed</li><li>Create a short summary and list the insights</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzSmf-NLOpm0PzdaphpsuQ.png" /><figcaption>Simulating System 2 thinking by piping output from one LLM to another multiple times</figcaption></figure><p><strong>Step 1: Extract Sections</strong></p><pre>Given the following text:<br>{text}<br><br>1) Identify and split it into primary sections.<br>2) Return sections in valid JSON format as an array of objects. Each object should have:<br>   - &quot;section_title&quot;: a short optional title or a placeholder if none is clear<br>   - &quot;section_text&quot;: the text belonging to that section<br><br>Example JSON format:<br>[<br>  {{<br>    &quot;section_title&quot;: &quot;Introduction&quot;,<br>    &quot;section_text&quot;: &quot;Some introduction text here...&quot;<br>  }},<br>  {{<br>    &quot;section_title&quot;: &quot;Details&quot;,<br>    &quot;section_text&quot;: &quot;Some details text here...&quot;<br>  }}<br>]<br><br>If the text doesn&#39;t naturally split, just return a single array element with the entire text.<br>Only output valid JSON.</pre><p><strong>Step 2a: Extract insights</strong></p><pre>The following section is:<br>{section}<br><br>Extract any unusual, concerning, or unethical points from this section as bullet points. <br>If there are none, output &quot;None&quot;:</pre><p><strong>Step 2b: Double check nothing was missed</strong></p><pre>These are the extracted insights so far from all sections:<br>{combined_insights_text}<br><br>Reflect on whether any important or concerning points might have been missed. <br>If there are additional insights, provide them as bullet points. <br>If there are no additional insights, just output &quot;No additional insights&quot;.</pre><p><strong>Step 3: Summarize and list</strong></p><pre>Summarize the following text into one or two paragraphs:<br>{ev.original_text}</pre><pre>These are the extracted insights from the text:<br>{ev.insights}<br><br>Return them as bullet points, one per line, filtering out any empty or trivial bullet points:</pre><p><a href="https://gist.github.com/ckoster22/d4acf5c6a80fe26cbe415f3cc75aa5e2">Privacy output</a><a href="https://gist.github.com/ckoster22/b5d769e29a2f7c468f0629c7782fbdeb"><br>Terms output</a><br><a href="https://gist.github.com/ckoster22/244a8a662373c132f6b70c959ca8d925"><em>Full source code</em></a><em> employing System 2 thinking in LlamaIndex</em></p><p>By splitting what could be one enormous prompt into smaller, well-defined prompts we are able to effectively mimic Systems 2 thinking. Several iterative thoughts were combined to produce a much more useful response.</p><p>Furthermore, this approach can enable you to use cheaper models to obtain similar results. The above example uses gpt-4o-mini which wouldn’t be able to provide as deep or as accurate insights were this analysis performed in a single well-engineered prompt.</p><p>Using more expensive <em>reasoning</em> models would likely yield similarly useful results. However, besides Deepseek, those reasoning models have historically been very expensive. Additionally, when those models are pausing to <em>think</em>, what do you suppose they are doing behind the scenes? They are employing chain-of-thought and System 2 thinking!</p><h3>Conclusion</h3><p>The shift from single-pass prompt engineering to multi-step, iterative workflows reflects a broader transition from System 1 to System 2 thinking in how we work with LLMs. While crafting the perfect prompt can be useful for well-defined tasks, complex problems often demand a more thoughtful, structured approach.</p><p>System 2 thinking is not just a tool for solving immediate challenges — it’s a stepping stone toward more advanced, autonomous workflows powered by LLM agents. By breaking tasks into smaller, manageable steps and leveraging iterative refinement, System 2 enables intelligent, agent-driven systems capable of handling multi-faceted problems with minimal human intervention.</p><p>As we look ahead to 2025, the role of LLM agents will become increasingly important. These agents, designed to think in deliberate, multi-step ways, are poised to redefine and disrupt how we approach problem-solving and decision-making. Paying close attention to their evolution will be critical for anyone aiming to stay ahead in the AI landscape.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cd9e4fdf7e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring o1-preview Use Cases]]></title>
            <link>https://ckoster22.medium.com/exploring-o1-preview-use-cases-c8512e6f6d32?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8512e6f6d32</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[lean-startup]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sat, 02 Nov 2024 15:36:20 GMT</pubDate>
            <atom:updated>2024-11-03T05:06:41.738Z</atom:updated>
            <content:encoded><![CDATA[<h4>Rapid Prototyping and Innovation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XC84PTDsDDVfYpQR_9fPkA.png" /></figure><p>In September 2024, OpenAI <a href="https://openai.com/index/introducing-openai-o1-preview/">released o1-preview</a>, a powerful LLM that I’ve leveraged extensively since its launch. Here are a few of my favorite o1-preview use cases in its first 7 weeks.</p><h4>SQL Assistant</h4><pre>Write a SQL query that provides me all of the chargebacks and refunds this past quarter using our online payments system, where `online_payment_id` is not null, and any invoices in the 90 days leading up to each chargeback or refund.<br><br>Group by builder, then job site. Sort by invoice amount descending.<br><br>Below are the schemas for the tables you&#39;ll need to query:<br>```schemes go here```</pre><p>Admittedly, other LLM models like GPT-4 and GPT-4o can produce good enough SQL queries. However, in my experience o1-preview performs better. Two minutes of crafting a prompt saves me 20 minutes of coming up with my own bespoke SQL query.</p><p>Although a trivial example, here I’m effectively delegating a mundane task to something competent that got it right on the first try. As someone who is constantly context switching I greatly appreciate not needing to context switch one more time into my DBA hat. Rather, I can maintain my train of thought without getting bogged down within the minutiae of crafting a non-trivial query.</p><h4>Prototype: Exploring Unfamiliar Capabilities</h4><pre>Help me understand whether this use case is an appropriate fit for a MongoDB Aggregation Pipeline. If we decide to pipe these specific events into MongoDB there may be certain events we want to track and group separately after applying sorting and filtering.<br><br>Tell me more about MongoDB&#39;s aggregation pipeline and ask me clarifying questions to determine if it&#39;s the right tool for the job.</pre><pre>Translate the aggregation pipeline you&#39;ve provided into C#. Extract the lists of filtered events into a constant. Use &lt;specific nuget package&gt;.</pre><p>One of my responsibilities as an <a href="https://www.linkedin.com/in/charlie-koster/">Architect</a> is to understand which tools are best suited for our needs. While I’m familiar with NoSQL data stores like MongoDB, I’m less familiar with its specific <a href="https://www.mongodb.com/docs/manual/core/aggregation-pipeline/">aggregation pipeline</a> capability.</p><p>o1-preview was effective in understanding a capability that was new to me, throwing together some prototype commands, iterating, refining, and ultimately translating it into something I can hand off to a developer. Again, o1-preview is helping me contribute without context switching into the minutiae.</p><h4>phenofuse.io: From Idea to Reality in 1 Week</h4><p>My daughters and I like to play a fun game with DALL-E 3 via ChatGPT. One kid asks it to create a hybrid animal from two other animals and 20 seconds later an image is generated and the other kid attempts to guess the two original animals. A trivial game for sure, but it’s fun and humorous seeing what ChatGPT provides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gIUjGwfm8XfLNwUi" /><figcaption>“Create a hybrid between a turtle and a polar bear”</figcaption></figure><p>This game is small, but also <a href="https://medium.com/swlh/a-sufficiently-complex-cloud-application-842a13529648">sufficiently complex</a>, and I decided to challenge myself and build a website entirely coded by OpenAI’s o1-preview model from scratch.</p><ul><li><strong>Milestone 1:</strong> Obtain an OpenAI API key and do a basic proof of concept. o1-preview provided the python script to invoke the API. <strong>Key Learnings:</strong> DALL-E 2 is not great at producing hybrid animals. Further, ChatGPT is in fact crafting a detailed prompt behind the scenes that is then sent to DALL-E 3. I need an additional LLM invocation to do the same.</li><li><strong>Milestone 2:</strong> Request o1-preview to modify the python script to initially invoke gpt-4o-mini to craft a DALL-E 3 prompt, then generate the image. <strong>Key Learnings:</strong> Due to o1-preview’s knowledge cutoff it didn’t know about the existence of 4o-mini. Once corrected I was able to iterate on a 4o-mini prompt that produced useful DALL-E 3 prompts.</li></ul><pre>gpt-4o-mini output:<br>A hybrid animal that combines the head of a giraffe and the body of an elephant. The creature has a long, slender neck like a giraffe, topped with a large, rounded head. Its face features the giraffe&#39;s big, expressive eyes and long eyelashes, but the ears are large and floppy like an elephant&#39;s. The skin is mostly gray and rough, resembling an elephant&#39;s, but with patches of soft, spotted fur across the neck and upper body, blending the giraffe&#39;s light brown and cream colors.<br>The torso is robust and powerful, shaped like an elephant&#39;s, with a broad, barrel-like chest and a strong back. The front legs are thick and sturdy, ending in wide, padded feet like an elephant’s, perfect for support. The hind legs are thinner and more agile, resembling a giraffe&#39;s, giving the creature a unique balance. It stands tall, towering over the ground with a height that merges both animals’ features.<br>Its tail is long and tufted, similar to a giraffe&#39;s, but with a tuft of coarse hair, blending into the textured skin of the body. The creature is in mid-stride, running gracefully across a simple, grassy plain. Its neck stretches forward, showcasing the giraffe&#39;s elegance, while its massive legs propel it with power. The lighting is soft and natural, highlighting the textures and colors of the animal&#39;s skin and fur.</pre><p><a href="https://chatgpt.com/share/67264367-8f18-8010-893b-6dd916560729">Try this in ChatGPT!</a></p><ul><li><strong>Milestone 3:</strong> Request o1-preview to throw together a quick Elm application with publicly exposed AWS Lambda Functions to generate the prompt and image. <strong>Key Learnings:</strong> ChatGPT’s image generation latency is a fraction of going straight to DALL-E 3 (upwards of 60 seconds!). I needed a way of working around API Gateway’s 30 second timeout and a way to keep the user engaged while waiting, such as with a rotating carousel of previously generated images.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlMnfR7ocdMlzbAO-WIH8w.png" /></figure><ul><li><strong>Milestone 4: </strong>Use o1-preview to assist with setting up SQS, DynamoDB, Route53, CloudFront, ACM, and a couple more Lambda Functions to asynchronously generate a prompt, generate an image, and poll the image status. Additionally, ask o1-preview to create a carousel in Elm from scratch. <strong>Key Learnings:</strong> This went great! I found that o1-preview typically gets the code right within the first or second try. Although I was less impressed with o1-mini.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glrOmQalJmt4FU0xjwx07g.png" /></figure><ul><li><strong>Milestone 5</strong>: <em>“Help me understand best practices for securing my website, </em><a href="https://phenofuse.io/"><em>https://phenofuse.io</em></a><em>. Send me cURL commands where I can paste you the resulting output and you determine whether I&#39;m following security best practices or if I’m vulnerable to common attacks. Lastly, given my tech stack provide me detailed instructions on how to plug any security holes.” </em><strong>Key Learnings: </strong>This was an incredibly effective way to improve the security of the website.</li></ul><p>My experience with building <a href="http://phenofuse.io/">phenofuse.io</a> was a real AHA moment for me. I spent a few hours every night for only <em>5 nights</em> rapidly going back and forth with o1-preview to take a website from idea to MVP. It was as if I had a Senior Developer, a Cloud Engineer, and an InfoSec Engineer ready to receive delegated tasks at a moments notice.</p><p>I’m impressed with o1-preview and I look forward to the <a href="https://openai.com/index/learning-to-reason-with-llms/">upcoming improvements</a> once o1 is released.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8512e6f6d32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The LLM Incentive Ratchet]]></title>
            <link>https://ckoster22.medium.com/the-llm-incentive-ratchet-d12f337845c1?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d12f337845c1</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[product-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[chat-gpt-ai]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Thu, 15 Aug 2024 03:14:58 GMT</pubDate>
            <atom:updated>2024-10-25T17:14:49.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5UleFI0bz2_9Enn6h89lQ.png" /></figure><h4>Why Your Senior Engineer Pipeline May Dry Up</h4><p>Predicting the future impact of technology is fraught with humorously inaccurate examples. The telephone, automobiles, televisions, computers — all were once predicted to fail in overtaking their predecessors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yv9L6vD5RmarCie7.png" /></figure><p>Further, there is no shortage of predictions that a new technology will take our jobs.</p><ul><li>1960s: <em>Computers will eliminate the need for office workers</em></li><li>1970s: <em>ATM machines will replace bank tellers</em></li><li>1990s: <em>The internet will lead to mass unemployment in the retail sector</em></li></ul><p>So what’s different this time around? Why would <em>this</em> technological advancement, specifically recent advancements in Large Language Models (LLMs), put some Product Engineering positions on the endangered species list?</p><p>Let’s take a closer look at how various incentives produce a unidirectional ratcheting effect toward more LLM adoption over human labor.</p><h3>Throttling the Junior-to-Senior Pipeline</h3><p>Are Senior level positions immediately in danger? I don’t believe so.</p><p>In fact, I expect the opposite in the short term. I expect the combination of highly competent engineers able to effectively use bleeding edge AI capabilities to be in high demand by employers over the next few years. I suspect we are entering a period where human productivity noticeably increases and peaks briefly before declining.</p><blockquote>Highly competent professionals able to leverage AI will be in high demand over the next few years.</blockquote><p>On the other hand, I expect junior and entry level positions to gradually shift from human workers to AI over time. This can come in two forms which are not mutually exclusive.</p><h4>#1 The Evaporation of an Engineering Mindset</h4><p>The ability to solve hard problems is a key skill that all software professionals must learn and dial in as it is crucial for advancing in their career and taking on harder, more complex problems.</p><p>Historically, search engines like Google have been used as a crutch to help individuals solve problems, but a search engine can only take you so far in doing your job. You, the person leveraging a search engine, still need to bring an engineering mindset in order to achieve an outcome.</p><p>LLMs are a step up from search engines as a crutch. They make it increasingly easy to paste in an issue, clarify, provide context, and eventually get the LLM to spit out a close enough solution if not a total solution.</p><p>The worrying reality of this situation is there isn’t a strong incentive for junior level individuals to learn how to think like an engineer. Quite the opposite. If one can solve a problem more quickly by outsourcing it to a <a href="https://developers.google.com/machine-learning/resources/intro-llms">token prediction black box</a>, then why not? The crutch is there and near at hand. It’ll be used and used often. The prior path to seniority now has an impassible obstacle: the diminishing need to learn to think like an engineer.</p><blockquote>With LLMs, junior software professionals will be incentivized to not think like an engineer.</blockquote><h4>#2 The LLM Utility Ratchet</h4><p>Large Language Models will integrate within technology organizations as micro-assistants at first. <em>“Write this unit test”. “Refactor this function”. “Produce this mockup.”</em> LLMs will shave tens of minutes off of mundane tasks and generally be a net positive for disciplined Product Engineering teams.</p><p>As LLM improvements evolve they will be more and more capable of taking on small to medium sized tasks from start to finish in a fraction of the time of a human with increasing accuracy and increasingly within acceptable standards of quality.</p><p><strong>Once small, simple software deliverables can be accomplished in near-autonomy by an LLM this will be a key milestone where we should expect to see the adoption and accommodations for LLMs to ratchet in one direction.</strong></p><p>There will be a powerful incentive to change product engineering execution to be more tailored towards crafting work items into a structure that an LLM can quickly solve. Feedback loops tighten. Productivity increases. Teams experiment with different accommodations. The good ideas persist, while the less effective ones are discarded. LLM utilization and reliance ratchets.</p><h4>The Diminishment of Juniors Today Leads to a Shortage of Seniors Tomorrow</h4><p>There’s not much of a logical leap here. Every Senior+ Product Engineering professional was once much less experienced and through hard work and experience they got to where they are years later.</p><p>If the Junior software professional pipeline is squeezed, either through partial replacement or because they haven’t learned how to take on more challenging problems, then 3–5 years later where will the Senior software professionals come from?</p><h3>“Bullshit. You don’t know what you’re talking about.”</h3><p>To the skeptics, I’d ask you to make the following considerations.</p><h4>Image and Video Generation Is More Difficult</h4><p>Image and video creation via generative AI was thought to be many years, even decades, away as recent as 2019. A few short years later it is becoming increasingly difficult to discern generated images from reality or whether artworks were produced by professional artists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*15n6KQeM5fTShEMXFqxhFA.png" /><figcaption>An image I generated for a recent Architecture conference talk (<a href="https://www.linkedin.com/posts/charlie-koster_architectingforchangepresentationpdf-activity-7220494313019822080-EG1c?utm_source=share&amp;utm_medium=member_desktop">link</a>)</figcaption></figure><p>Likewise, we are on the <a href="https://openai.com/index/sora/">precipice of video generation</a> sweeping through several industries. Consider that <strong>generating convincing videos with object permanence is a more difficult problem</strong> than generating mockups, database schemas, or code.</p><p>Tech companies are pouring hundreds of millions of dollars into advancing LLMs. They are quickly conquering text, image, and video generation. Code generation is well on its way to catching up.</p><h4>We All Helplessly Steer Towards Incentives</h4><p>Businesses will be incentivized to be more economical with their resources in order to survive and remain competitive and profitable.</p><p>As of today LLMs cost a few dollars per 1 million tokens. <em>(For context, the tokenized version of this blog post would cost 6/10 of a penny.)</em> Human software professionals, on the other hand, have six figure total compensation packages, benefits, time off, feelings, and are orders of magnitude slower within specialized tasks.</p><p>Even if a company on moral, ethical, or cultural value grounds opts to ignore the above statement they will increasingly feel the squeeze from a competitor who is able to operate more efficiently. LLMs will allow businesses to operate more efficiently and therefore businesses will need to adapt to survive, as they are incentivized to do so.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d12f337845c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting for Change]]></title>
            <link>https://ckoster22.medium.com/architecting-for-change-b12824b35ea3?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b12824b35ea3</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[volatility-decomposition]]></category>
            <category><![CDATA[agility]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 18:46:09 GMT</pubDate>
            <atom:updated>2024-03-02T15:17:52.177Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Decompose Systems by Volatility</h4><p>Every software system has an architecture whether intentionally designed or not. While there are many architectural styles on offer, hands down the best approach I’ve employed is <em>Volatility Based Decomposition</em>, an approach inspired by <a href="https://rightingsoftware.org/">Righting Software</a>.</p><p>This post describes an interpretation of the method described in <em>Righting Software</em> focusing on decomposition by volatility as the north star.</p><h3>Volatility Based Decomposition</h3><p>The essence of volatility based decomposition is the to answer the question “<em>What, if changed, wouldn’t I want to become a rippling change across multiple systems?”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hB46_FsZELGBh5y_K75WdQ.png" /><figcaption>An attempt to depict a noticeable change in one system does not impact another</figcaption></figure><p>There are obvious examples of encapsulating change such as abstracting away an integration with a third party vendor, however the right frame of mind is important in order to avoid alternative approaches like Functional Decomposition or Domain Driven Design which may sound good in theory but ultimately do not prevent changes from rippling across systems.</p><p>Broadly speaking there are a few activities involved when decomposing systems by volatility:</p><ol><li><strong>Identify core and supporting use cases</strong>, the behavioral essence of the system.</li><li><strong>Identify volatilities</strong> and subsequently prescribe the component or strategy for encapsulating each volatility from leaking outward.</li><li><strong>Validate the architecture</strong> by producing call chains. Any system components that aren’t in any call chain are unnecessary.</li><li><strong>Make note of non-obvious details</strong>. Boxes and arrows can only communicate so much.</li></ol><h4>Identify Use Cases</h4><p>Let’s scratch the surface of a system we designed and have used in production for the past two years, a Payment Processing System.</p><p>What should a system that encapsulates payments do? Perhaps more importantly, what should it not do?</p><p>Contrary to the opening line in this section it’s often better to <em>not start with the name of the system</em>. Rather, start by throwing use cases at the wall. Verbs and nouns. Get them out there.</p><ul><li>Pay invoice</li><li>Pay Bill</li><li>Delete payment</li><li>Set up bank account</li><li>Change bank account</li></ul><p>Continue throwing out ideas for potential use cases, scrutinize them, and iterate. Consider both <strong>current and foreseeable future use cases</strong>. Avoid functional terms and instead focus on <em>business behaviors</em>. For example:</p><ul><li>❌ Delete a payment -&gt; ✔️ Void payment</li></ul><p>Avoid leaking specific business entities that the system shouldn’t need to know about. For example, if something fundamental changes about Invoices or Bills should a system that processes payments change? Nope!</p><ul><li>❌ Pay invoice/bill -&gt; ✔️ Make payment</li></ul><p>So far these are <strong>user-driven</strong> use cases. Don’t forget <strong>system-driven</strong> use cases.</p><ul><li>Update payment status</li><li>Process refund/chargeback</li></ul><p>Lastly, group these more granular use cases and categorize them into a few buckets. These buckets represent core use cases, the <em>essence of the system,</em> and likely inform what you might want to name the system.</p><p><strong>Core use case: </strong>Manage payments<strong><br>Supporting use cases: </strong>Make payment, void payment, process refund/chargeback, update payment status</p><p><strong>Core use case:</strong> Manage payment methods<br><strong>Supporting use cases:</strong> Create payment method, change PM, remove PM</p><p>For brevity I’ve omitted a few use cases although I’ve captured enough to give a sense of what this system encapsulates. It’s a <em>payment processing system</em>. It encapsulates volatilities involved with moving money from Point A to Point B and it is shielded from other volatile areas of the business that aren’t strictly related to moving money.</p><blockquote>Note: At <a href="https://buildertrend.com/">Buildertrend</a> we’ve found it helpful to distinguish between product systems and supporting systems. A <strong>product</strong> is a system that a user logs in to, interacts with a UI, and manages user permissions and preferences in addition to other use cases. A <strong>supporting system</strong> is agnostic and reusable across multiple product systems and often interacted with via an API. This payments system is the latter variety.</blockquote><h4>Identify Volatilities</h4><p>What, if changed, would we want to encapsulate and prevent from rippling to other systems? I touched on this briefly in the prior section but I recommend calling out volatilities and encapsulations as a separate exercise to make them explicit.</p><blockquote>Volatile vs Variable:<br>Not everything that changes needs to be encapsulated. A common example of variability, as opposed to volatility, is a status value. Status enums do change but these types of changes are easily handled with code. I would not recommend encapsulating the changeability of something like payment status.</blockquote><p>A few payment processing volatilities and how they’re encapsulated:</p><ul><li>Third party vendors / Resource handler in code</li><li>Use case sequencing / Manager in code</li><li>Payment method specific decisioning / Engine in code</li><li>Transactional fees / Pass the fee into the API</li><li>Credit card entry / Iframe wrapper</li></ul><p>The first three volatilities are encapsulated using specific classes and layers at the level of code. The terms resource handler, manager, and engine all come from <em>Righting Software</em> and the names aren’t terribly important. What is important is the prescription to protect changing a third party vendor from rippling to other areas of the code, from keeping the decisioning for ACH vs credit card from rippling, or changing the order of events for specific use cases.</p><p>The final two volatilities are really interesting and worth diving into deeper.</p><p><strong>Transactional fees<br></strong>If the business wanted to conditionally change transaction fees based on business rules on the invoice, or whether the transaction was a rebate disbursement, or whether the user was in the premium subscription tier should a payments system care? Should we be tempted to <em>teach</em> a payments system what an invoice is, a rebate, or what a user + monthly subscription is?</p><p>No. These are volatilities in other areas of the business.</p><p>The volatility encapsulation in this instance is simple. If the business wants to change the fee on any given transaction for any reason then great! Pass in the fee you want when invoking POST /payment. No need for a code change in the payment processing system to accommodate fee changes.</p><p><strong>Credit card entry</strong><br>Some third party payment processors who offer credit card payments allow you to embed their PCI compliantly hosted iframe in your application. However, if we simply expose PaymentVendorA’s iframe in the consuming product and then we later change to PaymentVendorB that will require a coordinated code change across multiple systems. In other words, it’s a volatility that lacks encapsulation.</p><p>If, on the other hand, we wrap PaymentVendorA’s iframe with another iframe owned by the payments system, and create a standard contract between the iframe wrapper and the parent window, then the consuming product system can embed the iframe wrapper and not require a code change if PaymentVendorA changes to PaymentVendorB. The volatility of the payment vendor’s iframe is completely encapsulated.</p><h4>Call Chains and Architectural Validation</h4><p>We finally get to what is often the starting point in other architectural approaches: the boxes and arrows.</p><p>The purpose of this next step is not to exhaustively document the system in seemingly academically sound ways. Doing so is often overkill and even counterproductive as there is a point quickly reached when more documentation makes the reader <em>less likely</em> to read it.</p><blockquote>Two cents on good documentation:<br>Good documentation is read and understood by its intended audience. I’ve found the best way to accomplish that is to be concise, be accurate enough (but not exhaustively so), and iterate. Ask a colleague to read your documentation and validate whether it was read and understood by them.</blockquote><p><strong>Call chains<br></strong>The purpose of this step is to ultimately work towards <em>validating your architecture</em>. How do you know when you’re done?</p><p>The short answer is once all current and foreseeable future use cases are represented by call chains, and each system component is used in at least one call chain, then you’re done with designing this system.</p><p>To do this we need call chains!</p><p>Let’s begin with putting relevant system components on the screen. If something is encapsulated in code I make a point to ensure it is represented.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*8DmM6t5SK5eGtfo-DxJxDA.png" /><figcaption>Component diagram for a payment processing system</figcaption></figure><p>The next step is easy enough. For each supporting use case connect the boxes with solid (sync) or dotted (async) arrows. Here are two example call chains:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*Rk74Ukv-Ywtomu6nFs03uQ.png" /><figcaption>Make Payment Call Chain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*WVBUV-4ENOfUhlYJIS2BJQ.png" /><figcaption>Update Payment Call Chain</figcaption></figure><p>After depicting each call chain for use cases current and foreseeable identify if any system components are orphaned and have no arrows in or out. Any extra managers, engines, or caches? If yes, those are superfluous.</p><p>Once all use cases are represented by the minimal set of system components <strong>the architecture is validated.</strong></p><h4>Noted Special Considerations</h4><p>What is unique about this system that is worth mentioning explicitly? I’ve found this to be an incredibly beneficial step to help convey what diagrams and bullet points cannot.</p><p>The two examples in this system were mentioned above: nuances with transactional fees and the use of an iframe wrapper.</p><p>Another system I’ve designed is a Media System, the essence of which is to encapsulate volatilities for storing, managing, scanning, and overall handling photos, videos, and other media.</p><p>You’ll likely be able to guess at many of the verb+noun pairs that make up its behavioral use cases. That system also had some interesting volatility encapsulations that I documented separately as special considerations.</p><ul><li><strong>Virus definitions</strong> change after a file has been scanned. This informed another use case for rescanning files while remaining agnostic to the use case trigger, whether scheduled or by polling/webhooks.</li><li><strong>Signed URLs</strong> are a way to directly publish or retrieve files to and from a datastore. Unfortunately some vendors differ in implementation. By carefully designing the API we found a way to encapsulate that volatility.</li><li><strong>File templates</strong> are a common product feature, however, at the level of a reusable system that manages files it’s an unnecessary concept. <em>A file template is just another file</em> and there is no need to introduce constraining opinions on how every consuming system should treat file templates. In other words, we left file template volatilities as the responsibility of the consuming systems.</li></ul><h3>Next Steps</h3><p>What comes next is where we depart volatility based decomposition and enter organization-specific practices and processes for building software. I’ll leave a few notes for what has worked well for me when taking the system from designed to implemented.</p><p>Partner with the Lead Developer and <strong>sketch out the API contracts</strong>, even going so far as to put it in code with mock responses. Building to a mocked interface is an effective way to parallelize work and incrementally build the system. It also has the convenient side effect of gathering momentum on building out the system. It builds excitement around a green field system.</p><p>Similarly, put some low resolution <strong>DB schema sketches</strong> together. What are the main entity concepts in the system and how do they relate? How will you design for multi-tenancy? Are there performance considerations to anticipate at the level of the database? Like the prior point, this is a momentum builder that makes the system begin to feel real.</p><p><strong>Maintain focus on delivering customer value</strong>. Contrary to the book’s advice here, iterative value delivery to the customer is most important above all else. When we originally built the payment processing system at Buildertrend we went to production without implementing the use case for programmatically voiding a payment. With the beta group we rolled out to it was feasible for users to call in and for Support to manually void payments in the vendor’s UI.</p><p>We of course did implement voiding a payment within a few more weeks. However, had we followed the book’s project design prescription then customer value and feedback loops would have been delayed 4–6 weeks longer than necessary.</p><p>Of note, if you do go pick up a copy of <em>Righting Software </em>I can’t help but point out the similarities between the book’s advice on project design and waterfall. If you’re familiar with the <a href="https://ckoster22.medium.com/">noises I generally make on being agile</a> you won’t be surprised at my suggestion to ignore that part of the book.</p><p>Your mileage may vary.</p><p>And with that I’ll conclude this interpretation of Volatility Based Decomposition that has really worked well for me as a Principle Software Architect.</p><p>Reach out with any questions! It’s a really interesting topic to discuss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b12824b35ea3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Sprints Undermine Agility]]></title>
            <link>https://ckoster22.medium.com/when-sprints-undermine-agility-968180247f4c?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/968180247f4c</guid>
            <category><![CDATA[customer-focused]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sun, 12 Nov 2023 18:43:37 GMT</pubDate>
            <atom:updated>2023-11-12T18:53:27.874Z</atom:updated>
            <content:encoded><![CDATA[<h4>Sprint Anti-patterns and Why You Might Not Need Sprints</h4><p>There is a worrying momentum regarding adherence to process and sprints when it comes to agility in the software industry. If one is not careful, an organization may claim to be agile on paper but in practice it is not embodying the essence of Agile.</p><p>Here are two anti-patterns relating to sprints to be on the lookout for in your organization.</p><h4><strong>#1 No cohesive narrative</strong></h4><p>This first anti-pattern is about having the wrong focus when planning work. Temporal sprint boundaries and velocity are often first class considerations during planning while customer value is secondary, thus making the idea of wrapping work in a cohesive narrative fairly challenging.</p><p>You’ll be able to detect this anti-pattern in a sprint demo. When the demo is more recognizably described as a <em>sprint audit</em> that is a bright red flag indicating the team is more focused on process than customer value.</p><blockquote><em>This sprint we completed User Story 123. </em>Proceeds to demo clicking two new buttons.</blockquote><blockquote><em>We also completed some backend work in User Story 456.</em> Proceeds to demo a cURL request to the endpoint.</blockquote><blockquote><em>Lastly, here is a list of bugs we fixed these past two weeks.</em></blockquote><p>If the customer were in the room, which I strongly recommend if you’ve never tried it, what would they take away upon hearing what the team accomplished the past 2 weeks in this example?</p><p>Could the customer describe to their boss what the software team accomplished and more importantly why that matters? Would they have any idea what was going on during the API part of the demo? Do they have enough context on the fixed bugs to speak to the value?</p><p>From a customer value perspective a sprint audit indicates the software team is getting process tunnel vision while not having strong enough focus on delivering customer value.</p><h4>#2 Flexible scope is discouraged</h4><p>This second anti-pattern is the natural drawback of the phrase “sprint commitment”.</p><p>It’s often asserted that if the team did not <em>commit</em> to the work during planning then that work can’t be pulled into the current sprint. It’s said that unplanned work needs to go into the refinement, pointing, prioritization, and planning machine before being assigned to a sprint regardless of the ROI of pulling the work in now (where the cost is potentially minimized).</p><p>I challenge the reader to identify which of the <a href="https://www.agilealliance.org/agile101/12-principles-behind-the-agile-manifesto/">12 Agile Principles</a> this kind of pushback is in service of. Can you read past the second principle before having an AHA moment?</p><p>To be unwelcoming of changing, clarified, or unforeseen requirements is to pretend that software is easier than it really is and can surely be planned for in rigid two week cycles.</p><p>Let’s be clear. Making software is hard. Really hard. I can’t name a single team or organization I’ve worked with where we’ve perfectly threaded the needle on what can be accomplished in a two week timebox more than one consecutive sprint in a row. This anti-pattern is the assertion that that anecdote is the exception, not the norm.</p><blockquote>Process is a means to an end.</blockquote><p>This is a good opportunity to clarify that process, just like a tool, is a means to an end. We do not build software in service of rigidly adhering to process. Rather, process serves us so that we may serve our customers and if process is in the way then one must push it out of the way for the sake of our customers.</p><h4>Let’s Recalibrate: Narrative Planning &gt; Sprint Planning</h4><p>The prompt below is the remedy that guides planning a body of work for a high functioning, customer focused team.</p><blockquote>What’s the next story you’re going to tell your customers?</blockquote><p>The prompt is solely focused on cohesive value delivery for the customer and it does not matter if that value delivery takes 1 week or 3.5 weeks. I’ll reiterate, rigid 1 or 2 week sprint cycles is not the point. Delivering value to the customer is, even if that value delivery doesn’t neatly fit into a multiple of 5 business days.</p><p>Note what is absent from this prompt. There is <a href="https://ckoster22.medium.com/agility-without-story-points-55bbce542906">no mention of story points</a> or velocity. No mention of static two week cycles. No capacity planning and therefore no rigid commitments. Remember, software is hard. It’s ok to plan on some flexibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-P9hsrDRvagBszJUd-3hQ.png" /><figcaption>Process vs Customer Focused</figcaption></figure><p><strong>What you’re planning, and therefore what you will demo in a matter of weeks, is a narrative.</strong> It’s a story you’re going to tell about what value the customer has now unlocked. It’s the very essence of being customer focused.</p><p>You can think of this as introducing a <a href="https://ckoster22.medium.com/metrics-that-matter-e3fa74c8716d">metric that matters</a>. If the goal of a software organization is to iteratively deliver value then make that value delivery a first class concept. Invite a customer to your demo and afterwards have them repeat back to your their takeaways. If they can articulate in a couple sentences the unlocked value you know you’re doing planning correctly. This is a measurement, albeit somewhat subjective, toward a metric that matters most: value delivery.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=968180247f4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metrics That Matter]]></title>
            <link>https://ckoster22.medium.com/metrics-that-matter-e3fa74c8716d?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3fa74c8716d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[product-development]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sat, 19 Nov 2022 21:42:45 GMT</pubDate>
            <atom:updated>2023-11-03T21:32:55.408Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous post on <a href="https://medium.com/statuscode/product-engineering-best-practices-f28a5616062c">Product Engineering Best Practices</a> I asserted <em>you are what you measure</em>. To briefly unpack that assertion — metrics inform behavior, behavior defines who you are, <em>you become what you measure</em>.</p><p>What this means is your metrics define the essence of who you are as a team. This post zooms into how to view a metric for released bugs in the context of Waterfall vs Agile.</p><p>First, a look at the first couple of months after the release of a Waterfall project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*euWkl6rg6qltUAX1rudB1g.png" /><figcaption># User Facing Bugs vs Time (Waterfall)</figcaption></figure><p>In typical Waterfall fashion there is a big bang release and all user impacting bugs that slipped through the cracks are apparent all at once. The highs and mediums are worked first and some time later diminishing returns are reached. Not ideal but then again this isn’t news to anyone.</p><p>What does this look like for projects that are frequently and iteratively releasing to Production?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*mCL9ePvn-lROK36EMsEV5Q.png" /><figcaption># User Facing Bugs vs Time (Agile)</figcaption></figure><p>In this view we notice that Agile doesn’t eliminate user impacting bugs. In fact, over a similar period of time there may be a similar number of distinct bugs as compared to the Waterfall model. But why is the Agile approach better and why might we want to measure something other than total bug count for Agile teams?</p><h3>Measure What Matters: Area Under the Curve</h3><p>Take a look at the above Waterfall image again. If we wanted to improve how users are impacted by released bugs there are two dials we have available to adjust.</p><ol><li>Improve quality before release and <strong>use released bug count to measure success</strong>. This encourages behavior that brings the overall height of the graph down and therefore reduces impact to users.</li><li>Scale up the number of dev hours dedicated towards fixing post-release bugs, either by scaling up the team or by encouraging overtime. This steepens the drop in bugs and reduces impact to users.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mPnq33_OiiI1J--NROaQcg.png" /></figure><p>The second option is an obviously unattractive option and as a result it’s understandable why one would be tempted to lean heavily into the first option with Waterfall. Having said that, let’s not forget what both options are attempting to accomplish. At the end of the day, both options are trying to reduce impact to users, or put another way, both options attempt to <em>reduce the area under the curve</em>. Why? Because the area under the curve represents the total impact to users.</p><blockquote>User Impact = # Bugs per user x Time</blockquote><p>Something that’s worth celebrating is by moving to a more agile approach with frequent, iterative releases even if the total number of bugs summed over all releases remain the same the <strong>area under the curve is greatly diminished<em> </em></strong>compared to Waterfall. The fact that fewer bugs are released with each cycle to fewer users and the fact that bugs can be immediately fixed diminishes the curve area.</p><p>An agile approach now gives us more than just two options to improve user experience through a more quality product. Some options include:</p><ul><li><strong>Adjust the release cycle.</strong> Consider how continuous delivery could chop up the graph even more and further reduce curve area.</li><li><strong>Reduce how many users see each release.</strong> Consider targeting fewer users with especially risky releases and slowly scale up usage.</li><li><strong>Fully leverage sprint retrospectives.</strong> Why the area under the curve is larger for one team and not another may be team specific. Encourage teams to dig into root causes and experiment with solutions.</li></ul><h3>You Are What You Measure</h3><p>Metrics incentivize behavior and behavior defines who you are. Does a metric for counting released bugs matter? Do you want to become a bug minimizing machine? If you’re running Waterfall projects and developer burnout is important to you then sure. Concentrating on that specific metric could make sense.</p><p>However, should <em>Agile</em> teams become bug minimizing machines?</p><p>Perhaps not.</p><p>Total bugs is not the metric that matters. In fact, getting too focused on minimizing released bugs can be detrimental to the iterative agile machinery. It may disincentivize teams from frequently releasing until they’re more confident the bug count is closer to zero, effectively moving the needle in a more Waterfall direction. Ironically, this could <em>increase the area under the curve</em>.</p><p>Let’s get to the punchline..</p><blockquote>In Agile, bugs are ok as long as the overall user impact is minimized.</blockquote><p>If your team is frequently and iteratively releasing value it’s fine for bugs to slip through because the overall impact will be small. Don’t ignore quality. Rather, recalibrate on which metrics truly give a quality signal.</p><p>When you focus on the area under the curve with an agile mindset, which is to say if you focus on reducing negative impact to users while frequently releasing value, then who do you become? You are what you measure. In this view, you become a team who optimizes value delivery to end users while keeping a more holistic picture of quality in view.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3fa74c8716d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Message Queuing in the Cloud]]></title>
            <link>https://medium.com/statuscode/message-queuing-in-the-cloud-7816c1f2614e?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7816c1f2614e</guid>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[cloud-development]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Sat, 05 Sep 2020 19:47:46 GMT</pubDate>
            <atom:updated>2023-11-03T21:33:26.068Z</atom:updated>
            <content:encoded><![CDATA[<p>A message queue is a service that allows <em>producers</em> to place serialized messages onto a highly reliable queue from which one or more <em>consumers</em> can read those items. Message queues are an essential aspect of decoupled, event-driven architectures and in this post I cover a few of the potential use cases and tradeoffs.</p><h3>Message Queue Use Cases</h3><h4>Non-blocking UX: Word Counts for Medium Drafts</h4><p>When writing a Medium.com post you may notice the drafts list page will show a word count for each post. Upon closer inspection you may also notice it often takes several minutes for the word count to display accurately. This <strong>eventually consistent </strong>user experience could very well be implemented using a message queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*qVfiVrUZe1leHfI7XwZV4g.png" /><figcaption>A draft word count that doesn’t update in real time</figcaption></figure><p>The (476 words) so far word count isn’t updated in real time resulting in a tradeoff.</p><ol><li>Autosaving is much faster when words aren’t counted as part of the operation. This has the convenient side-effect of minimizing blocking actions such as an alert preventing you from navigating away while the draft is automatically saving.</li><li>Autosaving is more robust because parsing rich text in particular can be error prone. A failure to parse rich text should not prevent a draft from saving.</li></ol><p>One way to build a more robust autosave with lower latency is for the autosave operation to persist the draft immediately (blue arrows) while enqueuing the ID of the draft on a message queue to be processed later (orange arrow) by an autoscaled Function or Virtual Machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*Vu6AepCCHIJ0LcrO4EDRKw.png" /><figcaption>An eventually consistent word count feature for Medium draft stories</figcaption></figure><p>Note that the above diagram implements the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/claim-check">claim check pattern</a> in order to avoid message length limits (<a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html">AWS</a> / <a href="https://docs.microsoft.com/en-us/azure/storage/queues/storage-queues-introduction#:~:text=A%20queue%20message%20can%20be,limit%20of%20a%20storage%20account.">Azure</a>) which can be exacerbated by including draft text with extensive usage of rich text in queue messages.</p><blockquote><strong>Performance tradeoff:</strong> A potential improvement could be to use a conditional claim check pattern by conditionally including the entire draft in the message if the draft size is small enough. This will decrease database load for smaller drafts at the expense of causing the persisted word count to lag behind even more by counting words of a potentially outdated draft.</blockquote><h4>Batch and Throttle Requests: Usage Analytics</h4><p>Sites tracking concurrent usage using third-party APIs may have to work around rate limits imposed by those external APIs. Rate limiting becomes an increasing problem as the number of concurrent users increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*nBkTovoPzGgkB9v9kpsQKA.png" /><figcaption>Rate limiting enforced by a third-party API</figcaption></figure><p>The above diagram depicts thousands of users concurrently generating usage data and when a specified invocation per second threshold is exceeded some number of third-party API invocations may drop.</p><p>A simple way to address this problem is to use a message queue to buffer usage data, effectively throttling API calls, and process usage in batches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B3ZWPkguqHAd2lA7VttQ5w.png" /><figcaption>Message queues allow requests to be batched and throttled</figcaption></figure><p>The combination of processing usage data in batches while throttling third-party API invocations through tunable autoscaling parameters allows one to stay within third-party API rate limits even with high volume traffic.</p><p>An additional benefit that comes for free is if the third-party service temporarily goes offline the in-flight messages are not lost. When Usage Service fails to push messages to <a href="https://customer.io/">customer.io</a> those messages will become visible again on the queue to be processed at a later time.</p><h4>Priority Queuing: Serve Paying Customers First</h4><p>In a <a href="https://medium.com/@ckoster22/a-sufficiently-complex-cloud-application-842a13529648">previous post</a> I described how a message queue can be used to decouple cloud compute and storage services for a video stabilization service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*C-NAPvsKOCOmMPNF.png" /><figcaption>Architectural decoupling for a <a href="https://medium.com/@ckoster22/a-sufficiently-complex-cloud-application-842a13529648">video stabilization service</a></figcaption></figure><p>Hypothetically, this service could serve both paying and free trial members. In that scenario the SQS message queue above would be a single queue to handle requests from both paying and non-paying users. Understandably, the lack of improved service performance could be frustrating for users with paid membership.</p><p>With very little architectural change, the above SQS message queue can split into two creating a <strong>priority queuing architecture</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pZnWYZOqQ8Yc97bDS1__uw.png" /><figcaption>Priority queuing which favors paying members</figcaption></figure><p>A <em>free trial queue</em> would service free trial members and have limited compute resources serving their needs. Another <em>members only queue</em> would service paying members and dedicate substantially more resources to serve their needs, even borrowing compute resources from free trial members depending on how many messages are in each queue.</p><h3>Message Queue Considerations</h3><h4>Message Ordering and Duplicates</h4><p>Producers enqueuing messages on a queue may perform that operation as part of a more complicated action, an action not guaranteed to complete successfully. When a producer retries the action it may have already successfully enqueued a message causing a subsequent duplicate message to be placed on the queue.</p><p>Relatedly, some message queue implementations do not enforce strict message ordering. Messages may not enqueue in the same order they were received from producers.</p><p>One way to help mitigate these caveats is to process queue messages with idempotency. Operations like counting words of a draft specified inline to the message are naturally idempotent. Processing duplicate messages results in the same persisted word count. On the other hand, sometimes you’ll find yourself at the mercy of third-party APIs which may require <a href="https://blog.jonathanoliver.com/idempotency-patterns/">additional techniques</a> to achieve idempotency.</p><h4>UX for Eventual Consistency</h4><p>In another post I describe the <a href="https://medium.com/@ckoster22/product-engineering-best-practices-f28a5616062c">anti-pattern of happy path engineering</a>. From a user experience perspective eventual consistency is yet another less-than-happy path to be accounted for. For example, how should the tradeoff of counting words in near real time vs crafting a robust, minimally blocking autosave experience be considered?</p><p>This is yet another reason to involve Engineering early in feature ideation and design. It wouldn’t be reasonable for those outside of Engineering to be expected to master the ins and outs of cloud development and anticipate where eventual consistency will be employed. However, when everyone collaborates in unison these scenarios can be anticipated and accounted for earlier in the product development lifecycle.</p><h4>Poison Messages</h4><p>Popular cloud message queue implementations allow for peek or soft deletion policies where a failure to process a message will cause the message to remain on the queue. That works fine when the failure is a transient issue with the compute instance reading from the queue. Those may self-resolve. It’s a different outcome when the problem is with the message itself.</p><p>A queue message that will continually produce consumer failures and is repeatedly placed back on the queue for other consumers to ultimately fail is called a <em>poison message</em>.</p><p>Poison messages can be dealt with by observing message read counts and conditionally sending messages deemed as poisonous to a <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html">dead letter queue</a> to be processed separately and unblock consumers.</p><h4>Increased Complexity</h4><p>Message queues will add complexity even if incrementally. CI, traceability, dead letter queues, cognitive load for developers. Each consideration may be minor but it can add up.</p><p>This is a tradeoff that should be considered carefully. Is a real time word count feature worth the additional point of failure and a longer autosave operation? Does dropping a subset of usage data still provide enough signal for future business decisions? Should paying members have to wait as long as free trial members for video stabilization? Maybe!</p><p>Note that a message queue is not an all or nothing solution. It is completely possible to maintain a monolith while at the same time identifying a new service as needing to be backed by a message queue at the expense of incrementally added complexity. Message queuing is a tool and sometimes it’s the right tool for the job.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7816c1f2614e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/statuscode/message-queuing-in-the-cloud-7816c1f2614e">Message Queuing in the Cloud</a> was originally published in <a href="https://medium.com/statuscode">Statuscode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Engineering Best Practices]]></title>
            <link>https://medium.com/statuscode/product-engineering-best-practices-f28a5616062c?source=rss-86e753fbc92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f28a5616062c</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[product-engineering]]></category>
            <dc:creator><![CDATA[Charlie Koster]]></dc:creator>
            <pubDate>Thu, 20 Aug 2020 12:45:35 GMT</pubDate>
            <atom:updated>2020-11-14T15:24:09.213Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r6k-NOShh0Zd1NKhdjU0EA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@sxoxm">Sven Mieke</a> on Unsplash</figcaption></figure><p>In more than a decade in a variety of settings I’ve observed a few dos and don’ts reappear in several domains. In this post I share the anti-patterns and best practices I feel are most valuable to consider.</p><h4>Anti-pattern: Sending Requirements Over the Fence</h4><p>It’s not uncommon for product development to be partitioned by department or even by company as is commonplace in software consulting. There may be a somewhat discrete separation between the <em>what</em> and the <em>how</em> for software projects.</p><p>Requirements and designs that are conceived by one group and handed off to another can have some major drawbacks. Consider all the conversations that go into devising individual features. The weighing of pros and cons. The cost vs value-add tradeoffs. The research, the user interviews, the raw ideas that are debated and iterated on and molded into great ideas.</p><p>This context is extremely valuable when developers build the feature because very often there are dozens of micro-implementation tradeoffs that will be made autonomously by the Engineering team. Having that context in mind ensures those tradeoffs align with the overall vision of the feature.</p><p>On the flip side of this, when Engineers aren’t involved with specifications this creates an <em>empathetic vacuum</em>. Engineers will find it harder to relate to the end result making it easy to fall into a “devs for hire” mentality and become divorced from the mission of the product. Involving Engineering early in the product design lifecycle is a great way to combat this mentality and stoke the team’s intrinsic motivation. Everyone wants to feel a strong sense of fulfillment. Feeding into that as part of product development will quickly produce compounding benefits.</p><h4>✔️ Best practice: Ownership Over a Slice of the Mission</h4><p>In recent years the idea behind ownership in the tech industry shifted from teams owning a technology silo, such as the frontend or backend, to teams owning the full vertical stack of a monolithic slice or microservice. This shift towards vertical ownership is a huge leap in the right direction.</p><p>Pressing forward in that same direction means extending ownership beyond the technology stack and<em> </em>additionally embracing ownership over solving the end users’ needs<em>.</em> Phrased differently, <strong>it’s ownership over a thin slice of the product’s mission.</strong></p><blockquote>Extend ownership to encompass a slice of the mission in addition to the vertical stack.</blockquote><p>For example, rather than a team owning only the full stack behind a dashboard page the team would take on ownership over ensuring users can quickly find relevant information and efficiently administer common tasks. It’s <em>ownership over the problem that page is attempting to solve</em> from the database to the UI to how well the feature is meeting users’ needs.</p><p>It’s a subtle, yet important, distinction because <em>ownership defines the boundaries of accountability </em>and this boundary crosses the Product-Engineering line. Extending accountability across this line serves as a positive forcing function for Product, Design, and Engineering to empathetically work as a singular unit. Everyone is in the same boat and all must work together to serve the product’s mission.</p><h4>Anti-pattern: Happy Path Engineering</h4><p>One skill many end users have is the ability to find ways of using your product in ways you did not anticipate. Even though we have all observed this firsthand it’s a fact that is easy to breeze over when planning a project.</p><p>Consider what it takes to build a page which shows items in a virtual shopping cart. The happy path certainly seems simple. Users can review their carts items, add and remove them, and proceed to checkout. However, the paths users will encounter is something else.</p><ul><li>What should users see while the page is asynchronously fetching cart items from the database? Are there placeholder rows and images?</li><li>What happens if one of multiple API requests fail? A failure to retrieve a product’s thumbnail image may need to be handled differently than an API call that determines if an item is still in stock.</li><li>What does the user see if the cart is empty?</li><li>If they have many items in the cart which part of the viewport is scrollable? Can the shopping cart be so large there needs to be a “Back to top” button or pagination?</li><li>Do long cart item names get clipped or line wrapped?</li><li>What about accessibility, different devices, and different browsers?</li></ul><p>These less-than-happy paths will be handled eventually one way or another. Note that it is orders of magnitude cheaper to work through those paths earlier in the feature’s lifecycle as compared to waiting for users to discover these paths on their own.</p><h4>✔️ Best practice: Encapsulate Desired Behavior</h4><p>This best practice breaks down into two categories. The first is covered in <a href="https://medium.com/@ckoster22/invariant-driven-development-8231add95e33">another post</a> and relates to bettering technical implementation of a project by using an invariant first approach.</p><p>The second aspect more closely tied to product engineering is to <strong>use a Design System</strong>.</p><p>A Design System describes the tooling which encapsulates and enforces aspects of product design. At its heart, a team effectively using a Design System will start with the product’s <a href="https://bradfrost.com/blog/post/atomic-web-design/">atomic components</a>. The typography, layout, palette, and other fundamental concepts. These atoms compose together to make higher level components such as headings, menus, and <a href="https://uniform.hudl.com/patterns/forms">forms</a>. We’re talking about the rigidly constructed legos that snap into place with other components and give the product a sense of unity and holism. We’re also talking about <strong>never creating ad hoc components and styles</strong>. Everything on the site is a Design System lego and nothing is duct taped together.</p><p>Design Systems can be taken even further. I once worked on a team that implemented every new page disconnected from any backend within our Design System and <em>QA’d that page with the Designers </em>prior to putting it in the actual product.</p><p>This Design QA step included a simple yet incredibly powerful plugin which allows one to manipulate the Design System component in real time to understand how it deals with edge cases. Is it responsive? Does it handle long names? Is there a fixed header when scrolling? It allowed us to test many paths users are likely to encounter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*oiQWGAJ_huXH5lzDKnKoWg.gif" /><figcaption><a href="https://storybook.js.org/">Storybook</a> + <a href="https://github.com/storybookjs/storybook/tree/master/addons/knobs">React Knobs</a></figcaption></figure><p>Using a Design System to its fullest ability did wonders for delivering a quality and holistic user experience.</p><h4>Anti-pattern: Mythical Person-Months</h4><p>There are a variety of reasons why giving an FTE-Month estimate is problematic:</p><ul><li><strong>Scope will grow. Requirements will change.</strong> These are immutable facts of projects of any size and paradoxically are the facts most likely to go unacknowledged at the start of a project.</li><li><strong>Overly optimistic estimates are incentivized. </strong>This statement is unfortunately particularly true in software consulting, although, outside of consulting other forces such as quarterly metrics, upcoming investor meetings, or even the natural desire to report good news upward can lead to optimistic leaning estimates.</li><li><strong>The tech industry tends to use </strong><a href="https://medium.com/@ckoster22/agility-without-story-points-55bbce542906"><strong>unreliable methods</strong></a><strong> for estimating.</strong> Providing estimates for anything that spans multiple weeks or months is already a difficult undertaking. With software and tech generally being less tangible and more complex than other industries it is certainly no easier to confidently anticipate how long it will take to implement a large feature or project.</li></ul><p>Repeating advice from a <a href="https://medium.com/@ckoster22/agility-without-story-points-55bbce542906">previous post</a>, estimates are a means to an end and one way to achieve that end is to frequently and empathetically communicate with stakeholders. Strangely enough, frequent face-to-face communication is the most effective alternative to estimates I’ve ever employed. It achieves the same intent of providing stakeholders with information to make business decisions without presuming the effectiveness of software estimates.</p><h4>Anti-pattern: Viewing Project Completion as a Goal Line</h4><p>On paper a project has fairly well-defined boundaries. It’ll start around this date, staffed with these individuals, journey through specific milestones, and lastly it will roll out to production. Project compete.</p><p>Though the project may be done on paper there are two important ongoing activities to consider. Firstly and most importantly, if users are using new features in unanticipated ways, or their needs aren’t being met, then it is time to iterate (refer to the ownership best practice above). The project is only tentatively done so long as their needs aren’t met.</p><p>Additionally, with the release of new features comes with it growth in what that team is accountable to monitor, maintain, and <a href="https://medium.com/@ckoster22/a-zamboni-fire-drill-a8e649c751c4">fire fight</a>. This is analogous to the purchase of a vehicle. As that vehicle is driven by end users there may be problems along the way and general upkeep required. Purchasing additional vehicles (completing subsequent major features) only adds to the overall ongoing maintenance required.</p><h4>✔️ Best practice: Cut Scope Until It Hurts — Build &amp; Iterate</h4><p>The most successful teams I’ve worked with are the ones who took the cut scope <em>“until it hurts”</em> aspect of this best practice to heart. When you’ve cut project scope until it’s literally too painful to cut anymore, that’s when you know you’ve found your minimally viable product. That’s when you know you’re exercising lowercase “a” agility.</p><blockquote>When you’ve cut scope until it’s literally too painful to cut anymore, that’s when you know you’ve found your minimally viable product.</blockquote><p>Remember, scope will grow. Unexpected situations will come up. Overly optimistic estimates will be confronted by the real world. One way or another reality will take the plan that looked great on paper, turn it on its side, and you’ll find yourself pressed for time.</p><p>Tangibly, this best practice breaks down into the following:</p><ul><li>Start with a project scoped seemingly too small (Don’t worry! It’ll grow again). Break it down into small value-driven releases with bias toward the riskier and critical path items first.</li><li>Get something of value into production soon and repeatedly</li><li>Observe. What did you learn? What technical hurdles didn’t you anticipate? How are users using what’s released so far?</li><li>Iterate. Continually take what you’ve learned to reshape what you’ve already built or prioritize what to build next, build it, and release it to production. Do this again and again until the next most valuable thing the team can work on is something else.</li><li>Keep technical upkeep in mind. Make time to take a breath and tune up the clockwork that keeps those features ticking.</li></ul><h4>✔️ Best practice: Measure What Matters</h4><p>I am an avid subscriber of a <a href="http://web.mit.edu/hauser/www/Papers/Hauser-Katz%20Measure%2004-98.pdf"><em>You Are What You Measure</em></a> philosophy. In brief we all tend to, either purposefully or subconsciously, shape our decisions and behavior to optimize what we’re measured against.</p><p>As an example, a team measuring number of bugs may intend for that metric to lead to a higher quality product. However, with the metric aimed at bug count and nothing to do with quality the team may optimize on the bug count specifically. Similar bugs will tend to get grouped together. Would-be new bugs may get rolled into specifications for upcoming work. They may even make claims of “that’s a feature not a bug”. All of these resulting actions attempt to minimize bug counts while none optimize on quality, the intent behind the metric.</p><p>The opposite effect of this philosophy is <strong>that which is not measured is not optimized</strong> and often ignored until it reaches critical levels. Quality assurance, operating costs, time spent context switching, system wide complexity, sustainable work hours. When important metrics aren’t explicitly measured their remedies will naturally be superseded by actions that will optimize metrics that are explicitly measured.</p><p><em>You are what you measure.</em> Ask yourself whether the optimization of your metrics will reliably lead to a desired outcome. Or can it be gamified and accidentally lead to an unexpected outcome? Carefully designed metrics are extremely beneficial when the natural gamification of those metrics reliably leads to the intent of that metric.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f28a5616062c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/statuscode/product-engineering-best-practices-f28a5616062c">Product Engineering Best Practices</a> was originally published in <a href="https://medium.com/statuscode">Statuscode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>