<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marc H. Weiner on Medium]]></title>
        <description><![CDATA[Stories by Marc H. Weiner on Medium]]></description>
        <link>https://medium.com/@mhweiner?source=rss-8377f8231223------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Ivhd1JRjRquI9X4GG_XA0g.jpeg</url>
            <title>Stories by Marc H. Weiner on Medium</title>
            <link>https://medium.com/@mhweiner?source=rss-8377f8231223------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:57:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mhweiner/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Simplify Your Automated Releases with Autorel]]></title>
            <link>https://mhweiner.medium.com/introducing-autorel-simplifying-automated-releases-5ce5255e3a24?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ce5255e3a24</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[semantic-versioning]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Wed, 24 Jul 2024 13:23:23 GMT</pubDate>
            <atom:updated>2024-07-25T15:23:14.414Z</atom:updated>
            <content:encoded><![CDATA[<h4>I built a simpler and safer alternative to semantic release.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gSEFb3NBgVmTbq3zF2Zc0g.png" /></figure><p>When I adopted <a href="https://github.com/semantic-release/semantic-release">semantic-release</a> in 2017, it solved a big problem arising from the move to Continuous Delivery (CD).</p><p>At <a href="https://www.shutterstock.com/">Shutterstock</a>, we had an automated CI/CD pipeline, but it still required manually pushing the right tags in the right format, cutting releases, updating the changelog, and updating the version number in a file. Our pipeline would then perform actions based on those steps — including deploying to the correct environment.</p><p>Mistakes could take down a mission-critical system that processes millions of dollars annually.</p><p>All manual processes are prone to human error; the more steps, the more likely you are to make a mistake.</p><p>However, automation can be a great way to do the wrong thing <em>at speed</em>. For example, if you end up pushing to the wrong branch or tagging an incorrect version number, you may accidentally push a release that you didn’t intend to — <a href="https://medium.com/engineering-managers-journal/deploy-gone-wrong-the-knight-capital-story-984b72eafbf1#:~:text=In%202012%2C%20due%20to%20a,over%2010%25%20of%20their%20value.">with serious consequences</a>.</p><p>Automation is all about striking a balance between speed and accuracy.</p><p>Semantic-release was the first popular tool to address this problem by automating this part of the release process. It analyzes the commit messages to determine the type of change (major, minor, or patch), bumps the version number, generates a changelog, cuts releases, and publishes to registries.</p><p>Unfortunately, semantic-release has major issues that have only gotten worse over time. I built Autorel to address these issues and provide a simpler, safer, and more accessible alternative.</p><h3>The Problems with Semantic-Release</h3><p>Launched in 2014, semantic-release is very complex with over 430 dependencies (including deps-of-deps) for just the default configuration.</p><p>One of its most important dependencies, <a href="https://github.com/conventional-changelog/conventional-changelog/issues">conventional-changelog</a>, has a recently opened issue <a href="https://github.com/conventional-changelog/conventional-changelog/issues/1290">“Reduce amount of dependencies”</a>, with over 60 dependencies of its own. Even its deps are being called out for having too many deps. Yikes.</p><p>This results in glacial install times, slow execution, and a large surface area for bugs, security vulnerabilities, version conflicts, and breaking changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wntt4VFo5_Q-vJOJSaMmrA.png" /><figcaption>Semantic-release with common plugins adds 471 packages! Holy moly.</figcaption></figure><p>The documentation is a disaster — a sprawling mess, spread across its different plugins and dependencies. Defaults are confusing, or not listed at all. Despite being designed to be flexible, it can be quite hard to customize — which you’ll probably need to do.</p><p>For example, semantic-release does not support the Conventional Commits standard out of the box, and doing so requires installing extra plugins and configuring them correctly. And even then, I’ve found that it doesn’t work as expected. For one, the plugin conflates the standard with commit types, which the standard does not define.</p><p>In fact, modifying commit types is not supported at all, not without modifying plugin source code.</p><p>To say semantic-release has a steep learning curve would be an understatement.</p><p>There are over 1000 open issues across the core repo and its dependencies. Many are bugs, configuration issues, or confusion arising from the learning curve and poor documentation.</p><p>Critically, semantic-release is not defensive. It doesn’t do much validation and its easy to get wrong (see “<a href="https://blog.codinghorror.com/falling-into-the-pit-of-success/">pit of success</a>”). This can lead to frustration, wasted time, and bad releases. Again, automation is all about striking a balance between speed and accuracy. <a href="https://www.facebook.com/notes/10158791436142200/">Mistakes can be disastrous</a>.</p><p>Another longstanding and unresolved issue many folks have is GitHub rate-limiting errors. Semantic-release spams GitHub’s API and doesn’t do backoffs, which can result in your releases grinding to a halt.</p><h3>Introducing Autorel</h3><p>Similar to semantic-release, it’s a command line tool (though you can also use it as a library) to automate your releases. Here’s what it looks like in action:</p><pre>npx autorel --publish --run &#39;echo &quot;Next version is ${NEXT_VERSION}&quot;&#39;</pre><p>This will:</p><ul><li>Bump the version based on the commit messages following Conventional Commits and SemVer</li><li>Create a new release on GitHub with automatically generated release notes</li><li>Publishes the package to NPM with the proper tag</li><li>Runs the specified echo command</li></ul><p>It’s a simple, <a href="https://github.com/mhweiner/autorel/blob/main/README.md">well-documented</a> command that you can add to your CI/CD pipeline. You can even <a href="https://github.com/mhweiner/autorel?tab=readme-ov-file#example-usage-library">use it as a library</a> and build a custom release tool on top of Autorel.</p><p>Here is what the output looks like, this time in dry-run mode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnIfV3q8Vde_9N9GHNxg8g.png" /></figure><p>Here is an example configuration (all of which is optional, with sensible defaults):</p><pre># Define the branches and their respective channels<br>branches:<br>  - {name: &#39;main&#39;}<br>  - {name: &#39;next&#39;, prereleaseChannel: &#39;next&#39;}<br><br># Enable publishing to NPM<br>publish: true<br><br># Run custom script after publish<br>runScript: |<br>  echo &quot;$(date +&quot;%Y-%m-%d&quot;) ${NEXT_VERSION}&quot; &gt;&gt; versions.txt<br>  aws s3 sync . s3://my-bucket</pre><p>If you want to specify your own commit types instead of the defaults, add them to the configuration file:</p><pre>commitTypes:<br>  - {type: &#39;feat&#39;, title: &#39;✨ Features&#39;, release: &#39;minor&#39;}<br>  - {type: &#39;fix&#39;, title: &#39;🐛 Bug Fixes&#39;, release: &#39;patch&#39;}<br>  - {type: &#39;perf&#39;, title: &#39;🚀 Performance Improvements&#39;, release: &#39;patch&#39;}<br>  - {type: &#39;misc&#39;, title: &#39;🤷‍♀️ Miscellaneous&#39;, release: &#39;none&#39;}</pre><p>If you have an invalid configuration, Autorel will catch it and give you a helpful error message. It’s designed to be simple, safe, and flexible.</p><p>Autorel is written in Typescript with good <a href="https://github.com/mhweiner/autorel/tree/main/src">unit test coverage</a>, particularly around <a href="https://www.conventionalcommits.org/">Conventional Commits</a> and <a href="https://semver.org/">SemVer</a> compliance.</p><p>With a small codebase and only <a href="https://www.npmjs.com/package/autorel?activeTab=dependencies">three dependencies</a>, it has a much smaller surface area for bugs and security vulnerabilities.</p><p>It’s super easy to learn and incredibly fast to install and run. It’s also flexible, with straightforward configuration and support for bash scripting out of the box. Or use it as a library within your custom build tool.</p><p>Autorel currently has built-in support for NPM and GitHub, but you can easily add support for other platforms by writing simple bash scripts. Most integrations are just simple curl requests. Don’t over-complicate it :-)</p><h3>Now it’s your turn</h3><p>Semantic-release paved the way for automated releases, but it’s time for a simpler, safer, modern alternative.</p><p>Autorel won’t be everything to everyone. It has a different approach and isn’t as feature-rich. However, it could be a great alternative for those who want something simpler, safer, or that offers easier customization.</p><p>Autorel is currently being used in production at <a href="http://aeroview.io/">aeroview.io</a>, for my <a href="https://github.com/mhweiner">open-source projects</a>, and by others.</p><p>What do you think? I’d love to hear your feedback and suggestions. We’re also looking for contributors. You can find the code on <a href="https://github.com/mhweiner/autorel">GitHub</a> and the package on <a href="https://www.npmjs.com/package/autorel">NPM</a>.</p><p>Marc H. Weiner is a startup veteran (Advizr, acq’ 2019), specializing in mission-critical software. He’s worked on Electronic Health Records, clinical trial software, massive e-commerce infrastructure, financial planning apps, rocket testing equipment, and more. He’s the co-founder of <a href="http://aeroview.io/">aeroview.io</a> and writes about software reliability here on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ce5255e3a24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ugly Code Will Kill Your Startup]]></title>
            <link>https://mhweiner.medium.com/ugly-code-will-kill-your-startup-194c9005acae?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/194c9005acae</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Sun, 12 Nov 2023 19:07:40 GMT</pubDate>
            <atom:updated>2024-02-19T17:29:42.108Z</atom:updated>
            <content:encoded><![CDATA[<h4>Misconceptions in the move-fast-and-break-things era</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i8SaIuS_yn3FDLNmhSgGSQ.png" /><figcaption>Image generated by DALL-E v3</figcaption></figure><p>A Computer Science undergraduate program will teach you all about algorithms, data structures, and logic gates.</p><p>But one thing they will barely cover is code readability.</p><p>This trend is echoed in the software industry where rapid development often overshadows the need for clear code. The “move fast and break things” motto is the rallying cry for many tech startups.</p><p>Some even boldly claim that “ugly” code has never been attributed to the failure of a startup — a claim as dubious as it is misleading. It underscores a fundamental misunderstanding of software, how it’s written, and its associated costs.</p><p>Of course, beauty is in the eye of the beholder. If one were to take this claim literally, it would be impossible to prove or disprove, as opinions over aesthetics vary — unless we’re talking Perl or PHP, of course.</p><p>However, it’s much more likely they’re talking about a codebase that is messy, suffers from poor readability, doesn’t follow best practices, or suffers from “technical debt.”</p><p>Allow me to introduce you to Knight Capital, the company that <a href="https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html">wiped itself out in 45 minutes</a> after its automated trading system racked up $440 million in losses due to a simple coding mistake. If Knight Capital had cleaned up its unused “dead code,” it couldn’t have accidentally executed this disastrous so-called “Power Peg” code that took down the company. In fact, keeping around dead code is a <a href="http://cwe-561/">well-known vulnerability</a>.</p><p>Knight Capital might be an extreme example, but it’s not alone. Plus, ugly code is doing more harm to your business than just exposing it to liability, making your products unreliable, and frustrating your users. It’s slowing down your development, hampering your agility, and increasing your costs.</p><h3>The Need for Speed</h3><p>Whenever I see someone weigh the merits of speed vs. quality in software, my eyeballs roll into the back of my head.</p><p>This is a widely proliferated false dichotomy. It’s not a thing. According to computer science pioneer and Turing Award winner Edsger Dijkstra:</p><blockquote><em>Software seems to be different from many other products, where as a rule a higher quality implies a higher price. Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging.</em></blockquote><p>The more time we spend chasing down elusive bugs, the less time we’re pushing features and delighting users.</p><p>Code quality is the <em>ability for us to understand and change</em> said code. If we can’t understand it, we can’t change it. If it’s tightly coupled, we can’t change it. If it’s poorly tested, we can’t change it. If it’s relying on outdated dependencies, we can’t change it.</p><h3>What Move Fast and Break Things Really Means</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*4eiKY4axXcvJ_aOLwrn7Og.gif" /></figure><p>“Move fast and break things,” a mantra made famous by Facebook’s Mark Zuckerberg, actually speaks to a culture of short iteration cycles, not a compromise on code quality. And it wasn’t new.</p><p>This philosophy, with its origins in the <a href="https://en.wikipedia.org/wiki/Fail-fast">fail-fast principle</a> from the 1950s space race, encourages swift prototyping and testing to identify and discard flawed designs early on.</p><p>While Facebook, as a social networking platform, could afford more leeway in deploying less-tested code, it didn’t mean they cared any less about the quality of the code (ie, readability) overall.</p><p>How the fail-fast principle can be implemented varies across industries. In sectors like healthcare and aerospace, where stakes are higher, a longer development cycle is required to accommodate more testing and compliance. Instead, the emphasis is placed on short testing programs designed to help determine viability, and whether or not to continue to invest in a particular design.</p><h3>How Software Bugs Are Made</h3><p>John Carmack, of <em>Doom</em> and <em>Quake</em> fame, made an interesting observation on software bugs:</p><blockquote>A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in.</blockquote><p>Carmack’s claim might sound reductionist or even esoteric, but at its core is a profound insight that I strongly believe is at the heart of one the software industry’s biggest problems: The debate over code readability isn’t about style or aesthetics — it’s about human comprehension.</p><p>As much as we’d like to believe it, most software isn’t complicated mathematical algorithms, quantum mechanics simulations, or neural networks. It’s much more ordinary stuff, such as business logic, network calls, and user interfaces.</p><p>This kind of software is built on layers of abstraction, a structured metaphor devised for practical navigation through the complexities of our digital constructs. As Kevlin Henney often says, “Software is executable fiction.” Software isn’t f***ing real.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*fmPV6dD297IsemP8FKVPEw.gif" /></figure><p>The real challenge, it turns out, is to not get confused by the complexities of our design. Even Alan Turing, the father of modern computing, predicted this back in 1947:</p><blockquote><em>“It will be seen that the possibilities as to what one may do are immense. One of our difficulties will be the maintainence of an appropriate discipline, so that we do not lose track of what we are doing.”</em></blockquote><p>When I learned of this quote by Turing, I was shocked. To think that someone as brilliant as Turing, in the 1940’s no less, predicted the daily struggle of programmers today!</p><p>With the invention of higher-level programming languages such as FORTRAN, we stopped doing things by directly manipulating the CPU’s registers and instead, created higher-level constructs that we humans can better reason about.</p><p>Most software bugs are simply a misunderstanding of mundane details.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3fGHaVn5rGo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3fGHaVn5rGo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3fGHaVn5rGo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/764ebee8a7ff9d2a2a51e9b2955efaaf/href">https://medium.com/media/764ebee8a7ff9d2a2a51e9b2955efaaf/href</a></iframe><p>It doesn’t matter how smart you are, keeping track of a half dozen mutable variables in your head and following the myriad paths of logical branches is a Herculean task ripe for errors.</p><h3>We Can Do Better</h3><p>I hope by now I’ve made a strong argument for code quality, and the impact it can have on any organization, even fast-paced startups.</p><p>The trillion-dollar questions are:</p><ul><li>What tools or methods do we have to maximize human understanding and minimize mistakes?</li><li>Why aren’t more companies investing in them?</li><li>Why aren’t these included in the Computer Science curriculum?</li></ul><p>In a future article, we’ll explore some pragmatic ways we can improve code readability (with clear examples), such as:</p><ul><li><strong>Making wrong code look wrong</strong> through convention and best practices</li><li><strong>Clear naming</strong> (this is always a hard one)</li><li><strong>Literary tools</strong> that writers authors use to effectively communicate ideas</li><li><strong>Type systems</strong>, such as the ones used in Typescript</li><li><strong>Function and declarative programming</strong> (don’t worry, we’re not going full lambda calculus on you)</li><li><strong>Coding by proximity</strong> (keep relevant things together)</li><li><strong>Simplifying code </strong>by breaking things apart and leveraging the SRP (Single Responsibility Principle)</li><li><strong>Pragmatic unit tests,</strong> specifically Behavior Driven Development (BDD) and specification-first programming</li></ul><p>If you’re interested in seeing that, follow me and subscribe to email notifications so you don’t miss it.</p><p>Looking for help to make your company’s software more reliable? <a href="https://mhweiner.com/">I’m open to work!</a> 👋 I specialize in mission-critical applications, having worked on such projects in aerospace, healthcare, cloud computing, and finance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=194c9005acae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Worf’s Guide to Error Handling in Node.JS]]></title>
            <link>https://javascript.plainenglish.io/worfs-guide-to-error-handling-in-node-js-2b367c5ef079?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b367c5ef079</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[error-handling]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Wed, 08 Nov 2023 14:16:57 GMT</pubDate>
            <atom:updated>2023-11-08T14:16:57.801Z</atom:updated>
            <content:encoded><![CDATA[<h4>Error Handling in Node Applications From a Warrior’s Perspective</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qmuFtS2H3pUfZ2IsoEI7fw.png" /><figcaption>Image created by author with DALL-E v3</figcaption></figure><p><em>This piece is written for educational and comedic purposes, aligning with fair use provisions. Worf, Klingons, and all Star Trek characters are owned by Paramount and CBS.</em></p><p>Written by Worf, son of Mogh</p><p>In the heat of battle, a warrior must confront failure with courage and discipline. As someone who has navigated the treacherous terrain of both starship combat and server-side JavaScript, I offer you a guide to error handling that merges the art of war with the science of programming.</p><h3>Understand the Battlefield: Handle What You Can</h3><p>In the heat of development, not all errors are fatal (in fact, <a href="https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf">most are recoverable</a>) and not all require immediate attention. A warrior is never afraid of death but chooses their battles wisely. Remember the adage, “Handle what you can, let go what you can’t.”</p><p>If an error is not within your domain, do not intercept it, and let it pass through. This practice ensures that errors are not swallowed silently, leaving you to wonder why your application has failed without a trace of the enemy’s passage.</p><h3>The “Last Stand” Error Handler</h3><p>Every warrior knows the value of being prepared for the unexpected. In code, this translates to having a robust “last stand” error handler — a final checkpoint where unhandled exceptions are caught, and appropriate actions are taken. It should log the error, alert the engineering team, and perform any necessary cleanup, ensuring the application does not fall in battle but dies an honorable death, ready to be restored to glory.</p><p>In the context of an Express application, after the error is handled and logged appropriately, the server can continue to honor other requests without delay:</p><pre>// Define the &quot;last stand&quot; error handler<br>app.use((error, req, res, next) =&gt; {<br>  // Log the error for the engineering team to review<br>  console.error(&#39;Unhandled Exception Alert:&#39;, error);<br><br>  // Perform necessary cleanup<br>  // ... (Cleanup logic goes here)<br><br>  // Send a generic error response to the client<br>  res.status(500).send(&#39;An unexpected error occurred on our end. Preparations to restore service are already underway.&#39;);<br><br>  // The server continues to handle other requests<br>});<br><br>// Utilize this handler by wrapping your controllers in a try/catch<br>app.get(&#39;/some-route&#39;, async (req, res, next) =&gt; {<br>  try {<br>    // ... (Controller logic goes here)<br>  } catch (error) {<br>    // An error occurred, let the &quot;last stand&quot; handler take over<br>    next(error);<br>  }<br>});</pre><p>This is not a sign of defeat, but rather a strategic preparation for the worst-case scenario. It’s our last line of defense, ensuring that when our code meets its match, it does so with honor.</p><h3>Allow Failed Processes to Die With Honor</h3><p>Just as one does not dishonorably attempt to resuscitate a fallen warrior, do not try to save a fatal error, such as a Syntax Error, with an uncaughtException event handler. Processes kept alive in this way are unstable and can fail perilously without honor. Accept its fate, log the error, and let the process restart gracefully, such as with <a href="https://pm2.keymetrics.io/">pm2</a>. Today is a good day to die.</p><h3>Throw Real Errors</h3><p>A Klingon Bat’leth strikes true and so should your errors. Always throw real errors with stack traces, not mere strings. These are your weapons in understanding the enemy’s movements, revealing not just where the failure occurred but the path it took to get there.</p><h3>Custom Messages and Context Objects: Extending the Error Class</h3><p>Extend the Error class to accept custom messages and context objects, as a Dahar master would craft his own blade. This custom extension allows errors to carry with them their full story and context, providing invaluable insights for your team.</p><pre>export class ErrorWithData&lt;D&gt; extends Error {<br><br>    data?: D;<br><br>    constructor(message: string, data?: D) {<br><br>        super(message);<br>        Object.setPrototypeOf(this, new.target.prototype); // restore prototype chain<br>        this.name = this.constructor.name;<br>        this.data = data;<br><br>    }<br><br>}</pre><p>When you throw an exception, provide context as you would provide coordinates to a fellow warrior. An object that describes the situation gives your engineers the strategic insight needed to counter the problem swiftly and efficiently.</p><h3>Error Serialization and Structured Logs</h3><p>An effective error serialization process should be as resonant and disciplined as the war drum on a Klingon vessel, combined with the structured chain of command of a Bird-of-Prey. Output errors to STDERR in structured JSON format to announce the presence of issues clearly, ensuring they are detected, filterable, and searchable. Utilize libraries, such as <a href="https://github.com/mhweiner/jsout">jsout</a> or <a href="https://github.com/winstonjs/winston">winston</a> to automate the serialization of errors and structuring of log output, ensuring that they are consistently formatted for easy parsing by both developers and machines, thus maintaining order and clarity within your system’s error-handling protocols.</p><h3>Conclusion: The Warrior’s Path in Error Handling</h3><p>Error handling in backend software engineering is a path that requires both the heart of a warrior and the mind of a strategist. Embrace it with the understanding that each error presents an opportunity to strengthen your code’s defenses, much as each battle hones a warrior’s skill.</p><p>Remember, an application that has never faced errors is untested. It may hold potential, but its strengths— and weaknesses — are unknown. It is through the fires of failure and the forging of resolution strategies that both a warrior and a codebase prove their worth.</p><p>Qapla’!</p><p><strong>Up next: </strong>Data Migrations with Darmok: When the Walls Fell, So Did the Legacy Service</p><p>If you enjoyed this, please follow me and subscribe for email updates 🙏</p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li><li><em>Check out our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a>, <a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a>, <a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b367c5ef079" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/worfs-guide-to-error-handling-in-node-js-2b367c5ef079">Worf’s Guide to Error Handling in Node.JS</a> was originally published in <a href="https://javascript.plainenglish.io/">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Try/Catch Considered Harmful]]></title>
            <link>https://medium.com/better-programming/try-catch-considered-harmful-4238ddd7cd3c?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/4238ddd7cd3c</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Mon, 23 Oct 2023 09:37:37 GMT</pubDate>
            <atom:updated>2023-11-11T17:59:36.002Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring alternatives to improve readability in error handling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HvxI5D_ENJ1ca65-XhZLxA.jpeg" /><figcaption>Dijkstra at the blackboard during a conference at <a href="https://en.wikipedia.org/wiki/ETH_Zurich">ETH Zurich</a> in 1994. Credit: <a href="https://en.wikipedia.org/wiki/Edsger_W._Dijkstra">Wikipedia</a></figcaption></figure><p>In the annals of programming history, few papers have had as much impact as Edsger Dijkstra’s seminal work “GOTO Considered Harmful,” published in a 1968 <a href="https://www.acm.org/">Association for Computing Machinery</a> publication. His arguments against the indiscriminate use of the GOTO statement forever changed the trajectory of software development, giving rise to “structured programming.” In today’s world, while GOTO has become a relic, could there be modern constructs playing a similar detrimental role?</p><pre>10 PRINT &quot;Would you like to continue? (YES/NO)&quot;<br>20 INPUT RESPONSE$<br>30 IF RESPONSE$ = &quot;YES&quot; THEN GOTO 50<br>40 IF RESPONSE$ = &quot;NO&quot; THEN GOTO 60<br>45 PRINT &quot;Invalid response. Please type YES or NO.&quot;<br>46 GOTO 10<br>50 PRINT &quot;Continuing the program!&quot;<br>55 END<br>60 PRINT &quot;Ending the program now.&quot;<br>65 END</pre><p>Dijkstra’s primary grievance with GOTO was its disruption to<em> control flow</em>.</p><p>Control flow, at its essence, is the sequence in which a program’s instructions are executed.</p><p>To illustrate, imagine control flow as the conductor of an orchestra, ensuring each instrument plays in harmony. Overreliance on GOTO disrupted this harmony, akin to a scratched CD causing jarring skips in a song, throwing off the intended rhythm and melody. Those who recall the days of CDs can appreciate how such disruptions mar the listening experience.</p><p>GOTO’s arbitrary jumps introduced this chaos into code, leading to what many termed ‘spaghetti code.’ Such convoluted patterns not only obfuscated logic but also became hurdles in debugging and team collaboration.</p><p>In the current programming landscape, the overuse and deep nesting of try/catch can echo the challenges posed by GOTO. As we aim for clear and maintainable code, it’s crucial to draw upon past lessons to inform our present practices.</p><h3>The Problem with Try/Catch</h3><p>The main issue with try/catch structures is the non-linearity it introduces to the flow of code. This is especially egregious when nested. For human readers, the eyes have to jump around to follow the execution flow, much like following the jumps of the GOTO statement. This non-linear reading pattern strains cognitive processing and can make the code harder to understand at a glance.</p><p>Consider the following Typescript code:</p><pre>async function fetchUserData(): Promise&lt;UserData|undefined&gt; {<br>    let userData: UserData | undefined; // cannot use const<br><br>    try {<br>        const token: string = await fetchToken();<br><br>        try {<br>            userData = await fetchUserData(token);<br>        } catch (userError) {<br>            console.error(&quot;Error fetching user data:&quot;, userError);<br>        }<br><br>    } catch (tokenError) {<br>        console.error(&quot;Error fetching token:&quot;, tokenError);<br>    }<br><br>    return userData;<br>}</pre><p>To read this code, your eyes dart from the innermost function call to its associated catch, then outwards in a zigzag pattern. Wouldn’t it be more readable if the code simply flowed from top to bottom, in a linear fashion?</p><p>It’s also quite stateful, as const won’t work here due to the scoping issues created by the try/catch blocks, forcing us to declare mutable variables at the top. This prevents us from writing code in a more declarative style. This, in turn, introduces state change and possible “undefined” variable errors (Javascript’s equivalent to the null pointer exception).</p><p>Naturally, this is a very simple example. The real challenge and undue mental strain for developers arise when complexities increase.</p><p>For example, what if we wanted to re-throw new, more detailed errors instead of returning nulland logging to the console? This is actually problematic:</p><pre>async function fetchUserData(): Promise&lt;UserData|undefined&gt; {<br>  let userData: UserData | undefined; // cannot use const, must support undefined<br><br>  try {<br>      const token = await fetchToken();<br><br>      try {<br>          userData = await fetchUserData(token);<br>      } catch (userError) {<br>          throw new Error(`Error fetching user data: ${userError.toString()}`);<br>      }<br><br>  } catch (tokenError) {<br>       throw new Error(`Error fetching token: ${tokenError.toString()}`);<br>  }<br><br>  return userData;<br>}</pre><p>In this example, the innermost Error would be caught by the outermost one. Throwing a locally caught exception like this may be seen as poor practice. What if you wanted the first exception to act like an early return? Unfortunately, this isn’t possible with nested try/catch blocks.</p><p>Furthermore, we are required to have UserData | undefined as the return type, even though the function will always return UserData unless it throws. Typescript isn’t able to help us in this situation.</p><h3>A Better Way: The “Result” Pattern</h3><p>Borrowing from the Go programming language’s approach to error handling, we can define a new pattern that offers a linear and readable code structure.</p><p>I call this the “Result” pattern:</p><pre>async function fetchUserData(): Promise&lt;UserData&gt; {<br><br>    const [tokenErr, token] = await toResultAsync(fetchToken());<br><br>    if (tokenErr) throw new Error(`Error fetching token: ${tokenError.toString()}`);<br><br>    const [userDataErr, userData] = await toResultAsync(fetchUserData(token));<br><br>    if (userDataErr) throw new Error(`Error fetching user data: ${userDataErr.toString()}`);<br><br>    return userData;<br>}</pre><p>This pattern greatly simplifies the code by returning a tuple. The first value is an error (if any), and the second is the result.</p><p>Here, the flow of the program is clear, straightforward, and linear. Your eyes can glide smoothly from top to bottom, capturing the essence of the function’s behavior without needing to dart around.</p><p>This code is also significantly shorter, declarative with no stateful mutation, and is able to re-throw errors in a straightforward manner. Typescript is able to confidently tell us that this function only ever returns UserData, making it easier to use as well.</p><p>Let’s define these helper functions, which are provided for both synchronous and asynchronous use cases:</p><pre>export type Result&lt;E, T&gt; = [E] | [undefined, T];<br>export type PromiseResult&lt;E, T&gt; = Promise&lt;Result&lt;E, T&gt;&gt;;<br><br>export function toResult&lt;E extends Error, T&gt;(executable: () =&gt; T): Result&lt;E, T&gt; {<br>    try {<br>        const result = executable();<br>        return [undefined, result as T];<br>    } catch (e) {<br>        return [e as E];<br>    }<br>}<br><br>export async function toResultAsync&lt;E extends Error, T&gt;(p: Promise&lt;T&gt;): PromiseResult&lt;E, T&gt; {<br>    try {<br>        const result = await p;<br>        return [undefined, result as T];<br>    } catch (e) {<br>        return [e as E];<br>    }<br>}</pre><h3>Alternative Approach: Embracing Failure as Expected Outcomes</h3><p>Sometimes, what we perceive as an “error” or “exception” in a function isn’t truly exceptional but rather an expected and common outcome. Take, for example, user authentication. If a user inputs the wrong password, is that an exception? Or is it a frequent scenario that our system should handle gracefully? In cases like this, the delineation between standard return and error becomes blurred.</p><p>Instead of relying on exceptions to signal these common scenarios, we can redefine our function’s contract to return them as standard results. This not only simplifies the code but also shifts our perspective from seeing these outcomes as “failures” to viewing them as just another type of result that our functions can produce.</p><h4>Using Tuples for Expected Failures</h4><p>One way to handle expected failures is to return tuples directly:</p><pre>function authenticateUser(username: string, password: string): [boolean, string] {<br>    if (isValidUser(username, password)) {<br>        return [true, &quot;Authentication Successful&quot;];<br>    } else {<br>        return [false, &quot;Invalid Password&quot;];<br>    }<br>}</pre><p>The returned tuple consists of a boolean indicating success and a message string.</p><h4>Discriminated Unions for Richer Results</h4><p>For more complex scenarios where you might have a variety of results, you can use discriminated unions (also known as tagged unions):</p><pre>type AuthenticationResult =<br>    | { status: &#39;success&#39;; user: User }<br>    | { status: &#39;failure&#39;; reason: string };<br><br>function authenticateUser(username: string, password: string): AuthenticationResult {<br>    if (isValidUser(username, password)) {<br>        return { status: &#39;success&#39;, user: retrieveUser(username) };<br>    } else {<br>        return { status: &#39;failure&#39;, reason: &quot;Invalid Password&quot; };<br>    }<br>}</pre><h3>Wrap Up</h3><p>While try/catch is a powerful tool, its misuse or overuse can lead to tangled nests of code, reminiscent of the confusion sparked by the ‘goto’ statement. By leveraging patterns such as “Result” or “Failure as Return,” programmers can enjoy cleaner, less stateful, more maintainable code. A linear reading flow minimizes the cognitive overhead, allowing for quicker comprehension and more effective code reviews.</p><p>As always, the right tool should be used for the job, and understanding the context is essential.</p><p>Dijkstra’s paper on GOTO wasn’t merely a critique of a specific construct; it underscored the importance of linear control flow and minimal stateful mutation. The less we need to keep track of in our heads, the better. Humans can be great at abstraction and high-level concepts but generally make terrible compilers.</p><p>Whether we’re dealing with GOTO of the past or the potential pitfalls of try/catch today, the foundational principle remains the same — optimize for human understanding.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4238ddd7cd3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/try-catch-considered-harmful-4238ddd7cd3c">Try/Catch Considered Harmful</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Technical Debt is a Hard Sell]]></title>
            <link>https://mhweiner.medium.com/the-true-cost-of-technical-debt-5c668020b0e4?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c668020b0e4</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Tue, 17 Oct 2023 17:02:40 GMT</pubDate>
            <atom:updated>2024-09-04T05:46:03.528Z</atom:updated>
            <content:encoded><![CDATA[<h4>The psychology of tech debt and common misconceptions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vvqetTSyQ6DjEqsGozPJ0A.png" /><figcaption>Original image created with DALL-E</figcaption></figure><p>In the world of software development, we often find ourselves juggling between delivering features quickly and maintaining a clean, manageable codebase. This balance, or often the lack of it, gives rise to what’s commonly termed as “Technical Debt”. Much like financial debt, technical debt accumulates when you opt for a quicker, suboptimal solution now, with the intent to address its consequences in the future. It can also be the result of not keeping up with change, or by simply becoming obsolete.</p><p>While many developers often stress the importance of technical debt, it’s often dismissed by management. But why is addressing technical debt such a hard sell? We all know tech debt is bad, right? Are tech managers evil or incompetent?</p><p>The disasterous effects of technical debt are well documented. Take for example, the notorious healthcare.gov website for the U.S. Affordable Care Act. When it was launched in 2013, it faced significant issues due to it being buggy, inefficient, and hard to maintain, which <a href="https://www.washingtonpost.com/national/health-science/hhs-failed-to-heed-many-warnings-that-healthcaregov-was-in-trouble/2016/02/22/dd344e7c-d67e-11e5-9823-02b905009f99_story.html">led to its collapse upon launch</a>. It was so bad that it became a huge political issue for the Obama administration, with the President himself <a href="https://www.cnn.com/2013/10/21/politics/obamacare-problems/index.html">trashing the website</a>.</p><p>No, I don’t think most managers are stupid. I think there is a more practical explanation:</p><p>One reason why many don’t prioritize technical debt is that they simply don’t see the benefit when compared to other priorities. After all, it’s commonly understood that technical debt, similar to financial debt, can be paid back later. Even if it comes with added costs down the road, many theorize this cost will be worth paying once that big contract is landed, or that shiny new feature goes out.</p><p>And to an extent, they’re right. Technical Debt, similar to a loan, is a way to borrow from the future. It can indeed be a viable strategy for small rapid prototypes, or to hit a short deadline for small changes. As long as the code is re-written shortly after, we can call this debt “managed”.</p><p>A major caveat, however, is that this only works for tight-scoped projects where reliability or scaling isn’t a major concern.</p><p>Unfortunately, it seems many people don’t really understand the true nature of tech debt and its cost. Perhaps it’s because the financial metaphor only goes so far. After all, it’s just a metaphor — not a definition.</p><h3>Technical debt has an optics problem</h3><p>Many people believe having technical debt simply means added costs at a later date. However, that’s only a tiny part of it. Unfortunately, this common definition and the whole financial analogy are incredibly misleading.</p><p>At its core, technical debt is when it’s hard to change things. This can be due to outdated dependencies, vendor lock-in, absence of automated tests, lack of documentation, confusing code, outdated abstractions, tight coupling, or other deficiencies. Often, code with technical debt is hard to change because it’s hard to understand. Maybe it was actually good code once, but time has moved on, and the code hasn’t.</p><h3>The real cost of technical debt</h3><p>When Ward Cunningham coined the phrase in 1992, he noted that these “borrowings” manifest as “interest” not just for future costs, but more importantly, increased development times and decreased programmer productivity. For whatever reason, only that first part seemed to stick.</p><p>However, it’s actually <strong><em>so much</em></strong> worse. Unmanaged tech debt is the kiss of death for reliability, security, scalability, opportunity, and planning.</p><h4>Poor reliability</h4><p>A system bogged down with technical debt is exponentially more prone to bugs and crashes. This is due primarily to confusion resulting in a lack of understanding of the code and larger system.</p><blockquote>A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in. — John Carmack</blockquote><p>Software is basically a million-part <a href="https://en.wikipedia.org/wiki/Rube_Goldberg_machine">Rube Goldberg machine</a>, where it crashes if any one thing goes wrong. Why make it harder when you aren’t even sure how most of it works?</p><p>The tight deadlines and cumbersome development process these teams face also play a key role. Finding, understanding, and fixing these bugs can take a lot of time. And when things are already barely getting done on time, attention to detail and testing are often sacrificed, resulting in even more tech debt, which creates even more bugs. This is a vicious cycle that quickly results in an unrecoverable <a href="https://medium.com/better-programming/when-to-throw-in-the-towel-on-your-technical-debt-b1f6bbb3aa0c">software black hole</a>.</p><h4>Opportunity cost</h4><p>Every moment spent navigating the maze of a debt-laden codebase is time not spent on creating new features, innovating, or improving the system. This slows down the pace at which a team can deliver, causing missed opportunities for the business. For startups, this can be deadly.</p><h4>Security concerns</h4><p>Outdated dependencies or convoluted code can lead to vulnerabilities. With ever-increasing cyber threats, a system burdened with technical debt often becomes a low-hanging fruit for malicious actors. This can also result in major financial liability.</p><p>Equifax’s <a href="https://www.ftc.gov/news-events/news/press-releases/2019/07/equifax-pay-575-million-part-settlement-ftc-cfpb-states-related-2017-data-breach">2017 data breach</a>, affecting 147 million people and resulting in a $575 million fine, was largely due to its reliance on unsupported, obsolete technology.</p><p>Here’s an excerpt from the complaint filed by the FCC:</p><blockquote>ACIS [the affected software]… runs on old systems, many of which are no longer supported. Today, Defendant considers ACIS to be legacy infrastructure and its own documents describe the system as “archaic” and using “antiquated technology.”</blockquote><h4>Unpredictability</h4><p>Technical debt introduces uncertainty. Estimations become unreliable because engineers can’t predict the complexities they might uncover as they wade through the code. With a complex, debt-ridden codebase, every change becomes a potential landmine. The risk of introducing new bugs increases, making deployments a nail-biting event. Forget about your 12-month roadmap — you can’t plan for next week.</p><h4>High development costs</h4><p>Try putting together a 1000-piece jigsaw puzzle with some pieces missing, some the wrong shape, and the wrong picture on the box. That’s what it’s like working on a codebase with unmanaged tech debt.</p><p>The real cost isn’t just in the expensive refactoring tasks that loom in the future, it’s in the daily grind — the little inefficiencies developers face every single day, the workarounds, and the “this is how we’ve always done it” explanations. It’s the avoidable errors that compound over time. It’s the hours wasted chasing down preventable bugs, putting out fires, and the inevitable burnout and turnover.</p><h4>Scalability concerns</h4><p>Technical debt severely hampers scalability. Whether it’s scaling up the features, handling more user load, or even growing the team, tech debt makes it all daunting. Just look at the aforementioned healthcare.gov fiasco.</p><p>New hires face a steep learning curve, making onboarding inefficient. Deploying code might be an arduous, time-consuming task. The system itself may falter under load or become too rigid to adapt to new requirements.</p><h3>Programming is learning</h3><p>While a comprehensive guide to avoiding technical debt extends beyond the scope of this piece, there are a couple of simple things you can do right now, starting with the way you see and approach tech debt as a team.</p><p>One consideration is to restrict any debt “loans” to a single short development cycle. It’s crucial that any shortcuts or quick fixes introduced during this phase are promptly addressed afterward. Again, may not work with mission-critical or highly scaled systems.</p><p>More importantly, continuous code cleanup must be woven into the daily work of every software development team. Programming isn’t just about writing code — it’s a perpetual journey of learning and adapting. As teams deepen their understanding of the problem domain and encounter evolving requirements, the code needs to evolve accordingly. A conscious effort to perpetually refine, simplify, and clarify the code ensures it remains robust and adaptable, mirroring the knowledge of the team.</p><h3>TL;DR</h3><p>Technical debt isn’t just a developer’s concern — it’s a business concern — and a major one.</p><p>According to a <a href="https://www.it-cisq.org/the-cost-of-poor-software-quality-in-the-us-a-2020-report/">2020 report</a> from the Consortium for IT Software Quality (CISQ), the cost of poor software quality in the U.S. alone has ballooned to over $2 trillion every year. Much of this is a direct result of unmanaged technical debt.</p><p>Technical debt is a massive liability for your business, especially if your business is reliant on custom in-house or outsourced software.</p><p>It’s not merely about messy code or outdated libraries; it’s about the gradual erosion of reliability, security, agility, efficiency, and innovation. It’s a silent killer, often overshadowed by immediate business goals.</p><p>An engineering culture that values sustainable efficiency will simply be more productive over time than one that doesn’t. It’s also much less risky. Prioritizing continuous code refinement and clarity as a part of every sprint is a great start.</p><p>Organizations that fail to understand technical debt, refuse to take it seriously, or relegate it to something “only nerds care about,” are actively working against themselves. They might as well go punch themselves in the face.</p><p>Want to read more techniques for writing reliable software at scale with maximum developer efficiency? Follow me on Medium and read the article below:</p><p><a href="https://medium.com/failure-mode/do-these-10-things-to-improve-your-apps-reliability-fc114b16c2d1">Do These 10 Things to Improve Your App’s Reliability</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c668020b0e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Back to The Future II” Was Right: Modern Tech is a Buggy Mess]]></title>
            <link>https://medium.com/failure-mode/back-to-the-future-ii-was-right-modern-tech-is-a-buggy-mess-f2b96313148c?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2b96313148c</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Thu, 28 Sep 2023 10:47:15 GMT</pubDate>
            <atom:updated>2023-10-07T06:36:33.851Z</atom:updated>
            <content:encoded><![CDATA[<h4>Has reliability taken a back seat to features? If so, why? And what are the consequences?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltv_WIdtErdgRm1CTXrZ_A.jpeg" /><figcaption>Credit: Alexey Demidov on <a href="https://www.pexels.com/photo/glitched-photo-of-street-in-city-10596388/">Pexels</a></figcaption></figure><p>The digital age, with all its promise, has veered off course. Amid a backdrop of impressive technological advancements, there’s an unsettling undertone of unreliability. We spend vast sums on our tech, yet glitches, hiccups, and outright failures remain all too common. But why, in an era of seemingly limitless innovation, do we find ourselves mired in such mediocrity?</p><p>Why is our future looking less like the utopian future of <em>Star Trek </em>and more like the one depicted in <em>Back To The Future II</em>, albeit less funny and with <a href="https://www.caranddriver.com/news/a44185487/report-tesla-autopilot-crashes-since-2019/">more serious consequences</a>?<em> </em>And why does nobody seem to care?</p><p>In 1989, Robert Zemeckis’s <em>Back to the Future II</em> painted a vivid picture of the future with hovering skateboards, self-tying shoes, and flying cars. You may have missed it, but amidst the glitz and glamour, the future was also riddled with technological bugs. From the glitchy servers at the ‘80s-themed cafe to the malfunctioning TVs, kitchen appliances, and human implants, Zemeckis’s view of the future was prophetic in ways few talk about. <em>Go ahead, re-watch the movie, and count all the glitches. I know you want to.</em></p><p>Ironically, one of the film’s most dependable technologies is also among the oldest: the fax machine. Indeed, even today, fax remains a <a href="https://www.computerworld.com/article/3697270/the-fax-is-still-king-in-healthcare-and-its-not-going-away-anytime-soon.html">trusted method to transmit crucial documents</a> globally, notably in healthcare. And it’s worth noting that even during its 1980s peak, modern fax wasn’t exactly cutting-edge, having been developed by Xerox in the 60s.</p><p>So what is causing these issues? In the era of AI, why can’t we build anything that works?</p><p>It’s usually not the silicon. Modern computer chips, also known as Integrated Circuits, have achieved an impressive level of reliability after decades of iterative improvement and billions of dollars of investment by companies like Intel, TSMC, and ASML. The real culprit behind today’s tech glitches often isn’t the hardware, but the software that runs on it.</p><p>According to a <a href="https://www.it-cisq.org/the-cost-of-poor-software-quality-in-the-us-a-2020-report/">2020 report</a> from the Consortium for IT Software Quality (CISQ), the cost of poor software quality in the U.S. alone has ballooned to over $2 trillion annually. This staggering figure encompasses legacy system problems, unsuccessful IT projects, and operational software failures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bx0YSgwmBpJZu1SpDPhwxw.png" /><figcaption>Credit: <a href="https://xkcd.com/2030/">xkcd</a></figcaption></figure><h3>Software bugs in space</h3><p>From 1969–1972, when computers took up entire rooms and code was hand-crafted from ones and zeros (and <a href="https://www.righto.com/2019/07/software-woven-into-wire-core-rope-and.html">literally hand-sewn into memory</a>), we successfully sent twelve men to walk on the moon. But in 2023, we struggle to even send a probe there. That’s not to say sending stuff to the moon is easy.</p><p>Space is the most unforgiving environment imaginable. Between the radiation, vacuum, limited power, communication delays, and the high vibrations of launch, it’s the ultimate engineering challenge. Add astronomically expensive development and launch costs, and space becomes a highly risky venture.</p><p>So it’s nothing short of tragic when a software bug takes down a $90 million lunar mission and more than half a decade of development.</p><p>That’s what happened to Japan’s ispace’s Hakuto-R spacecraft when it<a href="https://qz.com/emails/space-business/1850377727/space-business-hard-landing#:~:text=What%20does%20a%20failed%20Moon,to%20land%20on%20the%20Moon."> crash-landed on the moon earlier this year</a>, knocking off $200 million of the company’s stock in one morning. The spacecraft’s software got confused by a crater on the moon, causing it to think it was closer to the ground than it was, and free-fell over 3 miles. <em>Lines of code</em> took down millions of dollars worth of highly specialized aerospace hardware, years of mission planning, and scientific opportunity.</p><p>The privately funded company tried to save a penny on software testing and it cost them $200MM, a black eye, and half a decade of work. They sent a state-of-the-art spacecraft to the moon with software that didn’t know what craters were.</p><p>Unbelievably, this isn’t the only example of relatively simple software defects taking down mega-million-dollar space missions. In 2006, NASA lost the Mars Global Surveyor ($210 million) due to a <a href="https://www.nasa.gov/mission_pages/mgs/mgs-20070413.html">memory allocation issue</a> after a software update. There’s also NASA’s Mars Climate Orbiter (1998, $327.6 million), Ariane 5 (1996, over $500 million), Phobos-Grunt (2011, $120–160 million), and others.</p><p>Those are staggering losses. Imagine what scientific and commercial opportunities there could have been if it were not for some bad code.</p><h3>Software bugs on Earth</h3><p>As of this writing (September 2023), several major online platforms are experiencing significant outages. Facebook, according to its own site, has been grappling with issues for days. <a href="https://downdetector.com/">DownDetector</a> also indicates that Microsoft 365, Venmo, and Instacart are facing similar disruptions.</p><p>For better or for worse, platforms such as these have become integral to our modern society. It’s how we connect, communicate, and do business. They have also become intertwined in almost all aspects of our daily lives, from workout machines to entertainment.</p><p>Beyond inconveniencing users and being disruptive and costly for businesses, software failures can have even graver consequences. Despite Tesla’s commendable progress in autonomous driving technology, its “Autopilot” system has been implicated in 736 crashes leading to 17 fatalities, as reported by the Washington Post. One tragic incident saw a Tesla in Autopilot mode striking a teenager at 45 mph after breezing past a stationary school bus.</p><p>Then there’s the Boeing 737 MAX. The aircraft’s Maneuvering Characteristics Augmentation System (MCAS), a software-based solution, was implicated in two catastrophic crashes in Indonesia and Ethiopia that claimed the lives of 346 people.</p><p>In an era where software’s reach is ever-expanding, its integration into vital sectors like medicine, self-driving cars, and aviation raises the stakes. Modern hospitals rely heavily on software-driven equipment, from patient monitoring systems to robotic surgery aids. Any software glitch in these “safety-critical” systems can be devastating.</p><h3>Why today’s software sucks</h3><p>While new forces shape today’s landscape, the software industry’s quality issues have deep roots in the past. In fact, these challenges became so pronounced in the 1960s that a term was coined to describe it: “The Software Crisis.”</p><p>As computers grew more capable, software became exponentially more complicated. Yet the tools and techniques needed to ensure code quality at this larger scale lagged behind. Managers were frustrated and confused as to why software projects always ran late and had so many issues.</p><p>In 1972, during his Turing Award acceptance speech — the highest honor in Computer Science — Edsger Dijkstra observed:</p><blockquote>“Society is getting dissatisfied with the performance of programmers and of their products. … Software seems to be different from many other products, where as a rule, a higher quality implies a higher price. Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result, the programming process will become cheaper.”</blockquote><p>Unfortunately, Dijkstra’s words seem to have fallen on deaf ears. This idea of “code quality” being the means by which we can mitigate costs, meet deadlines, and ensure the reliability of our software products remains overlooked and misunderstood by many.</p><p>When the primary focus is on rapid delivery or short-term gains, software engineering principles are often thrown out the window. This oversight has a domino effect. Not only does it increase the probability of bugs, but it also makes them costlier to fix down the road. Counter-intuitively, and ironically, this lack of quality has the opposite effect of what many intend. A lack of quality is likely to make development both more expensive and slower, as Djikstra observed. This stems from the accumulating technical debt and time wasted chasing down bugs.</p><p>The deeper issue, however, lies in the prevailing attitude within much of the industry. Many businesses may not truly appreciate “Software Engineering” as an engineering discipline, akin to Mechanical, Chemical, or Aerospace Engineering. This cultural oversight is not helped by the “move fast and break things” mantra, popularized by giants like Facebook. As a result, software teams often find themselves marginalized, lacking the recognition and resources they require.</p><p>However, the challenges don’t stop there. The tech sector’s rapid growth and relentless demand have outstripped our capacity to cultivate disciplined professionals. Despite their theoretical underpinnings, traditional computer science degrees often miss delivering pragmatic skills — it’s like having an electrical engineer versed only in theoretical physics. Bootcamps, on the other hand, tend to produce individuals proficient in mimicry, often relying on internet copy-pasting, rather than genuine comprehension and problem-solving. Ageism also plays a key role. The result? A booming industry facing a drought of well-rounded, pragmatically-trained, disciplined, and experienced talent.</p><p>Exacerbating issues even further, the tech sector’s hiring process is often out of sync with the real-world demands of the job. The brainteasers many professionals endure are hardly indicative of on-the-job success. It’s worth noting that, unlike other engineering fields, only Software Engineers are likely to face such abstract challenges during their interviews. Moreover, there’s an ageist bias that tends to undervalue experience, favoring younger candidates at the expense of seasoned professionals who bring a wealth of knowledge to the table. The emphasis on youth often disregards the wisdom that years of hands-on experience can impart.</p><p>Alongside ageism, there’s an undue emphasis on algorithmic problems despite their infrequent appearance in most developers’ day-to-day work. Such problems can be tackled with industry-standard libraries or online references when they do arise. This skewed focus misses evaluating true engineering skills. Additionally, high-pressure interview environments with unrealistic problems can be alienating for neurodiverse individuals, including those with ADHD, who might shine in practical coding but struggle in such settings. This grueling process not only detracts from finding the right job fit but can also pressure developers into settling for ill-suited roles, perpetuating industry inefficiencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*VY0aoVcR2V0PsLRcsBtrJA.jpeg" /><figcaption>Credits: Created by <a href="https://twitter.com/deniswsrosa">@deniswsrosa</a> via <a href="https://twitter.com/thebadprogramm3/status/1237101234571616258/photo/1">Twitter</a></figcaption></figure><h3>Our future hasn’t been written yet</h3><p>New technology is worthless if it’s not reliable. We’re in a ‘software crisis’, and it’s not just about missed deadlines. Lives are on the line. Spacecraft, meant for the stars, end up as expensive junk. Self-driving cars are slamming into kids. If this doesn’t scream crisis, what does?</p><p>It’s not just about producing better code. We need standardization, active involvement from professional bodies, and perhaps a thoughtful consideration of regulation, akin to safety standards in other critical industries like aviation or healthcare. Most crucially, there’s a complete overhaul needed in how we teach software. We must transform the hiring culture in tech, valuing hands-on experience and moving away from a glorified IQ-measuring contest. It’s about time we change the conversation, recognizing the importance of the Software Engineering discipline in our increasingly digital world.</p><p>We’re at a crossroads. It’s high time we get our act together and raise the bar, not just for the sake of technology, but for the very future we’re trying to build.</p><p><strong>Have any tech fails or aggravating software bugs you have recently encountered? Share them in the comments!</strong></p><p>Looking to help make your company’s software more reliable? <a href="https://linkedin.com/in/mhweiner">I’m open to work!</a> 👋 I specialize in mission-critical applications, having worked on such projects in healthcare, cloud computing, e-commerce, and aerospace for over a decade.</p><p>You can also read my other articles on software reliability and follow me! Let’s work together to make technology safer, less frustrating, and more useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2b96313148c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/failure-mode/back-to-the-future-ii-was-right-modern-tech-is-a-buggy-mess-f2b96313148c">“Back to The Future II” Was Right: Modern Tech is a Buggy Mess</a> was originally published in <a href="https://medium.com/failure-mode">Failure Mode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying NASA Techniques To Mission-Critical Software]]></title>
            <link>https://medium.com/failure-mode/applying-nasa-techniques-to-mission-critical-software-8f2815f4a674?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f2815f4a674</guid>
            <category><![CDATA[nasa]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[aerospace]]></category>
            <category><![CDATA[software-reliability]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Wed, 26 Oct 2022 17:25:18 GMT</pubDate>
            <atom:updated>2023-08-29T16:06:32.518Z</atom:updated>
            <content:encoded><![CDATA[<h4>Systems Engineering principles from Apollo can be applied to modern software to prevent costly failure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PXAs5pGOw-3kX1RQ" /><figcaption>Credit: <a href="https://unsplash.com/@markkoenig?utm_source=medium&amp;utm_medium=referral">Mark König</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>On a steamy Wednesday morning in Florida, three men sat down to steak and eggs while a crew cautiously monitored three-quarters of a million gallons of rocket fuel that was loaded onto their spaceship the evening prior. Shortly, these men would be driven by bus to this rocket, sixty feet taller than the Statue of Liberty, which would hurl them out of Earth’s orbit faster than a bullet, land them on the <em>friggin’ moon,</em> and then hopefully, return them safely back to Earth, for the very first time.</p><p>It’s hard to describe just how ambitious, experimental, and dangerous the Apollo program was. It might have been, as described in <em>The Right Stuff (1983)</em>, “the greatest death-defying stunt ever broadcast.”</p><p>Apollo depended on many technologies, materials, techniques, and mathematics that hadn’t existed just a few years prior. The Apollo Guidance Computer (AGC), a safety-critical component, was one of the first computers to use Integrated Circuits. In 1963, the Apollo program accounted for 60% of the United States IC production.¹</p><p>However, for all of the breakthroughs in technology, some believe the most important accomplishments were actually in Systems Engineering and risk management. In fact, they may have been the key difference between the United States’ highly successful space program and the failed Soviet program.²</p><p>In a July 1969 issue of <a href="https://www.fortune.com/">Fortune</a>, Tom Alexander wrote:³</p><blockquote>The really significant fallout from the strains, traumas, and endless experimentation of Project Apollo has been of a sociological nature rather than a technological nature: techniques for directing the massed endeavors of scores of thousands of minds … <strong>This is potentially the most powerful tool in man’s history</strong>.</blockquote><p>These techniques, which served Apollo so well, can be applied to modern software projects. Indeed, many in the aerospace sector already do just that. Unfortunately, it seems many others take a whack-a-mole approach to software reliability when cost-effective, preventative measures exist.</p><p>Systems Engineering isn’t just about reducing the risk of failure. It’s also about maximizing team efficiency while ensuring that the actual real-world problem is being solved, avoiding wasteful work.</p><p>You might notice that many of the following principles have overlap and circular dependencies. That’s a hallmark of Systems Engineering — nothing exists in a vacuum. It’s all about the bigger picture and how everything is connected in the real world.</p><h3>Test Like You Fly, Fly Like You Test (TLYF/FLYT)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JblJJ7ul2QmJE0d7" /><figcaption>Credit: <a href="https://unsplash.com/@simonfitall?utm_source=medium&amp;utm_medium=referral">Simon Fitall</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>While Apollo 11 was the first to land on the moon, it wasn’t the first to fly the Lunar Module (LM) in lunar orbit. That was done on Apollo 10, which brought the LM to within 10 miles of the lunar surface. The Apollo 10 crew wasn’t the first to orbit the moon or perform the Trans-Lunar Injection procedures to get there. That was first done by Apollo 8. They also weren’t the first to fly the LM. That was demonstrated on Apollo 9. Apollo 9’s docking and spacewalk (in case of docking failure) built upon the work done during the Gemini program. And so on.</p><p>All of the above are examples of the philosophy, “Fly Like You Test.” This means an incremental, iterative approach to unproven systems or maneuvers. The goal is to minimize risk (and waste) by only testing one or two things at a time, and doing this more quickly and more often. It’s all about learning and then applying that knowledge. In modern web-based software, some examples of the FLYT philosophy are soft releases, blue/green deployments, and small, iterative development cycles.</p><p>Continuous Integration/Continuous Deployment (CI/CD) is a method for making software releases as small as possible. Instead of one big, infrequent deployment, teams that practice CI/CD release smaller builds more frequently. There are many benefits, such as less code to test, a smaller surface area for risk, and ease of peer review. Another huge benefit is that code is more frequently integrated with other teammates&#39; changes, which could be an overlooked source of bugs and friction. Of course, CI/CD doesn’t come without its challenges, but many teams who practice CI/CD find the benefits outway the costs.</p><p>In short, if you can de-risk by running an experiment, collecting data, or building something smaller in scope first, you probably should.</p><p>A corollary, “Test Like You Fly” prescribes ground testing under the most realistic “flight conditions” possible. In software, TLYF is all about testing with realistic inputs, under realistic conditions, and on production-like hardware. One of my favorite TLYF examples is testing consumer web apps on older, low-end hardware and a simulated low-bandwidth internet connection instead of your $4k company-issued Macbook Pro and high-speed fiber-optic internet.</p><h3>Fail-Fast</h3><p>I interpret Fail-Fast in two ways:</p><ol><li><strong><em>To quickly invalidate fruitless ideas</em></strong></li></ol><p>This does <strong><em>not</em></strong> mean throwing a bunch of stuff at the wall and seeing what sticks. It does <strong><em>not</em></strong> mean you don’t have to do thoughtful design and validation. It simply suggests that we do the least amount of work to prove or disprove an idea, again, following the FLYT philosophy. Often, the real challenge will be to come up with a cost-optimized test that’s realistic enough, following the TYLF philosophy.</p><p><strong><em>2. To maximize the chances that you will catch problems during testing</em></strong></p><p>While taking flying lessons, an instructor was a little vigorous when testing the wing struts during our pre-flight check. When I asked whether or not that was good for the airplane, he answered, “if it’s going to break, I’d rather it be on the ground.”</p><p>You want your software to fail while it’s still “on the ground.”</p><p>There are a variety of ways to ensure mistakes are caught early on in the development process. That is one of the goals of <em>Defensive Engineering</em>.</p><h3>Defensive Engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5HHSgPOAaHBUf8Da" /><figcaption>Photo by <a href="https://unsplash.com/@samuele_piccarini?utm_source=medium&amp;utm_medium=referral">Samuele Errico Piccarini</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You’ve probably heard of the term “defensive driving.” Although it might sadly be a rare affair on American roads, the concept is to put yourself in a good position should a problem occur. One of the most important tenants of defensive driving is a safe following distance — giving yourself ample time and space for you to react and for your brakes to bring your vehicle to a stop before slamming into the car in front of you.</p><p>Defensive Engineering is similar. The goal is to minimize the odds of failure by design. There are many ways to do this. Minimizing failure points, redundancy, safety margins, and avoiding common mistakes through best practices are just a few. Perhaps most importantly, it’s designed to be testable — ensuring problems are found early on.</p><blockquote>“Reliability has to be designed into an object; it cannot be built into a badly engineered design.” — George E. Meuller, Director of Manned Spaceflight (1963–1969)</blockquote><p>One of the most effective ways to improve reliability in hardware is to keep things simple. An excellent example of this is the LM’s ascent stage rocket engine. It utilized hypergolic fuels that combust instantly when in contact with each other, with almost no moving parts.² There was no backup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*oVtshbnDKvvI2d5lwHVNNQ.jpeg" /><figcaption>Margaret Hamilton, team lead of MIT Instrumentation Lab (now Draper Lab), standing next to a complete printout of the AGC source code. Credit: Courtesy MIT Museum</figcaption></figure><p>Margaret Hamilton led the team that wrote the software for the AGC. She is credited for coining the term “Software Engineering” during the project, in an attempt to teach people that writing software was similar to designing hardware. I think we need to learn this lesson again.</p><p>In software, simplicity is even more important than it is in hardware, as it does not scale as hardware does — it has almost no physical bounds. Instead, the limiting factor is how much of it can fit neatly in one person’s head. The easier it is to understand, the fewer mistakes are made. In my experience, this is where teams leave the most money/reliability/efficiency on the table.</p><blockquote>“There is nothing a mere scientist can say that will stand against the flood of a hundred million dollars. But there is one quality that cannot be purchased in this way — and that is reliability. The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay.” — C. A. R. Hoare</blockquote><p>Besides simplicity, there are other defensive tools and techniques available to help ensure sanity and correctness in software:</p><p>Static analysis (i.e., dead code detection), statically-typed programming languages, and unit tests are all excellent defensive Software Engineering tools. These will catch potential issues before the code is even run, saving time debugging. <em>Rust</em>, a new programming language, offers guarantees about memory ownership and race conditions. Another common defensive technique is using guard clauses around functions, as a sanity check for correct/assumed conditions.</p><p>In aviation, there is a standard, DO-178, which specifies the requirements for software that runs on safety-critical airborne systems, such as avionics. Much of it is simply a list of best practices that apply to any software project.</p><p>Again, perhaps most importantly, software must be designed to be testable and observable. End-to-end integration tests and load tests may be widely adopted, but unit tests are often misunderstood and not properly implemented. Part of the problem is that in order to write a proper (and cost-effective) unit test, the code must be unit testable — that is, it must be easy to isolate it from the rest of the code and simulate its inputs and compare its expected behavior to its actual behavior.</p><p>Another huge item of importance is observability and error handling. Warnings and errors must be loud, or they will not be noticed. They also must be part of an immediate feedback loop, and not simply dumped into a log. People usually only look at these logs after a problem is detected; but the true value of observability in software should be of a preventative nature, not a reactive one, especially during the development process.</p><h3>“What If” Thinking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3nI5RKIrgc47o7QF" /><figcaption>Credit: <a href="https://unsplash.com/@dragos126?utm_source=medium&amp;utm_medium=referral">Dragos Gontariu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Andrew Chaikin, an award-winning space historian, believes that one of the most important activities directly contributing to Apollo’s success was something he called “What If Thinking.”² I’ll break this up into two categories, <em>fault prevention,</em> and <em>fault tolerance</em>.</p><h4>Fault Prevention</h4><p>In order to prevent failure, engineers and mission planners come up with requirements, scenarios, and testing procedures. This is done through informed guesses, data-driven research, creative thinking, experimentation, and iteration. But for Apollo, it didn’t stop there.</p><p>Everyone kept asking themselves, “what if <em>this</em> were to happen?”</p><p>You might be surprised to learn that most of the ingenious workarounds used on the Apollo 13 rescue mission were actually thought up beforehand. This included using the LM as a lifeboat and the free-return trajectory. Even the celebrated fix for the carbon dioxide scrubbers was ideated during a simulation exercise.²</p><p>In software, however, things are even more interesting. It’s common for “normal,” but unexpected conditions to cause a fatal error. According to a 2014 paper, “almost all catastrophic failures are the result of incorrect handling of non-fatal errors explicitly signaled in software.”⁵ In other words, the underlying errors were <em>non-fatal, </em>but the program didn’t know what to do with them, so they <em>became</em> fatal.</p><p>This is why it’s so essential to fundamentally understand all of your software systems, and what state may occur. I have a few recommendations:</p><ol><li>Keep your software as simple as possible, avoiding heavy, opaque libraries. Minimize dependencies.</li><li>Methodically go through all of your code branches and 3rd party RPC calls (ie, HTTP services), to understand how they might respond in non-ideal scenarios. Unit testing, in combination with branch coverage tools, is a good start.</li><li>Employ proper error-handling techniques and a closed-loop incident management system. I wrote about this in <a href="https://mhweiner.medium.com/do-these-10-things-to-improve-your-apps-reliability-fc114b16c2d1">Do These 10 Things to Improve Your App’s Reliability</a>.</li><li>Ask yourself, “what if?”</li></ol><h4>Fault Tolerance</h4><p>When dealing with complex systems, no matter how thorough your testing is, you should still expect surprises. Margaret Hamilton’s team saw this first-hand during Apollo 11&#39;s landing attempt. I’m talking about the infamous “1201/1202” computer alarms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sY4-_xZjRDt-D8_raXGxaw.jpeg" /><figcaption>Summary of AGC alarms. Credit: <a href="https://www.flickr.com/photos/erikcharlton/">Erik Charlton</a>, under <a href="https://creativecommons.org/licenses/by/2.0/">CC license</a>.</figcaption></figure><p>During the landing, a hardware fault caused the computer to become overloaded, resulting in repeating computer alarms, that almost resulted in an abort! Thankfully, the AGC sported a revolutionary kind of operating system to handle this very situation.</p><p>On paper, the mission should have failed (and it came really close). These alarms were theoretically impossible. The AGC was a highly specialized flight computer custom-tailored to each mission. It was designed to never become overloaded. However, the engineers knew that if for some reason, the computer did become overloaded at the wrong time, it could result in the death of the astronauts or the failure of the mission. To reduce the severity of such a failure, the AGC’s operating system had two key features: task prioritization and automatic fault recovery.</p><p>The system could drop lower-priority jobs in favor of higher-priority ones. A higher priority job might be something like keeping the Lunar Module upright during landing, while a lower priority job could be updating the display. In fact, that’s exactly what it did during the Apollo 11 landing.</p><p>The MIT team’s foresight, humility, and willingness to expect the unexpected saved not only the Apollo 11 moon landing but perhaps even the crew. As NASA Flight Director Tommy Holloway famously said, “you’re not as smart as you think you are.”</p><p>Failure is inevitable. It’s how we decide to handle these failures that make the difference. A formal way of studying <em>how</em> a component might fail, and the effect on the downstream system, is called <em>Failure Mode and Effect Analysis (</em>FMEA). As a Software Engineer, you often have options. Most errors are non-fatal, and it’s up to you how you decide to handle them. Controlled degradation is always preferable to critical failure.</p><h3>Systems Thinking &amp; Leadership</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qpG_-1w0DkvQEnRDJEq2hw.jpeg" /><figcaption>George Mueller briefs President John F. Kennedy and other officials on the Saturn V rocket. Front row, left to right: George Low, Kurt Debus, Robert Seamans, James Webb, President Kennedy, Hugh Dryden, Wernher von Braun, General Leighton Davis, and Senator George Smathers. <a href="https://www.nasa.gov/feature/remembering-george-mueller-leader-of-early-human-spaceflight">Credit: NASA</a></figcaption></figure><p>As challenging a field as rocket science is, people management might have been the biggest challenge of the entire Apollo program. People are highly complex, more complex than the Saturn V rockets that took men to the moon.</p><p>Systems Engineering is the study and design of systems; not just technical systems, but people systems, too. Wasteful work and poor communication greatly impact reliability and the bottom line. Generally, cracks form around organizational lines. Integrations fail and requirements degrade as the “game of telephone” continues.</p><p>George E. Mueller (pronounced “Miller”) took over NASA’s Office of Manned Spaceflight in 1963 at the request of Administrator James Webb (for whom the Webb telescope is named). Mueller was an Electrical Engineer who later became a Systems Engineer, working on ICBMs in collaboration with the Air Force and NASA at Ramo-Wooldridge Corporation.</p><p>In his book “Doing the Impossible,” Arthur L. Slotkin, who studied under Mueller, wrote:</p><blockquote>Mueller had the ability to analyze and understand systems … In particular, he could visualize a total system — hardware, software, people and processes — and everything necessary to accomplish the task at hand.⁴</blockquote><p>Mueller saw a program that would never make JFK’s deadline of 1970, be within budget, or be reliable enough. In fact, this was the conclusion of NASA’s own internal assessments at the time. Mueller later recalled that “milestones existed but were not believed” and that the project ran “open loop.”⁴</p><p>Mueller reorganized NASA and established several practices and a culture that without, Slotkin argues, we would not have achieved the landing by Kennedy’s deadline.</p><p>In his reorganization of NASA, Mueller cut through bureaucracy, allowing engineers to talk directly to their counterparts at different centers and with engineers at headquarters, instead of having to play “telephone” through many levels of bosses. One might say that NASA employed something resembling highly distributed and scaled cross-functional teams.</p><p>This reorganization also gave Mueller direct insight into the technical problems at the ground level and direct communication with the engineers, which in turn, gave him a strong hand in understanding and managing the system as a whole.</p><p>While very few projects have the scale of Apollo (it enjoyed about 5% of the national budget at the time), I believe it demonstrates the importance that direct communication, close collaboration, and overall systems design have on engineering projects — and that it is possible to achieve this no matter the scale.</p><ol><li>“A Computer Built from NOR Gates: Inside the Apollo Guidance Computer.” n.d. Righto.com. Accessed October 25, 2022. <a href="http://www.righto.com/2019/09/a-computer-built-from-nor-gates-inside.html.">http://www.righto.com/2019/09/a-computer-built-from-nor-gates-inside.html.</a></li><li>Appel, NASA. 2012. “Management Lessons of the Moon Program (Andrew Chaikin).” Youtube. November 30, 2012. <a href="https://www.youtube.com/watch?v=RaskWhy5pYE.">https://www.youtube.com/watch?v=RaskWhy5pYE.</a></li><li>Alexander, Tom. 1969. “The Unexpected Payoff of Project Apollo.” <em>Fortune</em>, July 1969.</li><li>Slotkin, Arthur L. 2012. <em>Doing the Impossible: George E. Mueller and the Management of NASA’s Human Spaceflight Program</em>. 2012th ed. New York, NY: Springer.</li><li>Ding Yuan, et. al. 2014. “Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems.”<em> </em>University of Toronto, 2014, <a href="https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf">www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f2815f4a674" width="1" height="1" alt=""><hr><p><a href="https://medium.com/failure-mode/applying-nasa-techniques-to-mission-critical-software-8f2815f4a674">Applying NASA Techniques To Mission-Critical Software</a> was originally published in <a href="https://medium.com/failure-mode">Failure Mode</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Tech Leaders Sabotage Software Developers And What They Should Do Instead]]></title>
            <link>https://medium.com/codex/how-tech-leaders-sabotage-software-developers-and-what-they-should-do-instead-898855293235?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/898855293235</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[management-and-leadership]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 21:20:13 GMT</pubDate>
            <atom:updated>2024-12-13T20:42:07.410Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Knowing where to cut and where to invest is the key difference in successful projects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJFxJO6LQ9dPn3r0OGqpaQ.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/niap3d-529886/">Niap3d</a> on <a href="https://pixabay.com/">Pixabay</a></figcaption></figure><p>Let me tell you a story. Perhaps you’ve heard this one before.</p><p>A startup in the wedding industry wants to build a new online registry for the Spring wedding season in a short, six-month timeline.</p><p>They bring in some experienced programmers, who bring in all of the latest and greatest technologies: TypeScript, React, Redux, TypeORM, MongoDB, GraphQL, Microservices, and many more. They spend the first couple of months setting everything up, discussing the overall architecture, and building out the services as rapidly as possible.</p><p>Things go well for the first few months. However, halfway through, they realize they won’t make their deadline. Development has slowed considerably, and there’s just too much to do. They go on a hiring spree, quickly growing their team. Engineers spend time interviewing and discussing candidates. Once hired, these new folks are slow to onboard and take up people’s time, slowing development even further. People start working late nights and weekends. More shortcuts are taken.</p><p>The deadline is now a week away. Although a lot of code has been written, and most of the functionality is there, it’s missing a few key features and has poor reliability and performance. Users churn out, and employees are burnt out.</p><p>This team had plenty of resources, time, and expertise to build a reliable, successful product by their deadline. So what happened?</p><h3>Knowing Where To Tap</h3><blockquote>“We’re a startup with a tight deadline.” — Anonymous</blockquote><p>You may have heard the parable, “<a href="https://medium.com/@oceanbcreative/the-ship-repair-man-story-dd959a4469d8">The Old Engineer and the Hammer</a>.” Knowing where to invest — and where to cut — is paramount. Unfortunately, many managers and lead software engineers are chasing trends instead of executing an informed strategy.</p><p>In the case of the wedding startup, which is based on a true story, they were lacking in a few key areas.</p><p>They made large investments into popular sophisticated tools but lacked investment in the fundamentals. Simple things like logging, error handling, and configuration lacked structure, tooling, or standardization. There were no <a href="https://lawrey.medium.com/unit-tests-ui-tests-integration-tests-end-to-end-tests-c0d98e0218a6">unit tests</a> (only some integration tests masquerading as unit tests). Their documentation was outdated and confusing. Worst of all, the codebase was far too complicated and difficult to understand for an MVP.</p><p>These “small” everyday things might be a lot less sexy and harder to sell to management than <em>Framework X</em>, but they are much more important. I’ve built successful apps without GraphQL, but not one without a comprehensive error-handling strategy.</p><p>When approached about some of these deficiencies, the CTO’s response was “Yes, I’ve noticed that too… but we’re not Microsoft. We’re a startup with a tight deadline.”</p><p>Sure, there are some corners that can be cut when you are small. Perhaps you don’t need as stringent security rules. You probably don’t need complex scaling tools like GraphQL or Terraform. You may not need to worry about database query optimization or replication. Maybe you don’t need as many caching layers. Building software to support a million users when you’re expecting to launch with 1,000 would be a waste of time.</p><p>However, software gets complicated very quickly. It’s just a bunch of statements of logic and instructions — not a physical thing. It’s a group thinking exercise.</p><p>As code grows more complex, it becomes more difficult to read, understand, observe, debug, and change. When this happens, velocity falls off a cliff. The risk of unplanned work increases exponentially. At some point, feedback loops push the codebase over the edge of the <a href="https://betterprogramming.pub/when-to-throw-in-the-towel-on-your-technical-debt-b1f6bbb3aa0c">software event horizon</a>. Change is now very difficult and completely unpredictable. Reliability is impossible. You cannot recover from this except to start over.</p><p>As a leader, cutting corners here is the worst decision you can make. Yet, I see many leaders make this exact same decision, putting their developers in a tough, unenviable situation. Even if things never get that bad, they are leaving tons of money (efficiency) and reliability on the table.</p><p>Most startups cannot afford the unexpected delays, outages, and lack of agility that this kind of development brings. This startup was, in fact, not Microsoft, as the CTO was keen to point out. Microsoft could get away with this sort of thing. They did not, and folded soon after.</p><h3>The Quality vs. Speed Fallacy</h3><blockquote>“If debugging is the process of removing bugs, then programming must be the process of putting them in.” — Edsger Dijkstra</blockquote><p>Another very similar mistake I see people make is attempting to trade quality for speed, or otherwise a lack of emphasis on code quality.</p><p>There’s a lot of talk about code quality. But what is it? One definition:</p><p><em>Quality software works as expected, is developed within a reasonable timeframe, and is reasonably easy to maintain or modify.</em></p><p>In fact, the “<a href="https://en.wikipedia.org/wiki/Software_crisis">Software Crisis</a>” of the early days of programming was described in similar terms. Back then, increasingly powerful computers resulted in increasingly large and feature-rich programs. The same tools and techniques used for smaller programs didn’t scale well to these larger, more complex programs.</p><p>To be clear, this didn’t mean that the old techniques didn’t have the required performance or functionality. On the contrary, early computers were so limited that programmers were forced to come up with the most efficient programs possible — in many cases, more efficient than software written today. However, to achieve this, they had been making a tradeoff: performance for readability. To be fair, programmers were also limited by languages and tools of the time, which had made the same design choices.</p><p>The Software Crisis wasn’t due to a lack of technical talent, but a deficiency in the ability to write correct, readable, understandable, and “safe” code at scale. This meant that projects were notoriously late, over budget, and unreliable. Some even resulted in the <a href="https://en.wikipedia.org/wiki/Therac-25">loss of life</a>.</p><p>This was such a big deal that it threatened the entire burgeoning software industry. In 1972, during his Turing Award <a href="https://www.cs.utexas.edu/users/EWD/ewd03xx/EWD340.PDF">acceptance speech</a>, Edsger Dijkstra described his take on the problem:</p><blockquote>“Society is getting dissatisfied with the performance of programmers and of their products. … <strong>Software seems to be different from many other products, where as a rule a higher quality implies a higher price.</strong> Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging.”</blockquote><p>The answer is both surprisingly simple yet challenging in its own right: make programming easier and harder to get wrong. This might be an old concept, but it remains as relevant as ever in software today.</p><p>This desire for programming safety at scale is exactly what the discipline of Software Engineering was invented to solve. Statically typed languages, compile-time error checking, and unit testing are just a few examples of ways to reduce the chances of mistakes. The most important strategy, however, is simplicity and human understandability.</p><p>Quite interestingly, and most impressively, this problem was predicted by Alan Turing in the 1940s:</p><blockquote>“It will be seen that the possibilities as to what one may do are immense. One of our difficulties will be the maintainence of an appropriate discipline, so that we do not lose track of what we are doing. We shall need a number of efficient librarian types to keep us in order.” — Alan Turing, 1947</blockquote><p>When we talk about code quality, at a high level, we are talking about code that is defensive, correct, performant, simple, understandable, well-composed, testable, and has good coverage. It is written utilizing the strategies of software engineering and defensive programming.</p><p>When software managers ask the question of quality vs. speed, they are asking a fundamentally flawed question.</p><p><strong>The better the code quality, the better the team’s efficiency. With poor code quality, you are working against yourself!</strong></p><h3>The Pit of Success</h3><blockquote>“The computing scientist’s main challenge is not to get confused by the complexities of his own making.” — Edsger Dijkstra</blockquote><p>Successful software projects are predicated on us understanding what we are building.<em> Is the code correct? Can we change it easily? Are we building the right thing?</em> As a leader, this understandability should be your top-most technical strategy. Everything you do should be in service to this goal. Thankfully, we have a lot more at our disposal than programmers did in years past, due to their hard work.</p><p>Before we get into that, however, I need to mention the importance of culture. It’s not enough to simply put best practices and tools in place. As a leader, you must also foster a <em>culture of quality</em>. Many software developers have been trained by previous managers that quality is relatively unimportant to them. Writing quality code requires a certain amount of trust that goes both ways. Some of the following are skills that must be learned through mentorship, guidance, self-study, and practice. It requires an investment. If your culture doesn’t place importance on these, then your engineers may not make these investments themselves.</p><p>Without further ado, below are a few tools, techniques, or areas that are a key part of my engineering strategy in service to human understandability and code quality.</p><h4>Simplicity</h4><blockquote>“There are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies, and the other is to make it so complicated that there are no obvious deficiencies.” — Sir Tony Hoare</blockquote><p>Simplicity is the most powerful tool in your toolkit and one of the most challenging. On the other hand, building something complex is easy. This is the job of a software engineer.</p><p>The simpler something is, the easier it is to understand, and the less that can go wrong. The easier something is to understand, the faster people will be — and the less wasteful debugging they will have to do, as Dijkstra observed.</p><p>Even the most poorly written function is relatively easy to figure out if it’s kept very small and does only one thing. In fact, this <em>encapsulation</em> can be a good strategy for dealing with situations where it makes sense to sacrifice readability for performance.</p><blockquote>“Compositionality is THE way to control complexity.”<br> — Brian Beckman</blockquote><h4>Readability and Narrative</h4><p>This might be the most difficult technique and also the most subjective. Just as a good writer should be able to paint a clear picture of the story, so must a good programmer.</p><blockquote>“The best programs are written so that computing machines can perform them quickly and so that human beings can understand them clearly. A programmer is ideally an essayist who works with traditional aesthetic and literary forms as well as mathematical concepts, to communicate the way that an algorithm works and to convince a reader that the results will be correct.”<br>― Donald E. Knuth</blockquote><p>One of the most important and difficult aspects of this is naming functions, variables, types, and the like. Good naming conventions can also help identify mistakes early on by “<a href="https://medium.com/@NoLongerSet/naming-conventions-matter-making-wrong-code-look-wrong-a7e95397a26c">making wrong code look wrong.</a>”</p><p><a href="https://en.wikipedia.org/wiki/Declarative_programming">Declarative Programming</a> is a style that facilitates understanding without having to do mental gymnastics and becoming a human compiler. Declarative style is more concerned with what the output should be given a certain input, whereas, in an Imperative style, the code is more focused on <em>how</em>. Reading imperative code is like reading a cooking recipe without knowing what you’re trying to make. Or like putting together a jigsaw puzzle without the box cover.</p><p>Some aspects of functional programming can also help improve readability by favoring “pure functions” and immutability over large, stateful objects. Such a style can help better identify mistakes in data transformations and make code much more easily testable. One of my favorite talks on the topic is <a href="https://www.youtube.com/watch?v=APUCMSPiNh4&amp;t=1126s">Refactoring to Immutability</a> by Kevlin Henney.</p><h4>Tooling</h4><p>When used appropriately, tools are indispensable to an efficient software team. The best tools prevent mistakes, enforce best practices, and automate time-intensive, repetitive tasks that are easy to get wrong. I’m talking about things like statically-typed languages, linting, and CI/CD release tools (like GitHub Actions and <a href="http://github.com/semantic-release/semantic-release">semantic-release</a>). Rust, a relatively new programming language, not only has a strong static type system but attempts to minimize race conditions and memory issues.</p><p>Linting tools should be used to limit function line counts, line lengths, and depth. This reduces complexity, nudges programmers towards composition and declarative techniques, and greatly improves code readability.</p><h4>Unit Testing</h4><p>One of the most underutilized and misunderstood techniques of software engineering is unit testing.</p><blockquote>“The real value of tests is not that they detect bugs in the code, but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code.” — Sir Tony Hoare</blockquote><p>One of the benefits of unit testing is that when there is a regression, it allows you to find the root cause faster since tests are more localized. Another is it makes it easier to explore all of the branches of your code when combined with a branch coverage tool.</p><p>By far, however, the biggest benefit is it forces the programmer to think more carefully about what the code should do. It also adds pressure to break larger chunks of code apart into smaller, better-defined, and more testable programs. The best unit tests are simple to read and act as a specification for the unit’s behavior.</p><p>Kevlin Henny has a great talk about this, <a href="https://www.youtube.com/watch?v=-WWIeXmm4ec">What We Talk About When We Talk About Unit Testing</a>.</p><h4>Configuration and Error Handling</h4><p>It might seem odd to have these together, or that these two things are not like the others I just mentioned. However, <a href="https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf">both are responsible for a disproportionate amount of bugs</a>. They are often the least tested part of your system and are difficult to cover with automated tests. They also tend to be the most neglected, as was the case with the wedding startup example earlier. Your configuration should be structured and typed, and be as logic-free as possible. I cover Error Handling in <a href="https://mhweiner.medium.com/do-these-10-things-to-improve-your-apps-reliability-fc114b16c2d1">Do These 10 Things To Improve Your App’s Reliability</a>.</p><h3>Wrap Up</h3><p>Don’t confuse quality with operational functionality. Quality is important no matter how many users you have or how small your team is. You can’t buy quality either.</p><p>If you are sacrificing quality for “speed,” you are sabotaging not only your developers but the entire project. Setting aside any external problems, if you are not able to deliver quality software by your deadline, then either you’re not really producing quality code, or you are simply not being realistic!</p><p>Imagine being a developer where you are discouraged from doing your best work. Now imagine being reprimanded because your code — which isn’t your best — doesn’t work the way it should, or is taking too long to work with. That would be frustrating, wouldn’t it? Imagine having to work long hours to correct it, only to run into the same problem again later. Would you want to stick around, or go somewhere you can be more productive and less stressed?</p><p>Leaders should set up their team for success by becoming a champion for quality code — it’s really the only way to produce software that works as expected, is developed within a reasonable timeframe, and is reasonably easy to maintain or modify. Go ask the inventors of programming themselves.</p><p>Want to read more techniques for writing reliable software at scale with maximum developer efficiency? Read the article below:</p><p><a href="https://mhweiner.medium.com/do-these-10-things-to-improve-your-apps-reliability-fc114b16c2d1">Do These 10 Things to Improve Your App’s Reliability</a></p><p><strong><em>Marc H. Weiner</em></strong> is a startup veteran with a special interest in safety-critical systems. He co-founded NextEMR, one of the first web-based Electronic Medical Record applications. He was also a lead engineer on <a href="https://www.koneksahealth.com/">clinical trial software</a>, <a href="https://marvel.com/">Marvel.com</a>, and infrastructure at <a href="https://www.shutterstock.com/">Shutterstock</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=898855293235" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/how-tech-leaders-sabotage-software-developers-and-what-they-should-do-instead-898855293235">How Tech Leaders Sabotage Software Developers And What They Should Do Instead</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I Learned Working for the Inventor of IBM’s Watson]]></title>
            <link>https://medium.com/better-programming/what-i-learned-as-a-developer-while-working-for-the-inventor-of-ibms-watson-7e76b4a2d4ba?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e76b4a2d4ba</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Thu, 12 Aug 2021 16:19:15 GMT</pubDate>
            <atom:updated>2024-05-24T03:03:58.299Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Sometimes being logical is more important than being right</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZgVAAMwZmD9N17plzb6yxw.jpeg" /><figcaption>Source: Wikimedia (CC BY-SA 4.0)</figcaption></figure><p>In 2018, I was hired by Dr. David Ferrucci, the inventor of <a href="https://en.wikipedia.org/wiki/Watson_(computer)">Watson</a>, IBM’s famous question-answering computer <a href="https://blog.ted.com/how-did-supercomputer-watson-beat-jeopardy-champion-ken-jennings-experts-discuss/">that beat champion Ken Jennings in “Jeopardy!”</a> in 2011.</p><p>After leaving IBM, Dr. Ferrucci started his own AI research company, <a href="https://ec.ai/">Elemental Cognition</a> with the goal of developing an AI that can understand the meaning behind language and give reasoning behind its answers as people do.</p><p>If EC is successful, the computer from Star Trek could finally become a reality. Aside from being a breakthrough in AI and disrupting a billion-dollar industry, such a computer could help doctors cure diseases or help engineers design solutions to fight global warming.</p><p>I ended up staying at EC for about a year, and wow, it was a challenge — not just intellectually, but emotionally. Although it turned out to not be a long-term fit, I did some genuinely interesting work and grew from the experience. Here are a few things I learned working for one of the most famous computer scientists in the world.</p><h3>Being logical is more important than being right</h3><p>As a leading expert on artificial intelligence, Dr. Ferrucci is obsessed with logic. I haven’t seen it, but he probably has a picture of Spock hanging on a wall somewhere. What I <em>have</em> seen hanging prominently on the wall in EC’s Wilton, CT office, is a poster titled “Thou Shall Not Commit Logical Fallacies.”</p><p>A <a href="https://yourlogicalfallacyis.com/">logical fallacy</a> is a flaw in reasoning. For example, a claim that a certain idea is either good or bad when the truth is usually more complicated and nuanced. This would be an example of “Black or White.” This is also known as a <em>false dilemma</em>.</p><p>We all make these kinds of mistakes sometimes, and some are quite easy to make — especially when our emotions get in the way. Although many make these innocently, others use them intentionally for nefarious purposes, to manipulate others. Some politicians or authoritarians do this to great effect.</p><p>Having started programming when I was a little kid, logic is important to me as well. I don’t have a picture of Spock on my wall, but I should <em>*scrambles to Amazon and gets distracted*. </em>However, despite my passion for logic and years of practice, I learned that I, too, had a bad habit of committing logical fallacies when put under pressure. In fact, this got me in trouble with Dr. Ferrucci.</p><p>I was often asked to explain the reasoning behind the decisions I made, even the ones that were widely accepted as an industry-standard best practice. For example, to explain why I chose <a href="https://reactjs.org/">React</a> over their home-grown <a href="https://jquery.com/">jQuery</a> framework, I had to not just give the pros and cons, but how the tools and methods caused these pros and cons. I had to reason using <a href="https://fs.blog/2018/04/first-principles/">first principles</a>, or irreducible truths.</p><p>I was capable of making that argument but didn’t. At least, not right away. First, I hadn’t made this particular argument in years and had forgotten how. jQuery is the Britney Spears of Javascript frameworks. Second, I was a little flustered that they would question something that seemed so obvious to me. We had a tight deadline and I felt time pressure to get stuff done. I also wanted to impress them. I just flailed my arms and said, “this is industry best practice and has been for years. This is how Facebook and Netflix do it.”</p><p>My logical fallacies were <a href="https://yourlogicalfallacyis.com/appeal-to-authority">Appeal to Authority</a> and <a href="https://yourlogicalfallacyis.com/bandwagon">Bandwagon</a>. Sure, the fact that something is widely accepted as a standard has merit. So does the fact that Facebook uses it. But that doesn’t mean it’s necessarily a good fit for our purposes. EC isn’t Facebook. Also, lots of things are popular that aren’t good ideas, like <a href="https://www.forbes.com/sites/stevedenning/2019/05/23/understanding-fake-agile/?sh=26fdc07a4bbe">corporate Agile</a> or <a href="https://korban.net/posts/postgres/2017-11-02-the-case-against-orms/">ORMs</a>.</p><p>Although I might have been right, my reasoning was lazy and flawed. I might have been frustrated or felt pressured, but that’s a part of the job.</p><p>Unfortunately, explaining things based on first principles can be an arduous and time-consuming task. One must organize their thoughts, perhaps draw a few diagrams, and find neutral sources to reference, which themselves must be written from first principles and not be logically flawed. This is one reason I believe it’s so important for management to let their experts be experts and not micromanage — employees will either be forced to commit logical fallacies or spend a lot of time you don’t have.</p><p>Years later, I still stand by my recommendations. After all, it worked out well. We went with my approach, and the project was successful and on time. It remained a scalable, effective, and performant solution for as long as I was there. So it all worked out, right?</p><p>Unfortunately, no. You see, being right isn’t as important as being trusted. Committing a logical fallacy demonstrates one’s lack of reasoning ability, which can be detrimental to your credibility. Rightly or not, I believe this is what started my problems at EC.</p><p>Knowing the logical fallacies will not only help you make better decisions at work but in life. It also helps keep you out of the crosshairs of those who try to manipulate you.</p><h3>Don’t be intimidated by fancy degrees or challenging problems</h3><p>I was one of the few employees in the room who didn’t have a Ph.D. I was a little nervous, but really excited to have an opportunity to work on real cutting-edge science, which is a bit different than the applications I typically work on.</p><p>What I did get concerned about was if they were going to take me seriously or not. Sometimes I have a feeling of inadequacy around people with fancy degrees from fancy colleges. I’m sure it’s something I project onto others more so than how they actually see me.</p><p>This experience taught me that I was indeed projecting. I was taken seriously and was able to have deep conversations with the scientists. Ph.D.’s aren’t necessarily smarter (although I’m sure these folks were), they just happen to have dedicated their lives to studying a specific field — and they are experts in that field. But I like to consider myself an expert in what I do, too.</p><p>At the end of the day, I was able to learn enough to work together with them and solve interesting, challenging problems. And that’s all that really matters.</p><h3>There can be multiple truths</h3><p>Whenever I approach a new software project, one of the most important questions I try to answer is “what are the things that we are talking about?” How are they defined, and more importantly, what are the relationships? This is called the <strong><em>data model</em></strong>. It&#39;s an abstraction of the world we are trying to describe. A simple example: A <em>class</em> has a <em>curriculum</em>, <em>teacher</em>, <em>students</em>, and <em>schedule</em>.</p><p>In my experience, the success of your software depends on how well you define your data model upfront. As it turns out, data modeling is also one of the hardest problems in AI. They call it Knowledge Representation, or KR for short.</p><p>Imagine the following fictional short story:</p><blockquote>Alice kicked the ball. “Oh, no!”, she said. “It went the wrong way!” John, standing behind her, got the ball and passed it to Jane, standing right in front of the goal. She kicked and scored! Alice’s team had won.</blockquote><p>What is going on here? Obviously, it’s a game of soccer, but there’s a lot more than that going on. Alice and her teammates are humans with motives, feelings, and actions. There is a timeline of things that happen with various actors and outcomes, and much more. But how does the computer represent all of this information in a structured way? Would that same structure be true for other stories? This turns out to be a really hard problem.</p><p>Part of my job was to come up with a way to visualize the machine’s understanding. One of the biggest lessons was to not try to fit everything into one point of view. Duplication around different pivots or lenses is often a favorable tradeoff to reduce an order of magnitude of complexity. You can use this same thinking to apply to other problems — multiple points of view can all be correct, and each tells a different part of the story.</p><p>Don’t worry about the D.R.Y. nonsense you were force fed at University. Some repetition is okay. Not everything needs to fit into one unified model. Sometimes, a unified model is just an overly idealistic constraint we impose on ourselves.</p><h3>Fail as quickly as possible</h3><p>If you have experience working for a startup, then you’re probably no stranger to building something with incomplete or constantly changing requirements. It’s frustrating but often unavoidable as the business is learning.</p><p>Working at EC, this was an order of magnitude worse. The science itself was constantly changing. This is a research project, after all. On top of that, I had no previous AI experience. In the beginning, I had a really hard time getting all this stuff to fit neatly into my head.</p><p>I was hired to design and build a new intuitive user interface that could present to ordinary people what the system was thinking, and help it learn. It could also be used by researchers as a tool to help gain insight.</p><p>It was difficult to know if what I was doing was right, and was nervous. I’m presenting to the inventor of IBM’s Watson, after all. What if he (and the scientists) thought I was full of it?</p><p>I could talk about how to cope with <a href="https://medium.com/search?q=imposter%20syndrome">imposter syndrome</a>, but I won’t, as there are enough great articles on the subject. Instead, I’ll give you an easy escape hatch: stop fighting it and just fail already.</p><p>I found that it was much faster to just present something that made no sense to a better-informed person than to spend my time trying to gain that understanding for myself upfront. Sure, I want to learn as much as I can; this makes me a better designer and architect. But I don’t need to learn it all from first principles on my own, or on a single pass. As long as they give you the chance to correct your mistakes, forget about people’s opinions. If I hadn’t started doing this, I don’t know if my project would have launched on time. And boy was it less stressful.</p><p>Dr. Ferrucci and the scientists were very receptive to my work, and we went through many quick iterations, ending up on something I think we were all proud of. I don’t think anyone ever questioned my ability.</p><h3>Knowing vs. understanding</h3><p>There is a big difference between “knowing” and “understanding.” <em>Knowing</em> usually means being able to recall individual facts. It’s static. <em>Understanding</em> means being able to reason about those facts logically, from first principles. It’s the difference between regurgitating information you read and being able to teach it to someone.</p><blockquote>I don’t know what’s the matter with people: they don’t learn by understanding; they learn by some other way — by rote or something. Their knowledge is so fragile! — Richard Feynman</blockquote><p>Our AI system had to not just be knowledgable, but understanding. I’m not going to say that I know how this was achieved in a Symbolic Reasoning AI; that remains firmly above my head (and under an NDA). However, it reinforced my belief that my own understanding does directly impact the value of the software I write or the interfaces I design.</p><p>I make a big deal out of designers and engineers understanding the underlying problem or technology, and not just doing what managers tell them to do or following popular patterns. Understanding the fundamentals will help you to select the right tool, and how to properly use it.</p><p>Part of the job of a designer or a programmer is to teach. Designers teach a person how to interact with software through their designs. Programmers teach other programmers how things work by writing readable and reasonable code — but you can’t teach without understanding.</p><h3>A bad culture fit is usually not your fault</h3><p>Towards the end of my gig at EC, I knew that I was struggling. But it wasn’t with the quality of my work, for which I continued to get good feedback.</p><p>Everyone was cordial and professional as you would expect. They were also brilliant and interesting to talk to. But I really didn’t fit in. I had come from a very different culture, and I wasn’t very good at adapting to theirs. I struggled to get on the same page. We had a different set of values, expectations, and ways of working and collaborating.</p><p>I started my first company in high school and since college, spent most of my career working at fast-paced bootstrapped startups. I’m obsessed with efficiency, shipping product, and building scalable, reliable software. My projects have stakeholders and customers. I was forged in startup culture.</p><p>Dr. Ferrucci once called me “a builder of stuff.” I thought it fit me well.</p><p>Much of the team was from academia, many of whom were from the original IBM Watson team. EC had no customers and no specific applications of focus. This was by design; they wanted to be a pure research company, free from the shackles of profit margins and customer deadlines. They didn’t want their AI to be overfitted for any particular problem. They didn’t want a repeat of what had happened at IBM, where some felt a <a href="https://medium.com/the-new-york-times/what-ever-happened-to-ibms-watson-d4ce7b497599">misunderstanding</a> of their technology and the need for a marketable product today had stifled their research after their success on “Jeopardy!”.</p><p>The result of all of this was increasing friction and failed communication. I was pretty hard on myself and took this personally. I may have understood intellectually that it wasn’t my fault, but it sure seemed like it was.</p><p>Cultural fit is important for any team. If your values don’t align, it’s not going to work out long-term. What EC does works well for them, but not for me. And there’s nothing wrong with that.</p><p>I did challenging work and broke into an entirely new and fast-growing field. I worked side-by-side with some of the leading computer scientists. I invented a new way of visualizing KR, and who knows, maybe I helped contribute to AI in some way.</p><p>To anyone at EC reading this, thank you for the amazing opportunity. I hope I helped. I learned a lot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FvsyPZdt6noE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DvsyPZdt6noE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FvsyPZdt6noE%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca416c68b6d5b837f3af57002904c4c8/href">https://medium.com/media/ca416c68b6d5b837f3af57002904c4c8/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e76b4a2d4ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/what-i-learned-as-a-developer-while-working-for-the-inventor-of-ibms-watson-7e76b4a2d4ba">What I Learned Working for the Inventor of IBM’s Watson</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When To Throw in the Towel on Your Technical Debt]]></title>
            <link>https://medium.com/better-programming/when-to-throw-in-the-towel-on-your-technical-debt-b1f6bbb3aa0c?source=rss-8377f8231223------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1f6bbb3aa0c</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Marc H. Weiner]]></dc:creator>
            <pubDate>Tue, 27 Jul 2021 16:02:33 GMT</pubDate>
            <atom:updated>2025-12-11T05:02:57.268Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Don’t get caught in refactor hell</h4><figure><img alt="Lots of electronics" src="https://cdn-images-1.medium.com/max/1024/0*GUcqkqaM_5pMTsis" /><figcaption>Photo by<a href="https://unsplash.com/@yungnoma?utm_source=medium&amp;utm_medium=referral"> Yung Chan</a>g on<a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral"> Unsplas</a>h</figcaption></figure><blockquote>Software that suffers from technical debt is hard to change. Fixing this by changing it, is… well, hard.</blockquote><p>I believe that most managers are aware of technical debt and that it is bad, but fail to understand its true impact. This misunderstanding can lead to some very costly mistakes.</p><p>One of those mistakes is keeping a codebase around past its expiration date. Yes, codebases have expiration dates.</p><p>Another costly mistake is prematurely shortening that expiration date, due to the accumulation of unmitigated technical debt.</p><p>Tech debt usually means that your code, abstractions, concepts, or data models no longer make the most logical sense. If you have any outdated dependencies, areas of friction, or lack of testability — that also counts.</p><p>Let’s say we have an application with a lot of tech debt. How do we resolve it? Do we refactor the application, break pieces off and rebuild those, or start completely from scratch? Of course, it depends on how that application was written and what kind of tech debt you have.</p><p>To answer the question, let’s look at two types of strategies: prototype development and sustainable/scalable development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3NfJcUDBtbaoMQIcA3P6bQ.png" /><figcaption>Image by <a href="https://mhweiner.medium.com/">Marc H. Weiner</a></figcaption></figure><h3>Prototype Development</h3><p>The goal of Prototype Development is to prove (or disprove) a hypothesis as quickly and cheaply as possible, such as “does this technology work?”.</p><p>Prototypes are generally not designed to scale or stick around for a long time. Observability, testability, and expensive in-house engineers are optional. For many, outsourcing is their only option. This might be the quickest and cheapest way to build working software, but with some serious limitations — it must be relatively simple and not mission/safety-critical. These codebases also have very short shelf lives, often just a few months.</p><h3><em>Sustainable/Scalable Development</em></h3><p>If you already know what you want to build and roughly how to build it (and have the money to fund it), this is probably the way to go. If your application must scale quickly or is mission or safety-critical, this is your only option.</p><blockquote><strong>In architecting a new program, all the important mistakes are made in the first day. — </strong>Robert Spinrad</blockquote><p>Sustainable development starts slow. Whether you use Waterfall or Agile, upfront work is required. You don’t need to solve every problem right away, but sustainable software must be based on good assumptions and abstractions. It must be designed to scale on some basic level regarding load, complexity, and team size. Best practices and observability infrastructure must be there from the beginning, or you will never catch up.</p><p>The team that builds and manages this application must also be experienced and have a particular set of skills and discipline. <em>This isn’t software development, it’s software engineering.</em></p><p>Once in a while, refactors will be required to maintain team efficiency and reliability. Think of this like the oil changes and preventative maintenance in your car. If you don’t do them, things become much more expensive and take much longer. Your car will also become less reliable. The reason for some of these refactors might be that an assumption may not pan out, or a business requirement changed — but that’s fine because your software was designed to handle this.</p><p>Changes to sustainable software are easy and estimable.</p><h3>Back to the question…</h3><p>Let’s do the easy one first.</p><p>If you have an outsized or long-lived MVP, then you have no choice but to start over. Your MVP has already served its purpose by now (or it wasn’t properly scoped) and has outlived its expiration date.</p><p>If one of the following scenarios applies to you: (1) your software followed the sustainable path but at some point got off course, (2) if your software was developed using a method somewhere in the middle, then refactoring could make sense. Answer the following questions:</p><ul><li>Does more than 50% of your code need to be modified to clear your debt?</li><li>Is your software based on poor abstractions or incorrect data models?</li><li>Is most of your software not written using best practices?</li><li>Is your tech debt not limited to decoupled subsystems?</li></ul><p>If you answered yes to any of the above questions, then I would strongly consider sunsetting instead of refactoring.</p><h3>The Legacy Migration/Refactor</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8sU_s26vU4pZVigLZqwvw.jpeg" /><figcaption>Photo by <a href="https://mhweiner.medium.com/">Marc H. Weiner</a></figcaption></figure><p>Ever hear of a legacy system refactor or migration that was on time and under budget? Me neither. The reason for this is that you are always fighting the worst effects of the technical debt you are trying to solve. This work is unpredictable, inestimable, and risky. It’s a management nightmare.</p><p>Legacy migrations often take years, and many never finish. While working at <a href="https://www.shutterstock.com/">Shutterstock</a>, I learned of a migration project that was ongoing for over five years! They were now moving from one obsolete technology to another.</p><p>You’ll still need to continue adding new features this whole time and keep up in a fast-paced environment. This usually means parallel development by a large team (or several teams), all on the same codebase. We’re talking about large changes on a challenging codebase with high churn and tight coupling. Engineers will have a difficult time breaking up the work into small pieces. This leads to large commits, stale branches, and horrific <a href="https://itnext.io/recommendations-to-avoid-merge-conflicts-845ec133676e">merge conflicts</a>. Merge conflicts leads to wasted work and even <em>slower</em> development, frustration, low morale, and risk of regression (bugs). You’ll be lucky if you’re not moving <em>backward</em>.</p><p>Want to solve the problem by throwing more bodies at it? Unfortunately, that’s also one of the problems with tech debt — onboarding sucks. This code is only understood (if it’s really understood at all) by the engineers who wrote it. Hiring also follows the law of diminishing returns; the more people you add, the more management, communication, and meetings are required.</p><p>Legacy migrations aren’t usually fun to work on, so you might have a hard time attracting talent. Try telling prospective employees that they will be working on a big legacy migration project under a tight deadline. The best engineers will likely pass (I usually do).</p><p>So why do companies do this?</p><h3>Don’t HODL Your Code (Hold On for Dear Life)</h3><p>Over the years, I have seen folks hesitant to throw away their code, gripping it like Gollum and the One Ring. I can only guess, but I think the rationality is because they spent years and lots of money to develop it, so it must be highly valuable. Perhaps it seems wasteful or drastic to throw it away and start over. Unfortunately, software doesn’t work this way.</p><p>The problem is, technical debt is named after financial debt because it is similar. Debt has negative value. In fact, you are paying for simply having it. Every. Single. Day. And the interest rate is high.</p><p>One of the biggest costs is opportunity cost — the ability to easily respond to changing business needs, new features, new technology, security concerns, bugs, or scale. Your code is a liability, and it is costing you dearly.</p><p>Let it go.</p><h3>Sunsetting (Do This Instead)</h3><p>Starting over doesn’t mean you have to throw all of your code away today. In fact, you’ll probably have to keep it going for a while as best you can until the new version is ready. This is called <em>sunsetting</em>.</p><p>With sunsetting, the rollout of your new system can be done in an unhurried way. You can either sunset the whole thing and move users over, replace the “backend” first and migrate the data, or break pieces apart. The specific strategy you use depends on your situation. No matter what you end up doing, this is all considered “sunsetting” with the same goal, to replace code and not to change it.</p><p>Sunsetting gives everyone time to adapt, time for the new system to build up load, and for any remaining issues to be identified and resolved.</p><p><strong>Most importantly, during this entire process, technical debt won’t be slowing you down.</strong></p><h3>Conclusion</h3><p>Just like an <em>event horizon</em>, the boundary of a black hole where not even light itself can escape, there is a point at which your technical debt becomes unrecoverable. Every codebase has an expiration date, some later than others depending on the development methodology and the skill and discipline of the team.</p><p>Starting over from scratch allows you to learn from the past, but with none of the baggage. Never be scared to throw away your code and start over. You’ll breathe a sigh of relief when it’s done.</p><p><em>Refactoring or starting fresh? Aeroview helps you debug faster with smart, scalable logging. Join the beta — </em><a href="https://aeroview.io/"><em>aeroview.io</em></a></p><p>Want to know more about building scalable, sustainable, and reliable software? Read my other article:</p><p><a href="https://mhweiner.medium.com/do-these-10-things-to-improve-your-apps-reliability-fc114b16c2d1">Do These 10 Things to Improve Your App’s Reliability</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1f6bbb3aa0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/when-to-throw-in-the-towel-on-your-technical-debt-b1f6bbb3aa0c">When To Throw in the Towel on Your Technical Debt</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>