<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Derrick Wippler on Medium]]></title>
        <description><![CDATA[Stories by Derrick Wippler on Medium]]></description>
        <link>https://medium.com/@thrawn01?source=rss-6f2de1748b41------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*r1rfBGXgNccYlU6SgAksAA.png</url>
            <title>Stories by Derrick Wippler on Medium</title>
            <link>https://medium.com/@thrawn01?source=rss-6f2de1748b41------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:46:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thrawn01/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Anatomy Of A Product: Handling Product Maturity]]></title>
            <link>https://medium.com/@thrawn01/anatomy-of-a-product-handling-product-maturity-e976baa822c9?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/e976baa822c9</guid>
            <category><![CDATA[refactoring]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Tue, 04 Jun 2024 23:27:56 GMT</pubDate>
            <atom:updated>2024-06-04T23:27:56.058Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rTQc1kQjttpkZv8KRYKdTA.jpeg" /></figure><p>This is a continuation of a previous post entitled <a href="https://medium.com/@thrawn01/anatomy-of-a-product-18b23eaac130">Anatomy Of A Product</a> . In our first article we discussed the basics of good code and good design and how understanding data ownership principles lead us to good decisions which will benefit our product into maturity.</p><p>We started this journey by following our imaginary product from creation to launch, now let’s follow it into maturity.</p><h3>Dealing with multiple versions</h3><p>Our customers are now asking us to support forwarding a single email to multiple recipients, in addition our parent company wants us to support forwarding emails via HTTP Webhook. While we could probably make this work as an additive change to our current v1 API, let’s introduce this as v2 of the forwarding product to see how our mental model might evolve to handle new versions of things.</p><pre>POST /v2/forwards<br>{<br> &quot;email&quot;: &quot;from@example.com&quot;,<br> &quot;forward&quot;: {<br>  &quot;recipients&quot;: [<br>   &quot;forward1@email.com&quot;,<br>   &quot;forward2@email.com&quot;,<br>  ],<br>  &quot;webhook&quot;: &quot;https://email.com/webhook&quot;,<br> }<br>}<br>GET /v2/forwards{email}<br>DELETE /v2/forwards{email}<br>POST /v2/forwards/import/csvjson</pre><p>Here is the new model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZoHtFh0YP4G9QMJrB5CIxw.png" /></figure><p>The most complex of the changes is probably in the ForwardEvaluator as it now must query both the V1 and V2 rule stores for rules to match. We want this duplicate behavior because it’s possible customers have rules in both versions of the product, and both should continue to work even as customers migrate rules from V1 to V2. We also introduced a WebhookClient and a new thread pool which will handle sending emails to customers via HTTP.</p><h3>A clean break</h3><p>You may have noticed that we made a clean break in version 2 from version</p><ol><li>While you might reason that it’s okay for version 2 to share some code with version 1, we generally avoid doing so. The reason for this is simplicity, and scoping test-ability. We will inevitably be making changes to version</li><li>When we make these changes the shared code changes might also inadvertently affect version 1.</li></ol><p>The more shared code you have the higher the complexity of your code will be, which in turn requires more <a href="https://en.wikipedia.org/wiki/Cognitive_load">Cognitive Load</a> to grok the system. In addition, it increases the probability of code changes in V2 inadvertently affect V1. In your mind you only made a change in V2 but in reality the code is shared so you actually made a change in V1 and V2. Because of this your testing strategy must reflect this or you risk making accidental breaking changes to V1 while working on V2.</p><p>This is especially useful if we do not plan on adding any new features to V1, as the V1 code base can remain completely untouched as we continue to work and improve V2, further reducing the risk of breaking our customers who still use V1. (and, If we are being realistic; there will ALWAYS be V1 customers. Just because we deprecate a thing, doesn’t mean customers will stop using it)</p><p>There’s also a psychological component where developers generally want to work on the latest and greatest V2, but if they know making changes to V2 is going to be a danger to V1 which in their mind is in the attic gathering cobwebs, they will be less inclined to work on a code base which is intertwined with an older version of the product. Instead, let’s aim for simplicity and break the “DRY: once and only once rule” and make a clean break between versions even if that means duplicating some code.</p><p>During the early days of <a href="https://mailgun.com/">Mailgun</a> we had an XML based public API for customers, when we deprecated it, we made a clean break. While the XML code continued to run in production, all new work continued on the JSON based API. After a few years as developers came and went, many all but forgot about the old XML version due to the clean separation of code. Even as code changes happened against the JSON API, we never broke the old XML version. The clean break was so successful that when we eventually retired the XML API years later, many of the new developers who worked on the JSON API for years where shocked that we even had an XML API in production. This is the value of making a clean break. See <a href="https://salmonmode.github.io/2020/08/14/the-harmful-obsession-with-dry.html">The Harmful Obsession With DRY</a> and <a href="https://wippler.dev/concepts/Clean-Code">Clean Code</a></p><h3>Premature optimization</h3><p>Developers as a whole are pretty obsessed with performance and efficiency, (I am among them) and so this clean break design puts many off as it is less efficient than other methods for handling these types of situations. I MUST caution against such thinking. One thing I believe completely is that you must “measure first and then act”.</p><p>Looking at the above model, it does appear that the ForwardEvaluator might struggle since it must query both v1 and v2 to match a rule. But in practice, and from my experience, such systems work fine until we&#39;ve actually measured a performance bottle neck. Maintainability and good design are always considered first, only after, when you can measure how well the application is running, should you go back and iterate on performance. So many times, I&#39;ve imagined that some part of the code will be slow, only to find out later, a completely different part of the code is slower and actually causing a problem. <a href="https://wiki.c2.com/?PrematureOptimization">Always avoid optimizing, before we know that we need too</a></p><blockquote><em>One way to solve rule matching performance issue is by implementing a </em><a href="https://en.wikipedia.org/wiki/Trie"><em>Trie</em></a><em> structure and storing the rules in a trie and then merging the v1 and v2 trie before evaluation. We can then serialize the trie to a buffer and store it in a cache. We didn’t need the Trie for the first year or so, so Premature Optimization saves us unnecessary work early on, and gives us time to refine the product before expensive optimization work is needed.</em></blockquote><h3>Enriching the data</h3><p>Another important aspect of data moving through the system is the concept of enriching or adding context to the data as it moves through the system. As an illustration, it might be useful to the end user to know which rule matched the forwarded email. We could do this by having the ForwardEvaluator attach the rule id to the forward data like so.</p><pre>forward_job:<br>  email_id: 1234<br>  to: forwarded@example.com<br>  from: from@example.com<br>  context:<br>    matched_rule_id: 1234<br>    customer_id: 4566</pre><p>Now when we pass the forward_job to the SMTPClient and WebhookClient this additional context can be forwarded via email via headers or as apart of the delivered JSON payload. Another typical use is that of authentication. If we can identify the customer via authentication then we can attach that information as context to the data so down stream abstractions and systems can make use of that knowledge without having to go through additional work of figuring out who the customer is.</p><h3>Patterns emerge</h3><p>Hopefully at this point you begin to see a pattern of abstracting inputs and outputs behind well defined abstractions. This results in a modular code base that can adapt as our product grows and expands in scale and complexity. Hints about what modules should be created can always be found by asking “who has ownership of the data”, or “who takes ownership after transformation”. A transformation doesn’t always own the data, but a transformation should typically be it’s own abstraction. Following this process does not guarantee you will never have to refactor code, but; by creating the right interfaces, abstractions and data ownership decisions early on we can greatly simplify maintenance of the code and support healthy scaling of the system.</p><h3>Getting it wrong</h3><p>Of course there are often times when the thing you think you are building turns out to be completely different, and you end up throwing a lot of code into the pit of /dev/null, while refactoring vast portions of the code base. This happens because we don&#39;t always know what we think we know when a project starts out, or assumptions are made that should not have been. Often, this is because the problem domain is not clear to us, or was not communicated clearly, or requirements have changed.</p><p>This should not deter us from adopting good code design. Far from it! In fact, application of the strategies discussed here can assist during the prototyping and discovery stage of a project. In my experience, I have found that prototyping a public interface without an implementation, is a great way to sus out flaws in our domain knowledge and design. Using the prototype public interface in a different module or part of the system will often lead to a discovery in of our knowledge gap.</p><p>Often times, going through the motion of documenting an interface, writing examples of usage, can be very useful in vetting an interface. This is especially useful when you share the documentation of the interface with those who will consume it. Users of a thing (especially developers) can be very vocal about how you got it all wrong.</p><p>I did this when I first started writing a mime parser in golang. I wrote a example.go where I took the imaginary mime parsing library and worked through a couple of example exercises of how the library should be used. I then shared the examples with the team and got immediate feed back on what they didn&#39;t like, or what the library needed to do, but didn&#39;t do. All without writing an single line of actual implementation.</p><p>Indeed the first part of any project should be a prototype of the interface, which you then share with everyone who might have an opinion or who might end up using it. Finding a major flaw in the design is often much harder to deal with once the interface is implemented, tests are written, and its ready for PR review.</p><h3>Services</h3><p>Up until now our product has been a monolith, and our parent company wants to make use of our excellent SmtpServer and EmailStore implementations for other projects. Micro-services, here we come! Let&#39;s see how this plays out.</p><p>Contrary to the beliefs of enthusiastic developers everywhere, products and teams often find <a href="https://www.youtube.com/watch?v=hIFeaeZ9_AI">very</a> <a href="https://blog.christianposta.com/microservices/istio-as-an-example-of-when-not-to-do-microservices/">little</a> <a href="https://medium.com/@mhetreramesh/when-to-choose-microservices-architecture-over-monolithic-why-794aed04d8db">benefit</a> from implementing micro-service architecture on day one of a product. However, as we’ve demonstrated above, organizing your code and mental model in a micro-service style design provides a great deal of benefit and is a gift that keeps on giving when you decide it’s time to transition to micro-services.</p><p>Since our parent company wants to use our SMTP server and other teams need a way to get emails that are received by our SMTP server, let’s break those two interfaces out into separate services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UWURpdLg8jvuEv25lZW2SQ.png" /></figure><p>As you can see, the EmailStore code remains intact and all we do is expose the EmailStore interface to the network via GRPCServer. Additionally since our interface really hasn&#39;t changed clients to the new EmailStore micro-service use a similar interface as before, the only difference is the interface is a GRPC client that connects to the remote micro-service.</p><p>Now that EmailStore is a separate service it might be helpful to review some previous confusion around the concept of data ownership and protection. A service, like the module it wraps, owns the data it stores. As such, it protects it&#39;s self from abuse both internally and externally. The mental model does not change from when we transition EmailStore as a module to a micro-service. Both implementations should protect the data and it&#39;s data store.</p><p>This is why rate limting happens in the EmailStore module, and not in the Receiving Service. Since becoming a micro-service It is even more important that we protect the EmailStore from accidental abuse as we expose the micro-service to more and more clients on the network.</p><blockquote>Even though rate limiting occurs in the Storage Service the Receiving Service should relay to the client (if possible depending on the protocol) when the Storage Service rate limits requests sent to it.</blockquote><h3>Breaking up the monolith</h3><p>Now that we have broken out the email store as a separate micro-service, We can also break out the WebhookClient and SmtpClient into services or even server-less functions which can handle delivery and scale horizontally as demand increases. For our purposes, we will spin them out as services and use Kafka as a replacement for our in-code queues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1fV9f0cMf9oxKCI-eFmvQ.png" /></figure><p>Because we organized our project around abstractions, domain driven principles, and ownership, transitioning to micro-service architecture doesn’t require a major refactor of the code. We simply take the existing `WebhookClient` and `SmtpClient` abstractions and place them behind something which consumes work from Kafka.</p><p>Even though we did not start this project as a micro-service the DACO mental model mirrors that architecture. In fact I would encourage you to think of abstractions as a collection of micro-service interfaces. Each interface represents what may in the future become a network interface to that micro-service. Services after all, are just an encapsulation of code behind an interface.</p><p>The power in thinking this way is two fold. One, we prepare our code for the micro-service architecture that it may become should our product be successful. Two, we force ourselves to think of data as moving through the system much like data would move through the network.</p><p>Due to the ownership principles applied at the outset we have been able to achieve a high level of quality and scalability with reduced maintenance burden.</p><h3>Domain Driven Design</h3><p>We’ve talked a lot about data ownership in this article, but I would like to quickly point out that applying DDD in addition to ownership helps us make good decisions when deciding where to put things in our code base. When defining the boundaries of our micro-services or code modules, we are not partitioning by the number of lines of code, endpoints or methods of our interface, but by the problem domain the modules are solving.</p><p>Each abstraction here has a very specific problem domain. ForwardEvaluator decides if an email matches a rule, it’s only concern is to solve this problem in the best possible manner. The same is true of the SmtpClient and WebhookClient, although you could make the argument that the problem domain they solve is so similar that they could be in the same abstraction. This is a decision you as the code architect have to decide upon, as there are trade offs to both solutions. There is <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjay4HPp4HzAhU_kmoFHfLXAasQFnoECAcQAQ&amp;url=https%3A%2F%2Fwww.amazon.com%2FDomain-Driven-Design-Tackling-Complexity-Software%2Fdp%2F0321125215&amp;usg=AOvVaw3lLATSzmgwNDV7fN_Je7Lg">much written</a> <a href="https://martinfowler.com/bliki/DomainDrivenDesign.html">on this subject</a> that can help you make the right decision for your project. See also <a href="https://wippler.dev/concepts/Domain-Driven-Design">Domain Driven Design</a> for a good jumping off point.</p><h3>Conclusion</h3><p>There is a lot to unpack here, and we glossed over a ton of detail to just so we can touch on a several high level topics.</p><ol><li>Always abstract your input and outputs within the bounds of your problem domain. There is no need to abstract away all interfaces; You wouldn’t abstract the ForwardEvaluator interface, but you would abstract any interfaces your problem domain uses to get data in and out. IE: HTTP or GRPC.</li><li>Our code is a reflection of our problem domain, in the same way our problem domain is a reflection of our code. It’s actually quite helpful to think of the abstractions in your code as micro-services solving a problem domain, they just communicate through method calls instead of network calls.</li><li>Establishing data ownership early can simplify validation and protection of data and will continue to pay dividends as the system grows.</li><li>Design of your abstractions is limited in scope by the problem domain it’s solving. Avoid overloading abstractions even at the expense of perceived performance issues. The product and maintainability always come first, performance can typically be iterated upon later.</li><li>Learning when and where to use abstractions can help keep the cognitive load low, thus increasing maintainability and scalability.</li></ol><p>We have found that centering the overall system design on a simple mental model using well known design methodologies can greatly improve the success of our projects and help us reduce both technical and architectural debt. Using this model of thinking has allowed us to grow and scale systems in a clean and maintainable way. It is my sincere hope that this can be shared and will assist you in your products as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e976baa822c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anatomy Of A Product]]></title>
            <link>https://medium.com/@thrawn01/anatomy-of-a-product-18b23eaac130?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/18b23eaac130</guid>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[refactoring]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Tue, 14 May 2024 17:39:15 GMT</pubDate>
            <atom:updated>2024-06-04T23:30:34.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eL0mhivCNMztx68dcxaZPw.jpeg" /></figure><p>So, you want a front row seat to how a product gets developed without the constant need for a re-write every 10 days or so. Welp, you’re in the right place.</p><p>This article is an explanation of methods I have used to design code and systems such that you maximize test-ability, velocity and reduce technical debt throughout the life of the product. This article represents a culmination of experience building and scaling products at <a href="https://mailgun.com./">https://mailgun.com.</a></p><p>So, we are going to follow an imaginary product from design, to deployment, to scale and unforeseen new requirements. We are glossing over a TON of detail and will jump around a bit, with the goal giving you a birds eye view of how good code architecture can become an asset over the life of a product.</p><p>Ready? Let’s do this….</p><h3>Mental Models</h3><p>Let’s start by thinking about how we should mentally model our code architecture. A mental model allows us to reason about the system as a whole without having to understand all the details of the abstractions that make up the mental model. The mental model I’m presenting here has successfully been used to create and reason about code and architecture at Mailgun. This way of thinking presents us with maximum flexibility, helps us avoid and manage technical debt, and allows us to maintain high-quality code over the lifetime of the product.</p><p>In this article we present an approach inspired by <strong>Domain Driven Design</strong>, <strong>Abstraction</strong>, <strong>Separation Of Concerns</strong> and <strong>Data Ownership</strong> principles. I call this <strong>DACO </strong>for short (Domain, Abstract, Concern, Ownership). Most of these principles are not new and have been around for years. Mastering these concepts are vitally important to writing <strong>good code</strong>, and <strong>good architecture</strong>.</p><blockquote>“What is good code”; you might say?</blockquote><blockquote><strong>Good Code</strong> is easy to change code. <strong>Good Architecture</strong> is easy to change Architecture. Thus your goal is to design systems which make change easy within your problem domain.</blockquote><p>Let’s consider the creation of an email forwarding product, and for the remainder of this article we will follow this product throughout its creation and maintenance life cycle to see how the movement of data, interfaces, and code change can play out over the life cycle of a product.</p><h3>The Forwarding Product</h3><p>Let us first consider a simple CRUD interface for our product. This is where users will create and delete forwarding rules for our email forwarding product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*XdRe5IEGHqycpPcQ68oFiQ.png" /></figure><p>Here we define the simplest of features for our product. With this, we can create a rule that says. “When we receive an email addressed to from@example.com we forward the the email to forward@email.com”</p><p>Now let’s assume we will store these rules in a database and model the interfaces that will make up this portion of our service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7WYpi4_xUVL_XJDk4b5oig.png" /></figure><p>The first two interfaces in our mental model are also abstractions of both input and output. The RuleStore abstracts away the database we are using and is an example of the <a href="https://martinfowler.com/eaaCatalog/repository.html">Repository pattern</a>. The second abstraction is the HttpAPI. You can think of HttpAPI not as the interface for the service, but as a network interface for the RuleStore. This is because it’s primary function is to expose RuleStore methods to the network.</p><p>But why don’t we just organize the code in a procedural manner? These interfaces don’t seem to follow the procedure of what is actually happening to the data as it passes through the service, why would we complicate things with interfaces so early in the project? To understand why these abstractions are SO important, let’s talk about how the <strong>rule</strong> (our data) passes through our system from a strictly procedural point of view. Here is a visualization of what is occurring when we create a new rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*-BO2n4ZMEU1kuUtN9_gWxg.png" /></figure><p>In the Procedural model, the user creates a JSON rule in accordance with our API and sends it via HTTP Post. We then validate the rule (aka, are the email addresses valid?) and then we store it in a database.</p><p>You may reason that when the HttpAPI handler receives the JSON rule, it should un-marshal it into a struct and then validate the rule before saving it in the database, and all of the code can live in the handler. Why would we litter this seemingly simple operation across two different interfaces and multiple methods?</p><p>This procedural logic might even seem intuitive, if you think of the HttpAPI handler as the gatekeeper protecting the rest of the service from bad data. But here I want to introduce the second major concept to think about when constructing your mental models.</p><h3>Data Ownership</h3><p>We should ask ourselves, who is responsible for the rules having the correct data? Who owns the rules? Who protects the rules and the database from abuse? This line of questioning is important to the mental model, because if we identify who owns what, we can consolidate validation and protect our data in one place and simplify our mental model. If we have no clear ownership, then validation and data protection code can become spread across many different parts of the code base and may be forgotten as the product grows and new features added and subtle bugs introduced. This idea of ownership works well with the <a href="https://martinfowler.com/eaaCatalog/repository.html">Repository pattern</a> as you can think of the Repository as owning the Rule while at rest (saved in a database). It also works well with pipelines, where ownership of an aspect of the data can transfer from one abstraction to another, such as when the data is transformed. Once the data has been transformed, ownership of the transformed data could be passed to a different module in the system.</p><p>However, one might reason that the HttpAPI interface should own the data as it takes ownership of the data from the network, since the data is transformed from a serialization format to in code structure. But, the data only passes through HttpAPI on it’s way to the RuleStore. In a way, the HttpAPI is just a transformer of the data. The only concern of HttpAPI is getting the data from HTTP via JSON and transforming it into a structure that RuleStore understands. You can think of HttpAPI as essentially a network adapter that allows access to the RuleStore via the HTTP protocol. Thinking in this way, we see that while both HttpAPI and RuleStore abstract input and output for the entire service, they can also be abstractions for internal modules. Building our code around these abstractions can become quite useful in the future as new capabilities to the API are added.</p><p>WHY tho? <a href="https://en.wikipedia.org/wiki/KISS_principle">KISS</a> (Keep It Simple Stupid) tells us to do the simplest thing that works, in that case we should just unmarshall the JSON, validate it and store it in the database all in the HTTP handler for /v1/forwards. What is all of this interface and ownership garbage?</p><blockquote>Don’t abstractions just obfuscate the code? JUST DO THE THING, and move on!</blockquote><p>By way of explanation, let us continue with the life cycle of the forward project to see how these interfaces/abstractions benefit us in the long run.</p><p>Returning to our product design, Since we have established that RuleStore owns the rules, it should also validate the rules before storing them in the database. We say that RuleStore is “protecting” the database from bad or incomplete data by validating the data given for storage. Let&#39;s update the diagram to reflect that ownership.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RWoyfZdgYeOpVzn4Yxur3g.png" /></figure><p>The next major part of our product is giving our customers an interface through which they will send us emails to be forwarded. We need an SMTP server we will use to receive emails, store them and forward emails that match our rules. Let’s diagram what this might look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ba2wgYpT2cqIH5a-cHRc_w.png" /></figure><p>Here again we abstract our application input with SmtpServer which stores the email (in the form of a <a href="https://en.wikipedia.org/wiki/MIME">MIME</a>) in the EmailStore. Next we have the ForwardEvaluator that decides if the email matches any of the email forward rules found in the RuleStore, if it does then it uses the output abstraction SmtpClient to send the email using the EmailStore to retrieve the stored email and send it to the destination SMTP server.</p><h3>The Data</h3><p>We have two primary forms of data for our product, the Rule and the Email. The Rule provides the transformation criteria and the Email is the thing we are transforming. We say <strong>transforming</strong> since we are changing the To address in the MIME, and if we are good email citizen, we add a Received header before forwarding the Email to the appropriate SMTP server.</p><p>When we receive the Email via SMTP we get a To and a From which are provided via the SMTP protocol, then we receive the body or the MIME of the email. We pass the MIME; which could be many megabytes of data, to the EmailStore to be persisted to the Email Database. The EmailStore then generates an emailID which is later used to retrieve the email from the EmailStore.</p><p>When the SMTPServer receives an email, it receives the body of the email (MIME) and some metadata about the email, like who its from, and who it’s addressed too. We represent that data in the following structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*2KOnzTsvAvRIYJQ7JaSX1A.png" /></figure><p>This metadata structure is what gets passed to the ForwardEvaluator in order to determine if the email matches a rule in the database.</p><h3>Product Launch</h3><p>At this point we have a minimal but complete email forwarding product. We launch the product, our customers are happy and we smoke brisket to celebrate. (I live in Texas, it’s what we do)</p><p>Soon after launching we notice the rule matching and forwarding is slowing down our ability to receive new email messages. We need to horizontally scale both rule processing and SMTP forwarding to keep up. Let’s implement two thread pools, the first is a pool of ForwardEvaluator threads which can process rules in parallel. Next we create a thread pool of SmtpClients to handle delivery of the messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2J9yTMYV4TnjzjMjmR2DVA.png" /></figure><blockquote>The HttpAPI and SMTPServer are both multi-threaded as they must accept concurrent connections, but I omit that in this diagram for simplicity.</blockquote><p>As you can see we use a queue (a channel in golang) to queue up work for the worker pools. The ForwardEvaluator pool takes a EmailMetadata as input (contains the from and to headers used to match the forward email address), then if a match occurs the ForwardEvaluator calls Queue(SmtpJob) for delivery via workers in the SmtpClient worker pool.</p><p>Of interest is how little actual code changed to accomplish this, most of the code is additive since we already had interfaces around each problem domain. All we did was put the interfaces into worker pools and now we can scale up to as many CPUs as the server permits. But let’s continue, and see how these interfaces continue to assist us as our product grows.</p><blockquote>By choosing abstractions around our input, outputs, and problem domains we avoid huge and painful refactoring efforts. In addition, our mental model remains simple enough to hold inside our little grey cells.</blockquote><p>Now….. 6 months later we are signing up bigger and bigger customers. Some customers have 60,000 to 100,000 email addresses to forward. And our less tech savvy customers are asking to import these email addresses via bulk import. So we decide to add a CSV upload endpoint</p><p>Lets update our model to see how this fits in with our current design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69m-L5v66bJpatNyiA5wnA.png" /></figure><p>Here we see that not much has changed with the core product, and since validation occurs in the RuleStore there is no validation code to duplicate in CsvImporter— See how that works?</p><p>We are simply adding a new input abstraction which handles CSV imports via HTTP. Because we correctly identified ownership and input/output interfaces early, this new functionality change requires no refactoring of the current code, only the addition of a new abstraction for a new problem domain and a new handler in HttpAPI.</p><p>Here we see ownership at work. The CsvImporter takes ownership of the CSV file (the data) when it is uploaded, and transforms that CSV file into rules which are then stored in the RuleStore. This is data flowing through the system, being transformed, and data ownership being transferred after transformation of the data has occurred. Each of those ownership transfers and transformations is a separate problem domain such that we are not over loading a single problem domain with several transformations.</p><p>If you want to use the analogy of a assembly line, each transformation is a station along the assembly line where each station adds or changes something about the car (or in our case, the data) before moving along to the next station on the assembly line. In this way, we simplify our <a href="https://en.wikipedia.org/wiki/Cognitive_load">Cognitive_load</a> and the code along with it.</p><h3>Growing Pains</h3><p>With the new addition of the CSV import, and bigger customers on boarding, our forwarding product is starting to feel the strain of success. Processing times are up and we need to improve performance if we want to continue to scale. Our metrics show the RulesStore is taking a beating from both the HttpAPI and the ForwardEvaluator looking for rules to match. It’s time to add a cache, but where to put it? Following our data ownership mental model the cache belongs with the rule store as the RuleStore owns and protects the data and the database. Additionally the RuleStore abstraction knows when to invalidate the cache as rules are deleted or updated and both the HttpAPI and ForwardEvaluator can take advantage of the performance improvements the cache provides without having to know of or access the cache directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDbiBEim9OR3A3Y3jSXRcA.png" /></figure><p>Here we’ve abstracted away the RuleStore interface and replaced it with a cache which sits in front of the RuleStore. While it’s completely valid to encapsulate the cache inside the RuleStore, implementing it this way has some advantages. First, this separation allows us to change the cache implementation from in-memory cache to <a href="https://memcached.org/">memcached</a> or <a href="https://redis.io/">redis</a> without requiring code changes to the RuleStore. Second, it also allows you to do really specialized things like use <a href="https://github.com/mailgun/groupcache">Groupcache</a> to farm out cache requests to other instances of your application and have them perform RuleStore operations, thus avoiding the thundering herd problem. <a href="https://www.mailgun.com/blog/golangs-superior-cache-solution-memcached-redis">See this Groupcache Blog Post</a> . Third, the separation of cache code and database code allows us to change either without affecting the other.</p><p>But now we’ve noticed another problem, some of our customers or random nefarious individuals on the internet have taken to abusing our SMTP server! They are sending us hundreds of megabytes of data a second, which is overwhelming our EmailStore. We need a way to protect the EmailStore from abuse! Typical wisdom might say we should limit the number of bytes customers can send us via the SMTP server. But if we think of this from a data ownership perspective, the EmailStore owns the data, shouldn’t it also protect the data from abuse?</p><p>The SMTP server can and should slow down data transfer, but it does this only when the EmailStore indicates it’s being overwhelmed. The decision to rate-limit data transfer should come from the EmailStore, not the interface (in this case SmtpServer) that abstracts the EmailStore from the network. If you are still puzzled by this decision, keep reading, things should eventually become clear.</p><blockquote>A naive implementation of the SmtpServer might load the entire contents of the received email into memory. In this case, it might make sense to protect the SmtpServer from running out of memory as receiving hundreds of emails at 50mb each could quickly drain the server of all available memory. However, a better solution is to always stream large data sets in chunks via an IOReader or similar to avoid such problems.</blockquote><h3>The price of Success</h3><p>Now that our product is so successful we are being acquired by a larger email processing company! However, our new parent company has settled on GRPC for all their internal inter service communication and they want our forwarding product to support emails from both our SMTP server and emails from their SMTP servers. The catch is, their SMTP servers are going to send us emails via GRPC!?!?! Let’s see how all this changes our mental model again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PoGgVl-OoOztCAqzzBj69g.png" /></figure><p>As you can see we now accept emails via both SMTP and GRPC, both interfaces benefit from the rate limiting we implemented in the EmailStore such that if both interfaces are receiving tons of emails, the EmailStore can still protect its database from getting pushed over. Additionally, we can add metrics to the EmailStore and get an accurate idea of when we need to scale our data storage.</p><p>Another often overlooked benefit to protecting our database is from internal abuse in addition to external abuse. Years ago I worked on a system that was using an ORM in Python. As such it’s often hard to decipher if the object you are interacting with is an in memory object or represents a table in a database. So while implementing a new feature I gleefully iterated over all the items in this object without realizing that the object was an ORM object that was sending N+1 selects to the database. The feature performed perfectly in tests, ran well in staging, but as soon as it was deployed it took down the production database.</p><p>If the production database had protections against abuses of this sort then we would have merely experienced an application performance degradation instead of an unexpected database outage.</p><h3>Switching Databases</h3><p>Speaking of overloaded databases, since integrating with our parent company SMTP servers, the number of emails being stored in the EmailStore has nearly doubled and our old database is feeling the strain.</p><p>Thankfully our new parent company has negotiated a really good deal with a cloud provider&#39;s cloud file storage product and they want us to store emails there. Since we have a nice EmailStore interface, we can switch from storing and retrieving emails from our database to the new file storage product.</p><p>But hold on, how do we migrate all those emails from the old db to cloud storage. That’s A LOT of DATA, some of those emails are 20–50mb in size! In addition our customers expect our service to accept emails during the entire migration! To solve this, we implement a CloudEmailStore with the EmailStore interface and rename the old EmailStore to OldEmailStore. Now we create an adapter that implements the same interface called EmailStoreAdaptor which calls the OldEmailStore when it is looking for old emails and calls CloudEmailStore when storing new emails. We then build an out of band migration process to mark emails as “migrated” in the OldEmailStore database which can then inform the EmailStoreAdaptor interface that the email has already been migrated to the CloudEmailStore and it should look there instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*u2SsBKjxt1K_1rT-AjPbKw.png" /></figure><p>Once the migration is complete, we remove the EmailStoreMigration adapter and CloudEmailStore implementation becomes the new EmailStore.</p><blockquote>This is an important point to emphasize, we only benefit from the interface and adapter patterns here because when we originally designed the system, we abstracted away the implementation from the EmailStore interface. If we had not identified data ownership early on, this change would have been a much more complex and painful.</blockquote><h3>The Miracle Worker</h3><p>Uh oh…. Looks like a routine maintenance change introduced a bug into the ForwardEvaluator. As a result, we didn’t forward any emails over the last 24 hours!</p><p>But all is not lost, thanks to a decent audit log from the SmtpServer we know all the emailID for emails we received that day. Since we abstracted our ForwardEvaluator and EmailStore from the rest of the system, we can quickly write up a new GRPC method called ReprocessEmail() to use as input for our ForwardEvaluator and reprocess all the email ids we received that day. This works because the SMTPServer stored all the received emails in the EmailStore. This combined with an audit log of all the email ids we received in the last 24 hours we can reprocess and deliver all the emails received! This looks something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tRFKaWCfJUoz5BUurz3xVg.png" /></figure><p>Now all we need is to write an out of band process to read email ids from our audit log and make a GRPC call to ReprocessEmail() for each email received that day and have it evaluated for possible forwarding; Crisis averted!</p><p>No need to hack together database queries, or wire up a one time un-tested code bit of code to preform all the processing. Instead we are using all of the existing code as is. We benefit from well defined abstractions organized by problem domain, which are thoroughly understood and tested through production use. The only new code we wrote to make this happen was in the GRPC method ReprocessEmail() and the script used to read the audit log and make calls to ReprocessEmail().</p><p>We gain a great deal of confidence our re-processing strategy will work through the re-use of existing interfaces. This would not be possible if we had applied KISS principles at the start and not taking the time to define domain driven interfaces, separated concerns, and assigned data ownership throughout our system. Congratulations, you now carry the title of “miracle worker” at the office; you’re welcome.</p><h3>Introducing new Versions</h3><p>Next, we will talk about introducing a new version of the forwarding product. But, as this post is already much longer than I originally planned, I’m breaking it up into separate posts.</p><p><a href="https://medium.com/@thrawn01/anatomy-of-a-product-handling-product-maturity-e976baa822c9">Anatomy Of A Product: Handling Product Maturity</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18b23eaac130" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You don’t know how to Cloud]]></title>
            <link>https://medium.com/@thrawn01/you-dont-know-how-to-cloud-5260f56d1fc6?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/5260f56d1fc6</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[efficiency]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Tue, 07 May 2024 16:24:28 GMT</pubDate>
            <atom:updated>2024-05-17T15:58:22.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EL3LJ2GQRh3_SCgN1hdhRg.jpeg" /></figure><p>Rejection of the cloud has been a popular topic as of late. This sentiment is only amplified by reports of cloud customers receiving seemingly inexplicable or exorbitantly expensive invoices from their cloud providers.</p><p>The prevailing narrative appears to be that the promise of cloud was somehow a mistake. That we all got suckered in by the “cloud Kool-Aid,” and those who aren’t embracing on-premises server racks are either intoxicated or delusional with cloud fever.</p><p>However, in my experience, the majority of cloud users simply don’t know how to properly leverage cloud computing. In essence,</p><p>Most people just don’t know how to cloud!</p><p>I helped build and scale <a href="https://mailgun.com/">Mailgun.com</a> which was known for having a very high profit to cost ratio. We achieved this by being cloud native, and keeping our costs low. I didn’t completely appreciate how rare this was, until we acquired a competitor of ours who was struggling to operate efficiently. As someone who has seen both efficient and inefficient cloud usage, I have a few choice words to say about how people should cloud.</p><h3>Always have an SLI &amp; Monitor it</h3><p>When it comes to ANY Software-as-a-Service (SaaS) cloud product that automatically scale, one should never develop on such a platform without first calculating or estimating expected usage, and then using that estimate to define a Service Level Indicator (SLI) which you then monitor for costs overruns and alert you when they exceed your estimated bounds.</p><p>Those estimated costs should be up on your dashboard. They are not nice to have, they are a requirement!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULyxwOlNF3rBwVB-OgPBpA.jpeg" /></figure><p><em>Screen shot from </em><a href="https://grafana.com/blog/2023/08/25/metrics-cost-management-in-grafana-cloud-guide/%5D"><em>Metrics Cost Management in Grafana Cloud</em></a></p><p>This is not as hard as it sounds, estimating lambda costs, network ingress and egress costs and essentially, anything that has an uncapped charge associated with it, should be step one of any design. Once you have that estimate in hand, use it to establish an SLO for the thing you are building and then monitor those for breaches.</p><p>GCP makes this relatively easy to do.</p><ul><li>Defining SLIs <a href="https://youtu.be/uhpAScSerec?si=eSaGlP_DRxDGlygn">https://youtu.be/uhpAScSerec?si=eSaGlP_DRxDGlygn</a></li><li>Creating Budgets <a href="https://youtu.be/F4omjjMZ54k?si=bvqjsyQiligDshpj">https://youtu.be/F4omjjMZ54k?si=bvqjsyQiligDshpj</a></li></ul><p>For an overview on compute estimation, I highly recommend this excellent repo and accompanying talk by <a href="https://twitter.com/sirupsen">Simon Eskildsen</a></p><p><a href="https://github.com/sirupsen/napkin-math">https://github.com/sirupsen/napkin-math</a></p><h3>Design for the cloud</h3><p>You MUST design your software for the cloud, forget any dreams of taking your existing product and moving it to the cloud and expecting magical cost savings. We’ve acquired companies that have tried to do this and it doesn’t work.</p><p>Cloud Native also means your software is designed to run on general purpose Compute. If your workloads require vertically expensive servers to run, forget it, you are likely better off running on-prem with a one-time, super expensive hardware purchase. Everything has to scale horizontally across cheap general purpose Compute, including your Databases.</p><h3>Databases</h3><p>Vanilla PostgreSQL and MySQL should be avoided — Note I said “Vanilla”.</p><p>You REALLY want a clustering capable Database which is designed to spread the load across many low-cost general purpose Compute and Disks. Traditional RDBMS are designed for highly optimized vertical scaling, typically in a Active-Passive HA configuration. This often results in big expensive servers multiplied by 3, (1 primary and 2 secondaries) as you scale. If you absolutely need an RDBMS (most people actually don’t) then use something that provides Active-Active HA, built-in Clustering and Sharding.</p><p>You REALLY want to separate your Compute and Storage workloads. This means you should avoid doing CPU intensive things on the Database. Instead, preform CPU based computational things in your app. Once the computation is done, you store the result on the DB in the form it will be retrieved. This allows you to scale your CPU bound processes quickly (because you don’t need any attached disk) which allows you to scale up during peak, and scale down during off peak.</p><p>Compute is more expensive than even the most expensive cloud disks. If your application is mostly crafting queries and waiting around for the database to compute a result, then your app is just a thin client for the database. Such designs are impossible to scale up or down on demand. Separating Compute and Storage saves you money and allows you to apply capacity management to compute and store separately.</p><p>There are PostgreSQL <a href="https://www.citusdata.com/">extensions</a> that provide much needed features on this front, and for MySQL; there is <a href="https://vitess.io/">Vitess</a>.</p><h3>Takeaways</h3><p>If you can’t or don’t want to commit to any of these things, then the cloud is not going to be as kind to your wallet as was promised.</p><p>The power of the cloud is in low cost general purpose hardware (“Commodity hardware” is the fancy phrase) that can be quickly spun up and spun down as needed. If your software is not designed to maximize this advantage then the promise of the cloud can never truly be realized.</p><p>For SaaS offerings like AWS Lambda, Aurora Serverless, etc., if your usage doesn’t have significant downtime (off-peak hours), you’re not going to see cost savings. If your app is running near 60% of capacity 24/7, you’ll almost always get more bang for your buck by self-hosting the database or application on a Compute instance under your control instead of a fully managed service.</p><p>I can’t stress this enough! The only way you’re going to see significant savings in the cloud is if you are scaling back during off peak. CPU’s are expensive, shut them down as quickly as possible.</p><p>This is the way; all others should invest in some server racks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5260f56d1fc6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preventing Foundation Movement via Landscaping]]></title>
            <link>https://medium.com/@thrawn01/preventing-foundation-movement-via-landscaping-de00e090f2c4?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/de00e090f2c4</guid>
            <category><![CDATA[irrigation]]></category>
            <category><![CDATA[landscape]]></category>
            <category><![CDATA[texas]]></category>
            <category><![CDATA[foundation]]></category>
            <category><![CDATA[drought]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Sun, 30 Jul 2023 20:33:49 GMT</pubDate>
            <atom:updated>2023-07-30T20:33:49.084Z</atom:updated>
            <content:encoded><![CDATA[<p>Alt: How to avoid soil pulling away from the foundation</p><h3>Moisture content effect on foundation</h3><p>The moisture content of the soil has a big impact on your foundation. When it’s hot and dry outside, the soil tends to contract and shrink. This can lead to gaps forming between the foundation and the soil. However, during wet or normal seasons, the soil expands and puts pressure on the foundation.</p><p>In order to maintain your foundation you must manage the moisture of the soil year round to avoid this expansion and contraction which will eventually lead to foundation movement and cracking requiring costly repairs. Here are some of the strategies I’ve used for maintaining the moisture content of the soil.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V3nL7rh3euTFdWyloXSiow.jpeg" /></figure><h3>Proper Drainage</h3><p>Depending on how your property is laid out, you might want to consider installing gutters or French drains to give that water a one-way ticket away from your precious foundation. Too much water in the soil can create abnormal expansion during wet seasons or heavy rain storms.</p><p>This was the case with my yard, the east face of the yard would retain water and become a muddy mess after rain. Installing a French drain at the edge of grass which emptied into to the green way behind my house completely transformed the area, allowing me to better manage the moisture content of that side of the house. The west side of my property had the opposite problem, as it was situated on a hill and water would wash away from the house. In addition that part of the house would get the brunt of the afternoon and evening heat thus further drying out the soil around the foundation.</p><h3>Irrigation Systems</h3><p>After consulting with an engineer, I was advised to have an irrigation system installed as an automatic way of maintaining the moisture of the soil around the foundation. In my case, both the east and west sides of the foundation had St Augustine grass butting up next to the foundation, with the west side drier and in direct sunlight. As such sprinkler heads were installed in separate zones (east and west) which allowed me adjust how long the watering cycle would be in order to water the dry side more than the wet side of the foundation. The irrigation systems did indeed have a positive effect and allowed me to improve soil moisture year around. However, in very dry seasons and during water restrictions (We occasionally have water usage restrictions in Texas during droughts) the soil would still pull away from the house even as I watered as often as the restrictions would allow.</p><h3>Thermal Expansion of the foundation</h3><p>Many foundations are exposed directly to the sun during the afternoon or evening hours. (Which is the warmest part of the day) The exposure can create thermal expansion and contraction of the foundation which over time can contribute to foundation movement. Planting shrubs and bushes next to the house in those areas can act as a sun barrier and reduce the amount of thermal expansion that occurs during these times.</p><h3>Rock beds</h3><p>Installing a rock bed with a landscape fabric base next to the house appears to have provided the best moisture protection. The white rocks help reflect sun and heat from drying the ground. In addition the lack of grass means water is not quickly evaporated through the leaves of the grass which reduces the rate at which water will evaporate from the ground. Landscaping fabric is porous, so it allows water to seep through, while protecting the moisture from evaporation during the dry season. My back yard patio would suffer from massive soil contraction during the summer months. Once I installed a rock bed with the Landscape fabric underneath the soil contraction almost completely stopped. The picture below was taken during the driest months of the summer. (I brushed away the rocks to expose the fabric underneath)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SPNgD3GbWkQt1QfjnRZo3Q.jpeg" /></figure><h3>Observations</h3><p>During extremely dry seasons, the Rock beds preformed much better than the areas with grass and irrigation systems. You can see from the picture below the difference a few feet of rock bed makes.</p><p>I believe this is due to the way grass and plants evaporate water for photosynthesis in the leaves. So, the more lush greenery you’ve got, the quicker that soil will lose its moisture. It’s like a double whammy of moisture evaporating from the soil, plus the grass drinking up moisture which compounds the drying-out effect, resulting in rapid dehydration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1_H2pfIE8aiaNti4mdxJIg.jpeg" /></figure><h3>Recommendations</h3><ul><li>If possible, create a 3–5 foot bed of rocks that butts up next the foundation all the way around the house. This barrier of Landscape fabric and rocks provides a powerful barrier against moisture loss.</li><li>Plant shrubs and bushes in the rock bed to provide extra barrier from direct sun light on the foundation which is exposed to the sun in the afternoon and evening. This avoids thermal expansion and contraction of the foundation which occurs every day.</li><li>Install an irrigation system to help regulate the moisture content of the surrounding property. (Including shrubs and bushes planted in the rock garden)</li><li>Install gutters or french drains to move water away from the foundation during rain storms.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de00e090f2c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Failure to take my own advice]]></title>
            <link>https://medium.com/@thrawn01/failure-to-take-my-own-advice-33990193c0c8?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/33990193c0c8</guid>
            <category><![CDATA[failure-stories]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Sun, 16 Apr 2023 16:00:28 GMT</pubDate>
            <atom:updated>2023-04-16T16:00:28.384Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I wrote a blog article entitled “Great Managers Ask Why.” However, just a few weeks later, I failed to follow my own advice. This failure to ask “why” led to the accidental destruction of internal accounts, which then required over 12 hours of time and energy to restore.</p><p>A coworker I’ve worked with for many years, whom I respect and trust, asked a seemingly harmless question:</p><p>“What’s the “right way” to run a script against prod? ….. I have 133 accounts I wanna delete, but I’m not sure how….”</p><p>At this point, the words “account” and “delete” should have caught my attention. However, I made the same mistake many others have made — I assumed the individual requesting this information had already performed the necessary due diligence before taking such drastic action. I gave no thought to how the accounts had been vetted, I assumed all potential repercussions had been considered.</p><p>In short, I neglected to stop and ask “why” we needed to delete these accounts. I didn’t delve into the rationale to gain knowledge and avoid assumptions. I paid so little attention, such that when our external API testing suite started paging developers, informing them that our entire site was down, it took me at least an hour to connect our authentication failures back to the account deletion request.</p><p>As I later found out, this failure to ask why was repeated multiple times — by the person requesting the account deletion, the person compiling the list of accounts to delete, the person executing the deletions, and finally, by me.</p><p>A whole day and evening of wasted time and effort could have been avoided if any one of us had stopped to ask “why”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33990193c0c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Great Managers Always Ask “Why?”]]></title>
            <link>https://medium.com/@thrawn01/great-managers-always-ask-why-c1cf84719113?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1cf84719113</guid>
            <category><![CDATA[leadership-development]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[management-and-leadership]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Fri, 31 Mar 2023 18:05:51 GMT</pubDate>
            <atom:updated>2023-03-31T18:05:51.592Z</atom:updated>
            <content:encoded><![CDATA[<p>As a manager, if a developer approaches you requesting a server, how do you respond? Do you provide a server immediately, log into the AWS console to create one, or do you ask “Why do you need a server?” Taking the time to understand the developer’s reasoning is crucial; perhaps they don’t need a server, but instead want to run a service in a container or deploy a serverless application.</p><p>Great managers consistently ask, “Why do you need this?”, “To what purpose?”, and “Is it in alignment with the rest of the organization?”</p><p>While most people are aware of what you do and how you do it, they often lack insight into why you do it. For a business organization, the simplistic answer to “why” is usually to make money. However, understanding the deeper “why” provides intuition about whether initiatives or requests will succeed or fail.</p><p>By grasping why the organization is investing its resources, pursuing certain goals, and striving to achieve them, you can, as a manager, infer whether an individual action will be approved or rejected even before requesting it.</p><p>Recognizing the “why” can offer insights into the actions and motivations of individuals, managers, and C-level and D-level executives. It enables you to discern which battles to fight and which to avoid.</p><p>When you correctly identify the “why” and engage in a battle that aligns with the organization, you can be confident that upper-level management will support you, as you understand their underlying motives.</p><p>Sources such as town halls, financial reports, profits, and EBITA can provide insights which help you navigate the complexities of prioritizing tasks and deciding which battles to fight. In a sense, understanding the “why” can serve as a guiding force, making others perceive you as someone with unique insights or a special connection to the organization’s leaders.</p><p>Moreover, when leadership sees that you comprehend what’s important to them, they will view you as an asset rather than someone they must constantly reject or correct in order to achieve their goals.</p><p>Asking “why” and understanding the motivations of individuals and organizations can be the key to unlocking future potential. It allows you to bring out the best in yourself, those around you, and the organization that employs you. Embracing the power of “why” can help you excel as a manager and better align with your organization’s objectives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1cf84719113" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HTTP/2 Cleartext (H2C) Client Example in Go]]></title>
            <link>https://medium.com/@thrawn01/http-2-cleartext-h2c-client-example-in-go-8167c7a4181e?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/8167c7a4181e</guid>
            <category><![CDATA[cleartext]]></category>
            <category><![CDATA[clients]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[2ch]]></category>
            <category><![CDATA[http2]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Wed, 05 Jun 2019 20:45:04 GMT</pubDate>
            <atom:updated>2019-06-05T20:45:04.827Z</atom:updated>
            <content:encoded><![CDATA[<p>Since my internet foo failed me, and the only workable example of a H2C client I can find was in the actual go test suite I’m going to lay out what I<br>discovered about H2C support in golang here.</p><p>First, incase you may be wondering what H2C is, It is essentially HTTP/2 but without TLS. H2C is understandably not widely publicized as HTTP/2 with TLS is more secure and not open to <a href="https://http2.github.io/http2-spec/#rfc.section.10.2">some types of attacks</a>. However H2C isn’t without it’s use cases, for instance GRPC uses H2C when you build a client with grpc.WithInsecure().</p><p>Second, is that the standard golang code supports HTTP2 but does not directly support H2C. H2C support only exists in the golang.org/x/net/http2/h2c<em> </em>package. You can make your HTTP server H2C capable by wrapping your handler or mux with h2c.NewHandler() like so.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a638bf5a3ef0852e6a1d1b4808053a6c/href">https://medium.com/media/a638bf5a3ef0852e6a1d1b4808053a6c/href</a></iframe><p>The above code allows the server to support <a href="https://http2.github.io/http2-spec/#discover-http">H2C upgrade</a> and <a href="https://http2.github.io/http2-spec/#known-http">H2C prior knowledge</a> along with standard HTTP/2 and HTTP/1.1 that golang natively supports.</p><p>If you don’t care about supporting HTTP/1.1 then you can run this code which only supports H2C prior knowledge.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7a3a6942b32dbb80924782f926dad49/href">https://medium.com/media/e7a3a6942b32dbb80924782f926dad49/href</a></iframe><p>Once you have a running server you can test your server by installing curl-openssl and use curl to test your H2C enabled server.</p><pre>$ brew install curl-openssl<br># Add curl-openssl to the front of your path<br>$ export PATH=&quot;/usr/local/opt/curl-openssl/bin:$PATH&quot;</pre><p><strong>Connect via HTTP1.1 then upgrade to HTTP/2 (H2C)</strong></p><pre>$ curl -v --http2 <a href="http://localhost:1010/">http://localhost:1010</a><br>*   Trying ::1:1010...<br>* TCP_NODELAY set<br>* Connected to localhost (::1) port 1010 (#0)<br>&gt; GET / HTTP/1.1<br>&gt; Host: localhost:1010<br>&gt; User-Agent: curl/7.65.0<br>&gt; Accept: */*<br>&gt; Connection: Upgrade, HTTP2-Settings<br>&gt; Upgrade: h2c<br>&gt; HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA<br>&gt;<br>* Mark bundle as not supporting multiuse<br>&lt; HTTP/1.1 101 Switching Protocols<br>&lt; Connection: Upgrade<br>&lt; Upgrade: h2c<br>* Received 101<br>* Using HTTP2, server supports multi-use<br>* Connection state changed (HTTP/2 confirmed)<br>* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0<br>* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!<br>&lt; HTTP/2 200<br>&lt; content-type: text/plain; charset=utf-8<br>&lt; content-length: 20<br>&lt; date: Wed, 05 Jun 2019 19:01:40 GMT<br>&lt;<br>* Connection #0 to host localhost left intact<br>Hello, /, http: true</pre><p><strong>Connect via HTTP/2 (H2C)</strong></p><pre>curl -v --http2-prior-knowledge <a href="http://localhost:1010/">http://localhost:1010</a><br>*   Trying ::1:1010...<br>* TCP_NODELAY set<br>* Connected to localhost (::1) port 1010 (#0)<br>* Using HTTP2, server supports multi-use<br>* Connection state changed (HTTP/2 confirmed)<br>* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0<br>* Using Stream ID: 1 (easy handle 0x7fdab8007000)<br>&gt; GET / HTTP/2<br>&gt; Host: localhost:1010<br>&gt; User-Agent: curl/7.65.0<br>&gt; Accept: */*<br>&gt;<br>* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!<br>&lt; HTTP/2 200<br>&lt; content-type: text/plain; charset=utf-8<br>&lt; content-length: 20<br>&lt; date: Wed, 05 Jun 2019 19:00:43 GMT<br>&lt;<br>* Connection #0 to host localhost left intact<br>Hello, /, http: true</pre><p>Now, Remember when I said that the golang standard library does not support H2C? While that is technically correct there is a workaround to get the golang standard HTTP/2 client to connect to an H2C enabled server.</p><p>To do so you have to override DialTLS and set the super secret AllowHTTP flag.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91751c25d5a1edcb7b358f2e4ce944f7/href">https://medium.com/media/91751c25d5a1edcb7b358f2e4ce944f7/href</a></iframe><p>Although this all looks a little wonky it actually works really well and<br>performs nicely in production environments.</p><p>A complete working example is available here <a href="http://github.com/thrawn01/h2c-golang-example">http://github.com/thrawn01/h2c-golang-example</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8167c7a4181e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should never test private methods]]></title>
            <link>https://medium.com/@thrawn01/why-you-should-never-test-private-methods-f822358e010?source=rss-6f2de1748b41------2</link>
            <guid isPermaLink="false">https://medium.com/p/f822358e010</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Derrick Wippler]]></dc:creator>
            <pubDate>Tue, 14 Mar 2017 13:05:53 GMT</pubDate>
            <atom:updated>2017-09-27T17:21:32.767Z</atom:updated>
            <content:encoded><![CDATA[<p>Traditional software development disciplines teach us to control public functions and interfaces exported by our code. As the exported code forms what is known as the public surface of the code package. It is through this public surface that we proclaim to our users which interfaces and functions are available for use.</p><p>Making an interface public proclaims two things</p><ol><li>The interface is well tested and approved for use.</li><li>The signature of this interface is guaranteed not to change for this major version.</li></ol><p>Good developers who are cognizant of their public surface usually write new functions private first, and are very explicit in exposing only a subset of functions public. Developers new to golang and which fall into this category are often appalled to discover they are unable to access these private functions when writing tests using the built in <strong>_test</strong> package. The typical response is to place the public test functions in the main package, and avoid the <strong>_test</strong> package all together. The result is a littering of the public surface with hundreds of test functions! Precisely opposite of what we wanted to accomplish in making our functions private!</p><p>So what are we todo?</p><p>We get clues by examining the golang standard library. Take the golang standard <strong>fmt</strong> package as an example. It’s public surface is very clean and well tested with 36 tests in the <strong>fmt_test</strong> package. Yet closer inspection reveals many private functions of varying degrees of complexity within the package! How does a package as well tested and widely used as <strong>fmt</strong> get away with zero tests on private functions? The answer might surprise you, as this simple rule is one of the hallmarks of great software design and good testing.</p><h3>Only test the public surface</h3><p>One of the promises of the public surface is that the interface is well tested, as such; the public surface should be the focus of all our tests. If the public surface exercises all possible execution paths within our public and private functions no private function testing is required. With thorough test coverage of the public surface, all possible code paths can be covered.</p><p>Put another way; if you’re testing private functions, you’re breaking the rules of encapsulation. If you are unable to test all code paths in a private function, it instead should be a public function, as private functions form the core of the encapsulation and should only provide code paths that satisfy the public surface needs. No matter how many ways you say it, the result is the same. Testing private functions defeats the purpose of encapsulation.</p><h3>More work and less benefit</h3><p>When writing private functions first; good practice is to write tests for the private functions simultaneously. This can result in less rigorous testing of the public surface, as the developer might reason “the private function is well covered, no need to duplicate the same test for the public function”, or at best the developer duplicates testing for both the public and private functions resulting in more work now, and in the future.</p><p>Recently I saw a pull request where the developer moved the existing tests out of the <strong>_test</strong> package and into the main package so he could add tests for a private function. He then neglected to add the same tests for the public function. This could not only leave a gap of untested code in the public function but also breaks encapsulation. Causing headaches for future pull requests that don’t realize the testing gap or that now have to deal with a tested private function that should have been a detail of the encapsulation, but is now apart of the testing suite.</p><p>Because the function is now apart of the test suite future code changes are likely to avoid removing it or changing it’s signature even if is in the best interests of the code base, purely on the basis that the function is already well tested. If changes <strong>must</strong> be made to the private function; the developer is forced to update the tests for the private function instead of testing the thing that actually might be effected by the change, the public surface.</p><p>If the initial developer correctly tested the public surface first, future pull requests should be judged successful if the public surface tests pass, regardless of what private functions were modified. As long as all code paths within the private functions are covered, the pull request can be judged a success. By following the “test public surfaces only” rule we can avoid extra work in the future, increase the amount of code covered and reduce test overlap.</p><h3>How do I ensure my private functions are covered?</h3><p>To ensure our public surface tests cover all our private functions, we can use code coverage tools. golang comes with a built in code coverage tool to identify un-exercised code paths. For example, we can see all the test coverage for private functions in the <strong>fmt</strong> package by running the following in a terminal. <em>(See </em><strong><em>fmt/format.go</em></strong><em> which contains many of the private functions)</em></p><pre>$ go test -coverprofile=coverage.out fmt<br>$ go tool cover -html=coverage.out</pre><p>This tool or one like it should be an important part of any CI setup.</p><h3>How do I avoid crowding the public surface?</h3><p>Inevitably you will write some general interfaces which are useful to many parts of our code, but are not strictly apart of the public surface you wish to present. These interfaces are perfect candidates for placement in a sub-package. In this way we create a testable public interface for our private code to use, but which is not strictly apart of our main packages public surface. golang supports sub packages naturally and can be a strong indicator to users which interface surfaces are specific to our package.</p><pre>github.com/thrawn01/package<br>github.com/thrawn01/package/utils<br>github.com/thrawn01/package/db</pre><p>go doc supports this naturally by generating documentation for the current package and ignoring sub packages unless specified directly. Granted this does not keep users from using the sub package interfaces; but the generated go docs can help steer users toward the correct interfaces.</p><p>A feature added in golang 1.5 is to name the subpackage<br><strong>github.com/thrawn01/package/internal</strong> which disallows external code to<br>import the subpackage <strong>package/internal</strong>.</p><h3>How do I know what interfaces to make public or private?</h3><p>In general, functions and interfaces which are tightly coupled to the calling function are good candidates for privatization; all others should be public.</p><p>When deciding if a function should be private or public it can be helpful to ask the following.</p><ol><li>Is this function generally useful in other parts of the code base? (make it a public/sub package)</li><li>Could this function be useful to others? (make it a public/sub package)</li><li>Is this function tightly coupled to the package? (make it private)</li><li>Can all code paths be covered by testing the public interface? (If yes make private else public)</li></ol><h3>Conclusion</h3><p>Separation of package code and package testing is a wonderful feature of golang which forces the developer to make sound choices about how their package is used and tested. Over my many years of C++ and Java development I can’t tell you how many times I’ve come across useful and reusable code that was thoughtlessly left private and thus inaccessible. I’ve been at companies where entire classes were copied 10+ times in different locations of a vast code base because the original author never imagined the class would be useful to any one but himself. Because of this, I write my code public first and make thoughtful decisions about how my code is used.</p><p>My hope is that this post will encourage others to make their code public first, use the golang <strong>_test</strong> package, and encourage more code reuse.</p><p><em>If you liked this, click the💚 below so other people will see this here on Medium.</em></p><p>#JobsPlug If you are interested in collaborating on a team with no dev managers and working for a company that empowers developers to focus on making great product outcomes, mailgun.com is hiring! <a href="https://mailgun.recruitee.com/">https://mailgun.recruitee.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f822358e010" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>