<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Dm03514 Tech Blog - Medium]]></title>
        <description><![CDATA[Dm03514’s Tech Blog - Medium]]></description>
        <link>https://medium.com/dm03514-tech-blog?source=rss----9135414d39d---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Dm03514 Tech Blog - Medium</title>
            <link>https://medium.com/dm03514-tech-blog?source=rss----9135414d39d---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:46:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dm03514-tech-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Python Algorithms — Implementing a FIFO Queue Using a Linked List]]></title>
            <link>https://medium.com/dm03514-tech-blog/python-algorithms-implementing-a-fifo-queue-using-a-linked-list-57cf700a6395?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/57cf700a6395</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Sun, 29 Aug 2021 19:37:41 GMT</pubDate>
            <atom:updated>2021-08-29T19:37:41.740Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python Algorithms — Implementing a FIFO Queue Using a Linked List</h3><p><a href="https://en.wikipedia.org/wiki/Queue_(abstract_data_type)">Queues</a> are commonly used to lineup work to be done. Consider a queue at a store. The first person in line is the first to be serviced. This is referred to as a “first in first out queue” (FIFO). This post details an implementation of a FIFO queue using a linked list. This post hopes to be a useful algorithm interview study reference. All code is available on github at <a href="https://github.com/dm03514/python-algorithms">dm03514/python-algorithms</a>.</p><h3>Queues</h3><p>Queues support the following methods:</p><ul><li><strong>Enqueue</strong>: Adds an item to the end of the queue.</li><li><strong>Dequeue</strong>: Removes an item from the beginning of the queue.</li><li><strong>Peek</strong>: Returns the item from the beginning of the queue, without modifying the queue.</li><li><strong>Size</strong>: Returns the length of the enqueued items.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sp65wUPTdbN_bDTn.png" /></figure><p>An abstract interface to support these methods looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6BuQbDCbvoZpxsVC.png" /></figure><h3>Time Complexity</h3><ul><li><strong>Space — O(n)</strong>: The queue requires space proportional to the size of the queue. 10 item queue will require 10 items worth of space, and a 75 item queue will require 75 items worth of space.</li><li><strong>Enqueue — O(1)</strong>: Adding an item to the end of a queue takes the same amount of time complexity regardless of queue size.</li><li><strong>Dequeue — O(1)</strong>: Removing an item to the end of a queue takes the same amount of time complexity regardless of queue size.</li></ul><h3>Implementation</h3><p>The core of the linked list based queue is the concept of the linked list. In a linked list each item (or node) contains a reference to the next item in the list:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*aKXIkf4Ok7bl_JjV.png" /></figure><p>This allows a list to be built up like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hu7Pm0z74_3w74fl.png" /></figure><p>The list above can be instantiated directly using the Node class, or built up over time. The following shows how to instantiate the linked list directly:</p><pre>Node(</pre><pre>  value=&#39;first&#39;,</pre><pre>  next=Node(</pre><pre>    value=&#39;second&#39;,</pre><pre>    next=Node(</pre><pre>      value=&#39;third</pre><pre>    )</pre><pre>  )</pre><pre>)</pre><p>The linked list does all the heavy lifting. The rest of the queue implementation is wrappers around the linked list and some record keeping to track the head, tail, and size of the linked list.</p><h3>Queue Record Keeping</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*omyl7Jn_3uxf6DpF.png" /></figure><h3>Enqueue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1vL7p00GxrqF5X6c.png" /></figure><h3>Dequeue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aLUaq5xbHwy6UmRm.png" /></figure><h3>Peek</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mq5MzQ0fWxCM47Xa.png" /></figure><h3>Size</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1ufJOqAKZl-qQLwd.png" /></figure><p>Thank you for reading!</p><p><em>Originally published at </em><a href="https://on-systems.tech/119-python-algorithms-fifo-queue-linked-list/"><em>https://on-systems.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57cf700a6395" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/python-algorithms-implementing-a-fifo-queue-using-a-linked-list-57cf700a6395">Python Algorithms — Implementing a FIFO Queue Using a Linked List</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Probes Partition the Debug Space]]></title>
            <link>https://medium.com/dm03514-tech-blog/how-probes-partition-the-debug-space-823c57a1009c?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/823c57a1009c</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-management-tools]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Sun, 30 Aug 2020 17:47:41 GMT</pubDate>
            <atom:updated>2020-08-30T17:47:41.737Z</atom:updated>
            <content:encoded><![CDATA[<p>Probing is a technique to perform regular checks on a service using a short interval. Probes provide signals that can significantly cut down debug time. This post describe probes and how they can be used to drill down into errors and make debugging more focused; how they can partition the debug space.</p><h3>Probes</h3><p>Probes are targeted checks, performed as request / response actions, on a short (~1 minute or less) interval. Some common applications of probes are:</p><ul><li><strong>Uptime Probes: Internal Debugging</strong> — The focus of this post. Probes make a binary Yes/No determination of if a service is functioning as expected.</li><li><strong>Load balancer health checks</strong> — <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html">Amazon ELB defaults to 30 second health checks</a></li><li><strong>Uptime Probes: Status Pages</strong> — Feed uptime data for presentation to end users.</li><li><strong>Uptime Probes: SRE</strong> — <a href="https://medium.com/dm03514-tech-blog/sre-strategy-deploying-slos-across-an-organization-59906c0d8f5">Provides SLI/SLO data</a>, related to status pages above.</li></ul><p>Probes are a critical component of status pages and customer facing health metrics. Companies often use probes to provide Yes|no down up determinations as the data for their status pages. Think of <a href="https://www.pingdom.com/">Pingdom</a> or a load balancer which requests a website every minute to determine if the site is up or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nu3VAGBmL4Gip172YaQ0yQ.jpeg" /></figure><p>(Pingdom status page. Source [Pingdom](https://www.pingdom.com/))</p><p>Common solutions to implement probes are:</p><ul><li>SAAS (<a href="https://www.pingdom.com/">Pingdom</a>, <a href="https://docs.datadoghq.com/integrations/http_check/">DataDog</a>)</li><li>Open-source (<a href="https://medium.com/dm03514-tech-blog/sre-availability-probing-101-using-googles-cloudprober-8c191173923c">Google CloudProber</a>)</li><li>Built-in to Load Balancers/Cloud Components</li><li>Homegrown probing on an interval</li></ul><h3>Output</h3><p>Probes are most effective when they return a binary yes|no result and the latency to achieve the result. Since probes are synthetic the expected results are known before hand. The follow describes a prober to check backend health from a load balancer:</p><ul><li>Protocol HTTP</li><li>Target Endpoint /health</li><li>Success: 200 Status code</li></ul><p>The above probe defines success as 200 and failure as any other status code. The status code in this case isn’t important, only the binary success|failure determination.</p><p>Graphing the probe above would look like a timeseries of invocations with their success and failures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVHLUyv1tMoKW4-TB-_b9A.png" /></figure><p>If you’re wondering why the graph above isn’t uniform, it’s because the metrics are being reported by the system at delayed intervals. The chart below shows the actual prober executions are uniform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcLf5dz_pFn0v6zxlJHWcQ.png" /></figure><p>It’s also common to see probes aggregated and expressed as a ratio between success and failure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_VgTU-FP-Yt8OsRFZ3nI7w.png" /></figure><p>Probe results can be rolled up over larger intervals to calculate aggregate availability. The following examples shows the aggregate availability over 7, 30 and 90 days:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LonXGQz6hJZGW4EwcBfWxA.png" /></figure><p>Probes are very simple, they contain the following properties:</p><ul><li>Protocol</li><li>Target</li><li>Expected response</li><li>Interval</li></ul><p><a href="https://github.com/google/cloudprober/blob/master/probes/proto/config.proto#L1">Google CloudProber contains an example of the config required</a> for a production-ready self hosted solution. These simple operations are the building blocks to determining if services are up or down, client reporting, debugging, reliability, and load balancer targets, among others.</p><h3>Debugging Using Probes</h3><p>The binary success|failure output probes generate make them well suited to debugging. Probes can be used to partition of the debug space along critical &amp; common inflection points. Imagine a service with external customers, the customers complain that something is wrong. An effective debugging workflow takes an huge or unbounded debug space and narrows it down:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qtS2dt0i6O3tk6So-6vOg.png" /></figure><p>The goal of debugging and the flowchart above is to take a large problem space, “Service is not working” and drill down into the causes. With each question the debug space gets smaller and more targeted. This can be visualized as a bounded space, where each question partitions the space in half. Good debug questions will binary search the space along critical inflection points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*2ShhSgEGk05PrxyWVfeY2Q.png" /></figure><p>Probes can be used to test these inflection points and make it trivial to dig into the problem space. To partition the debug space using the questions above only a single probe is needed. This probe needs to measure success and latency.</p><p>A single probe per service significantly partitions the debug space using just availability and latency.</p><p>A prober per service may seem like a lot of moving parts but I have found probes to be extremely easy to maintain. The requirements to execute them are pretty small. Their focused nature makes it trivial to write probe logic and the fact that they execute on an interval makes them easy to operate.</p><p>Probes are a primitive operational tool and are critical for debugging. I find systems with probes are easier to debug and determine if they are running. Probes also provide information to external stakeholders on system health.</p><p>Happy Probing!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=823c57a1009c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/how-probes-partition-the-debug-space-823c57a1009c">How Probes Partition the Debug Space</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Logging: JSON’s Not Enough]]></title>
            <link>https://medium.com/dm03514-tech-blog/scaling-logging-jsons-not-enough-f6920cc2788e?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/f6920cc2788e</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Sun, 07 Jun 2020 18:01:43 GMT</pubDate>
            <atom:updated>2020-06-08T01:08:27.694Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4XDSt8By0hZ_NKGDeLRr3A.jpeg" /></figure><p>JSON is the de-facto logging standard. JSON is so ubiquitous that the popular logging data tools (such as Elasticsearch) accept JSON by default. Although JSON is an evolution over previous logging standards, JSON’s lack of strict types make it insufficient to use for long-term persistence or as a foundation for a data-lake. This post describes the problem with JSON and proposes a solution using a strictly typed interchange format such as Protocol Buffers.</p><h3>The Trouble With JSON</h3><p>JSON logs establish an explicit structure. JSON parsers are available in most languages which make it accessible as a log standard. JSON logs are referred to as “structured” logging. When applications log as JSON it means that they emit fully formed JSON to their log sinks. Common sinks are standard out/error, files, syslog etc. These sinks are commonly connected to a daemon such as flunetd which routes the logs to long-term storage such as S3 or Elasticsearch.</p><p>The application emits JSON through its logging library. An example could look like:</p><pre>{<br>    message: &quot;an important event&quot;,<br>    client_id: 1,<br>    timestamp: &quot;2019-02-04T12:23:34Z&quot;,<br>    pid: 12121,<br>    http: {<br>        request: {<br>            path: &quot;/some-path&quot;<br>        }<br>    }<br>}</pre><p>Structured logging is a solution to address the previous era of logs, which are referred to as “unstructured”. Unstructured logs are commonly logged as a single line and require custom parsers (regex) for each different log type. Web server access logs are one example of unstructured logs. Th3 following shows an apache log line:</p><pre>127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] &quot;GET /sample-image.png HTTP/2&quot; 200 1479</pre><p>Parsing this log may require a different solution than parsing an nginx vs varnish vs haproxy vs etc. Clients of unstructured logs need N separate parsing methods, one for each log type. A client of JSON needs a single parser for all logs. A JSON consumer using python can parse <em>any</em> structured log using the same implementation:</p><pre>import json<br><br>log = json.loads(log_line)</pre><p>Unfortunately, parsing is one of many steps in making logs usable. Logs need to be transformed, aggregated, validated, and stored/loaded. JSON has multiple properties which make it a poor candidate as a logging protocol.</p><h4>Unenforceable Structure</h4><p>Even though JSON is more structured than a text line, JSON does not encode the full structure. This makes logs “schema-on-read”. Applications can send any JSON log, but it’s the readers responsibility to parse and use that log. And it’s not until reading the log that the schema (the full structure of the log) is used. Consider an application that logs:</p><pre>{<br>    http: {<br>        request: {<br>            path: &#39;/test&#39;<br>        }<br>    }<br>}</pre><p>A consumer that counts paths:</p><pre>import collections<br>import json</pre><pre><br>paths = collections.defaultdict(int)<br><br>log = json.loads(log_line)<br>paths[log[&#39;http&#39;][&#39;request][&#39;path&#39;]] += 1</pre><p>With schema on read the structure is only important to the reader of the data. There is no feedback on the writer. The application logging the JSON can log anything and it’s up to the consumer of that data to apply a “schema”. The full structure only becomes important when the log is read.</p><p>Consider what happens when another upstream application logs a different format:</p><pre>{<br>    http: {<br>        request: &#39;http://www.test.com/test&#39;<br>    }<br>}</pre><p>This log breaks the log consumer! The burden of consuming this log falls on the consumer! What often happens is the consumer is updated to accommodate this new log format:</p><pre>import json<br>from urllib.parse import urlparse<br><br>paths = collections.defaultdict(int)<br><br>log = json.loads(log_line)<br><br>try:<br>    paths[log[&#39;http&#39;][&#39;request][&#39;path&#39;]] += 1<br>except KeyError:<br>    u = urlparse(log[&#39;http&#39;][&#39;request&#39;])<br>    paths[u.path] += 1</pre><p>Schema on read becomes unmanageable as the number of producers increases. Imagine if there were 5 different structures? 10? 50? In large organizations and/or microservices, one team may be responsible for log ingestion but many teams are responsible for log production. Even with shared tooling such as logging libraries it’s still possible to produce a variety logs due to the lack of feedback (missing schema-on-write) on the producer side.</p><h4>Unknown types</h4><p>Not only is the structure not known until read but the data types are not known either. Consider 3 different log producers:</p><p><strong>Producer 1</strong></p><pre>{<br>    age: 1<br>}</pre><p><strong>Producer 2</strong></p><pre>{<br>    age: &#39;1&#39;<br>}</pre><p><strong>Producer 3</strong></p><pre>{<br>    age: null <br>}</pre><p>Even though the structure of each log is the same, the data types change. The consumer needs to account for different structure and different datatypes because of schema on read.</p><h4>Feedback loops</h4><p>Schema on read creates very long feedback loops for log producers. It’s not until the data is read by a downstream consumers validates whether the schema is valid and the usability of the log. Consider a common logging architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*Rqnb_-D-qZCXV-fIcTiENA.png" /></figure><p>In this setup:</p><ul><li>The application logs to syslog or a file</li><li>fluentd reads from that file or syslog</li><li>Fluentd sends the logs to a log stream</li><li>The logs are processed from the log stream and are put in Elasticsearch</li></ul><p>This log isn’t validated until the consumer tries to use it, due to schema on read. This means that the engineer developing the application sends what they think is a valid log but doesn’t get feedback on its validity later in the log stream. How does an engineer know if their log is failing? Where do they go to figure it out? How do they observe fluentd? The stream? The stream consumer? Elasticsearch. JSON logs using schema on read are difficult to debug.</p><h3>Solving Logging-&gt; Schema on Write</h3><p>An explicit schema combined with schema on write must be used in order to make log consumption sane. The technologies commonly used for explicit data schemas are:</p><ul><li>Protocol Buffers</li><li>Avro</li><li>Thrift</li></ul><p>The rest of this article will focus on Protocol Buffers. Protocol Buffers requires defining a schema and generating a language specific bindings for that schema. An example log schema may look like:</p><pre>syntax = &quot;proto3&quot;;<br><br>package log;<br><br>message Log {<br>  required string message = 1;   <br>  required string level = 2;   <br>}</pre><p>A language specific binding needs to be created for each log producer. Afterwards the log producer can generate a protobuf Log object and then serialize that object. The producer receives feedback on the log, structure and types during serialization. Explicit schemas move logs from schema on read to schema on write!</p><p>The consumer needs a reference to the Log message using the python specific language binding, which is generated using <a href="https://developers.google.com/protocol-buffers/docs/pythontutorial">Protocol Buffer standard library</a>. Afterwards, python can parse the log line directly to a Log message.</p><pre>from google.protobuf.json_format import Parse<br><br>import log_pb3<br><br>log = log_pb3.Log()<br><br>Parse(log_line, log)<br><br># log.message<br># log.level</pre><p>Protocol Buffer provides a fully validated log structure and strict field types!</p><h4>Wire Protocols</h4><p>Protocol Buffers ship with their own binary serialization format. Most Protocol Buffer client libraries (python, node, go, etc) support serializing and deserializing between json and Protocol Buffers. This means they can parse a JSON message into a Protocol Buffer, and output JSON from a Protocol Buffer. This reduces the friction involved in introducing Protocol Buffers into a JSON logging pipeline.</p><h3>Conclusion</h3><p>JSON and schema on read are not sufficient for building multi service log processing systems. JSON’s lack of explicit structure leads to complexity for the log consumers. An explicit schema such as Protocol Buffers, which supports schema on write can address JSON’s issues. Protocol Buffers enable fast feedback for clients on the validity of logs and provides consumers with the full object structure and datatypes of the log, before consuming. These properties are especially important for data lakes where raw logs may be materialized into multiple different systems for consumption. Without an explicit schema JSON quickly becomes untenable for these use cases.</p><h3>Links</h3><ul><li><a href="https://medium.com/dm03514-tech-blog/scaling-services-explicit-data-contracts-using-protocol-buffers-5ce7041347e6">https://medium.com/dm03514-tech-blog/scaling-services-explicit-data-contracts-using-protocol-buffers-5ce7041347e6</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6920cc2788e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/scaling-logging-jsons-not-enough-f6920cc2788e">Scaling Logging: JSON’s Not Enough</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Love AWS Lambda (and Think Serverless is the Future)]]></title>
            <link>https://medium.com/dm03514-tech-blog/why-i-love-aws-lambda-and-think-serverless-is-the-future-532f3ca8c843?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/532f3ca8c843</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Tue, 26 May 2020 02:44:58 GMT</pubDate>
            <atom:updated>2020-05-26T15:04:49.690Z</atom:updated>
            <content:encoded><![CDATA[<p>Since February I’ve been working on an asynchronous AWS Lambda service processing 60,000 events / second from Kinesis. Lambda provides minimal operational overhead, fast deploys, predictable pricing, and enforces many tenants of <a href="https://12factor.net/">12-factor apps</a>, out of thee box. After working with Lambda almost daily for the past 3 months, I’m convinced it’s the future for asynchronous processing, and here’s why.</p><h3>Outsourcing</h3><p>Lambda removes complexity over stateful deployment models. This minimizes the surface area service engineers are responsible for. Lambda removes the need for system alerts, instance sizing, autoscaling, provisioning servers, and choosing process monitors, amongst others. There’s no SSH’ing into instances since Lambda manages them. Lambda removes infrastructure reducing the amount a company needs to manage and reason about. In VM or container based deployments service teams are responsible for keeping services up and running 24x7 365. With Lambda, scheduling and execution becomes Amazon’s responsibility. A companies responsibility is minimized to ensure that their Lambda functions work correctly, and can keep up with throughput, and costs are bound.</p><p>There’s an opportunity cost with keeping processes running. Time spent on instance maintenance and process upkeep is time <em>not</em> spent on other work. Although I don’t have any concrete metrics on the break down of time in a serverless vs non-serverless systems, but I would have expected to SSH into a system some amount of times in the last 3 months, especially for a system of this level of throughput. Other sources of time saved are:</p><ul><li>provisioning servers</li><li>reproducing environments locally</li><li>monitoring processes</li></ul><p>All of these result in time saved when using Lambda. Anecdotally, I feel like I’ve saved 10–15% of time in just not having to manage servers, concurrency, scaling, alerting, OS upgrades, integrations, etc.</p><h3>Reliability</h3><p>Amazon is better at running distributed systems than most of the the industry. They devote a huge amount of resources and budget to ensuring they are. Lambda minimizes the surface area of components that can fail. It would be naive to think that Amazon doesn’t go down because it does, but there are just less moving parts on a Lambda consumer’s end. Lambda shifts the operational complexity to Amazon, who’s core competency is to execute Lambdas reliably. Additionally, Amazon has more resources to mobilize when there are customer facing issues.</p><p>Outsourcing infrastructure to Amazon also changes the dynamic of the issues that customers encounter. Instead of servers, processes and scaling, it shifts issues to a higher level — closer to the business domain. Lambda shifts the failure domain up a level of abstraction to one that is closer to the business and the consumer’s domain.</p><h3>Predictable Pricing</h3><p>The service I work with is cheaper to run on Lambda than EC2. Lambda is especially good for bursty traffic since it can idle down to 0 resources. Even if it’s not cheaper in all cases, Lambda is more predictable and simple. Lambda billing is based on the number of invocations and the duration of each invocation. Measure how long the Lambda takes on average, and multiple that by the projected throughput. Amazon also supports <a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html">“Provisioned Concurrency”</a> which keeps a certain number of Lambdas ready for execution, which adds another dimension to price.</p><p>I have found this to be significantly simpler than cost projections using long lived resources. Accurately performing cost projections on long lived resources requires benchmarking in a prod-like environment to understand throughput. It requires correct instances sizing and monitoring of system level resources, which is time consuming and a largely manual process. Any changes to instances sizing, or shifting workloads necessitates repeating this process! The simplicity and predictability of pricing makes it trivial for anyone to perform Lambda price projections at the beginning of projects, and for the company to understand how costs will scale with traffic.</p><h3>Deployment velocity</h3><p>Lambda deployment only requires:</p><p><strong>Package a Lambda function as a zip file and put it in a known S3 location</strong></p><p>That’s it. The next Lambda invoked will use the most recent s3 zip file. Any current Lambdas will finish their invocations and then use the new code. This enables rapid feedback. It also makes rollbacks trivial; just put a previous version of the zip file in s3. In my experience deploying a lambda takes &lt; 1 minute!</p><h3>Forced Structure — Development Velocity</h3><p>Lambda imposes a number of constraints in order to execute a lambda on AWS. Lambdas must:</p><ul><li>Be configured through environmental variables</li><li>Accept known structured payloads and require a specific call signature</li></ul><p>This enables quick development velocity. The Lambdas I work on use Golang and Amazon provides strict types for all Lambda inputs. Tests use well defined contracts and receive the exact same input and types that Amazon provides at runtime. Verifying a function locally using a unit test provides a very high level of confidence the function will work on Lambda. The failure domain for deploys is minimized to:</p><ul><li>Environmental configuration</li><li>IAM permissions</li></ul><p>I have found that these failures are solved a single time and then are pretty much stable.</p><h3>Conclusion</h3><p>Lambda has been revolutionary and has enabled rapid iteration and deployment, while removing a whole class of failure domains. I’ve found lambda to be: reliable, predicable, simple, structured, and cost effective. Lambda is the closest solution to my idea developer experience. Even with the potential for function sprawl, I think it’s a net positive for the industry and can’t wait to see what companies do with it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=532f3ca8c843" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/why-i-love-aws-lambda-and-think-serverless-is-the-future-532f3ca8c843">Why I Love AWS Lambda (and Think Serverless is the Future)</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go: AWS Lambda Project Structure Using Golang]]></title>
            <link>https://medium.com/dm03514-tech-blog/go-aws-lambda-project-structure-using-golang-98b6c0a5339d?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/98b6c0a5339d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Sat, 16 May 2020 21:45:12 GMT</pubDate>
            <atom:updated>2020-05-16T23:56:20.497Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS Lambda is a serverless solution which enables engineers to deploy single functions. AWS Lambda handles orchestrating, executing, scaling the function invocations. It’s important to structure go lambda projects so that the lambda is a simple entry point into the application, equivalent to cmd/. After a project is structured, it important to keep logic outside the lambda, which allows for easy reuse and testing of the application logic. The following are a series of steps which can be used in Go based lambda projects to help keep projects structured and increase the testability of lambda-based projects.</p><h3>Structure</h3><p>In Go, it’s common to see a <a href="https://github.com/golang-standards/project-layout#cmd">cmd/ directory</a> which contains CLI entry points into an application. Using a test project with 2 separate apps, the layout appears as:</p><pre>$ tree test-go-lambda/<br>test-go-lambda/<br>└── cmd<br>    ├── app1<br>    │   └── main.go<br>    └── app2<br>        └── main.go</pre><p>It’s helpful to take the same approach with lambdas:</p><pre>$ tree test-go-lambda/<br>test-go-lambda/<br>├── cmd<br>│   ├── app1<br>│   │   └── main.go<br>│   └── app2<br>│       └── main.go<br>└── lambda<br>    ├── lambda1<br>    │   └── main.go<br>    └── lambda2<br>        └── main.go</pre><p>All lambdas live in the lambda directly, and each directory within lambda contains a single lambdas main command. In the example above there are two lambdas, lambda1 and lambda2. Each contains a main.go file with a main command which can be executed by AWS lambda&#39;s go runtime. The benefits to this convention are the same as the cmd/ convention:</p><ul><li>Makes it easier to inventory entry points into the code base</li><li>Helps to reduce onboarding friction</li><li>Provides a structured convention which makes builds easier</li></ul><p>This convention simplifies build tooling by providing a single location for all lambdas. It’s trivial to package all lambdas in the same .zip or generate a .zip per lambda.</p><h3>“Thin” Lambdas</h3><p>“Thin” lambdas delegate to other functions. I like to structure it so that a lambda delegates to a single function. <a href="https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code">AWS Lambda documentation</a> recommends this as a best practice:</p><blockquote>Separate the Lambda handler from your core logic.</blockquote><p>To achieve this, lambdas can delegate to a single domain logic entry point. Each lambda will:</p><ul><li>Parse the environment and initialize a domain specific config in main</li><li>Initialize domain logic in Handler</li><li>Delegate to the domain logic and return an error</li></ul><p>Which looks like:</p><pre>func Handler(ctx context.Context, e events.CloudWatchEvent) error {<br>    var conf Config<br>    // initialize conf<br>    doer, err := domain.NewDoer(conf)<br>    if err != nil {<br>        return err<br>    }<br>    return doer.Do(ctx, e)<br>}<br><br>func main() {<br>    lambda.Start(Handler)<br>}</pre><h3>“Lambda” Structs for Configuration</h3><p>AWS Lambda can reuse execution environments for individual lambdas, which means that AWS may keep handlers alive and invoke them multiple times. Resources, like database connections, initialized outside of the handler function can be maintained for multiple handler invocations! I like to call these main scoped resources, opposed to handler scoped resources. I like to setup each lambda main.go file to have a structure:</p><pre># lambda/x/main.go<br><br>type Lambda struct {<br>    Conf domain.SpecificConf<br>}<br><br>func (l Lambda) Handler(...) error {<br>    // handler scoped l.Conf configuration<br>    doer, err := domain.NewDoer(l.Conf)<br>    if err != nil {<br>        return err<br>    }<br>    return doer.Do(ctx, e)<br>}<br><br>func main() {<br>    // initialize resources<br>    // build conf<br>    l := Lambda{<br>        Conf: domain.SpecificConf{<br>            // main scoped configuration<br>        }<br>    }<br><br>    lambda.Start(l.Handler)<br>}</pre><p>Examples of main scoped configuration are:</p><ul><li>Environmental Variables</li><li>Session i.e. AWS-SDK sessions and clients</li><li>Database connections (redis, mysql, postgres, elastic search, etc.)</li></ul><p>Examples of handler scoped configuration:</p><ul><li>Times / Timing</li><li>AWS Lambda Payload / Parameter based configuration</li></ul><p>Each lambda is configured through its environment. I like to use <a href="https://github.com/joeshaw/envdecode">envdecode</a> to handle parsing the environmental variables in the main function:</p><pre>func main() {<br>    // initialize resources<br>    // build conf<br>    conf := domain.SpecificConf{<br>        // main scoped configuration<br>        // db connections, aws-sdk sessions, etc<br>    }<br><br>    // pull in environment based config<br>    if err := envdecode.Decode(&amp;conf); err != nil {<br>		panic(err)<br>	}<br><br>    l := Lambda{<br>        Conf: conf,<br>    }<br><br>    lambda.Start(l.Handler)<br>}</pre><h3>Expose SQS Interface on Cron Lambdas</h3><p>One great use case for lambda is “cron” based workloads (through cloudwatch events). This is where AWS executes a lambda function on a fixed schedule. Many of these include a dead letter queue for failed messages and a lambda that consumes from dead letter queue. Since the messages in the dead letter queue contain the original message the dead letter queue lambda is usually very similar to the original lambda in terms of configuration and resource dependency. What does change is that the dead letter queue messages have a different structure, <a href="https://github.com/aws/aws-lambda-go/blob/3574a030467ca312f135e64a4a1bf293a216febd/events/sqs.go#L5">events.SQSEvent</a>.</p><p>One way to handle this is to expose a new SQSHandler which delegates to the original:</p><pre>type LambdaConf struct {<br>	HandlerType string `env:&quot;HANDLER_TYPE,default=cloudwatch_event&quot;`<br>}<br><br>type Lambda struct {}<br><br>func (l *Lambda) SQSHandler(ctx context.Context, sqsEvent events.SQSEvent) error {<br>    var e events.CloudWatchEvent<br>    for _, record := sqsEvent.Records {<br>        if err := json.Unmarshal([]byte(record.Body), &amp;e); err != nil {<br>            log.Printf(&quot;event: %+v\n&quot;, record)<br>            return fmt.Errorf(&quot;unable to parse event into CloudWatchEvent: %s&quot;, err)<br>        }<br><br>        // delegate to l.Handler(ctx, e)<br>    }<br>}<br><br>func (l *Lambda) Handler(ctx context.Context, e events.CloudWatchEvent) error {<br>    doer, err := domain.NewDoer()<br>    if err != nil {<br>        return err<br>    }<br>    return doer.Do(ctx, e)<br>}<br><br>func main() {<br>    l := Lambda{}<br><br>    lambdaConf := LambdaConf{}<br>    if err := envdecode.Decode(&amp;lambdaConf); err != nil {<br>		panic(err)<br>	}<br><br>    if lambdaConf.HandlerType == &quot;sqs_event&quot; {<br>		lambda.Start(l.SQSHandler)<br>	} else {<br>		lambda.Start(l.Handler)<br>	}<br>}</pre><p>The SQSHandler delegates to the Handler in order to reduce duplication and keep the handlers &quot;thin&quot;.</p><h3>Conclusion</h3><p>If careful attention isn’t paid to go-based lambdas, projects risk creating untestable, hard to work with lambdas. Placing lambdas in their own directory makes lambda discovery and builds easy. Minimizing business logic inside of lambdas makes it easier to isolate logic for unit tests. I’ve found that projects that follow the above structure are much easier to understand, test, build and extend. Happy Hacking!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98b6c0a5339d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/go-aws-lambda-project-structure-using-golang-98b6c0a5339d">Go: AWS Lambda Project Structure Using Golang</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Lasting Change in Organizations By Modifying Behavior]]></title>
            <link>https://medium.com/dm03514-tech-blog/creating-lasting-change-in-organizations-by-modifying-behavior-473cfa2ea1e7?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/473cfa2ea1e7</guid>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[life-lessons]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Thu, 02 Jan 2020 18:42:29 GMT</pubDate>
            <atom:updated>2020-01-02T19:58:47.071Z</atom:updated>
            <content:encoded><![CDATA[<p>Creating sustainable long lasting change in human systems is only achievable though modifying human behavior. Anyone that’s worked with people should understand the difficulty in this task. One common method for enforcing change is command and control, which often leads to hostility and resentment. The other end of the spectrum, No control or oversight, can be equally as damaging, leading to a free-for-all or local fiefdoms emerging. This article describes a middle ground method to creating change through enhanced information management and by carefully designing system constraints.</p><p>This strategy is frequently seen in tech and especially relevant to distributed organizations. It is achieved through:</p><ul><li><strong>Informing</strong> through structured data</li><li><strong>Alerting</strong> of actions when they go out of bound</li><li><strong>Regulating</strong> the set of possible allowed actions</li></ul><h3>Inform</h3><p>Inform structures and surfaces information to users</p><p>Informing requires understanding which metrics are important, where to instrument them, actually instrument them and then surface them through dashboards .</p><p>Informing is the foundation of change. Once data is available it makes it possible to empirically understand the problem, and profile the problem. Many times humans think there is a problem based on any number of common biases but when data is finally available it tells another story.</p><p>Informing helps to answer: what’s the state of the system right now? Informing is required for later steps in order to measure the affects of any excitements. It’s important that information is surfaced in a system that supports alerting, and programmatic querying.</p><h3>Alert</h3><p>Alerting notifies a user in response to performing some action. The purpose of alerting is to establish a feedback loop between a person and actions that they take in order to help people understand the affect of their actions. Most alerts are setup to take place on certain important thresholds:</p><ul><li>“You submitted 3 Reviews Last week! Keep Up the Good Work!”</li><li>A gas light has come on</li><li>“The cost of this action costs $X”</li></ul><p>Alerts help establish consequences for actions, by actively notifying individuals of the affects of their actions. Without alerts it would be the individuals responsibility to consult the metrics surfaced through the <strong>Inform</strong> stage.</p><h3>Regulate</h3><p>Regulation influences behavior by controlling (limiting) the number possible actions available to people. Regulation is frequently applied in traditional law and often has established consequences for certain sets of actions. In tech there are many ways to regulate actions without making explicit rules like “Thou shalt not ____”. Explicit prohibition is absolutely effective in some cases: lead paint and CFCs, and many other laws.</p><p>Examples of regulation in tech are:</p><ul><li>Builds with failing tests are not promoted</li><li>Build system only operates on Docker Images</li><li>Service Scaffolding is created through a shared tool</li></ul><p>It’s really important to stress how profound and subtle the affects of Regulation can be. Regulation influences systems in its own ways, especially when there are competing concerns. Consider a team that is being incentivized to deliver but encounters failing builds. Without sufficient checks and balances throughout the process, one option is to remove the failing tests in order to get the build to “pass”!</p><h3>Example — Controlling Cloud Costs</h3><p>In order to illustrate this in action consider the case of a tech organization with uncontrolled infrastructure spending. For this example, consider a company that uses a cloud provider like AWS. They have a global bill showing increasing AWS costs and would like to keep the costs from growing out of control.</p><h4>Inform</h4><p>Informing requires surfacing data and presenting it in a structured human consumable way. For cloud costs this may involve tagging resources by key dimensions such as:</p><ul><li>Team</li><li>Service</li><li>Resource Type</li><li>Resource Size</li></ul><p>This information then has to be saved (surfaced) somewhere that supports creating actionable alerts. This may be a data warehouse, it may be a regular CSV dump from the cloud provider and load into a metric system. Surfacing may take place asynchronously through web hooks when resources are created.</p><p>Once data is surfaced to a metric store, a dashboard can be created in order to drill down costs by any supported dimension. This allows end users to see the cost of their resources. It also allows for profiling of the highest cost contributors and seeing the affect of different cost controlling experiments.</p><h4>Alert</h4><p>There are many different strategies for alerting on cloud costs. An initial strategy may be to email or slack a weekly report to each team of their actual costs for the last week. There may be an alert when costs go above a certain threshold in an interval. More intelligent alerts could learn “normal” usage for a certain service or team, and then send out alerts when cost deviates over that norm.</p><p>It’s important to consider the cadence of alerts in order to prevent alert “fatigue”, but also to keep stakeholder and humans informed of their actions. This creates a spectrum: One end with no alerts ie humans have no automated notifications of the impact they are having on costs through the resources they are using. On the other end of the spectrum humans are alerted of every single action they are making. An acceptable solution that can be used to inform and affect human behavior usually sits somewhere in the middle. Targeted weekly rollups and high threshold based alerts are usually a good place to start.</p><h4>Regulate</h4><p>There are many different ways to regulate cloud resource costs, each having their own tradeoffs in terms of throughput and latency. A naive approach may be to only allow key members the ability to provision or approve requests for resources. This requires that all resource requests be approved and enables the organization to tightly control cloud costs. The cost control that this method offers comes with serious throughput and latency tradeoffs, as it inserts a person for manual review.</p><p>Another way to regulate costs is to grant each team a static fixed amount of resources for certain intervals. It’s then up to each team to use those resources effectively. A logical extension of this is to give each teams a fixed budget for resources.</p><p>Each of the examples above limits a teams ability to create cloud costs.</p><h3>Conclusion</h3><p>Long lasting change requires influencing human behavior. Many times people will align with missions and change their behavior when presented with data that describes the affects of their actions. In other cases its necessary to actively alert individuals of the effects of their actions. Sometimes neither of these is enough to promote changing behavior and regulation is required.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=473cfa2ea1e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/creating-lasting-change-in-organizations-by-modifying-behavior-473cfa2ea1e7">Creating Lasting Change in Organizations By Modifying Behavior</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Interview: Technical Level Setting]]></title>
            <link>https://medium.com/dm03514-tech-blog/how-i-interview-technical-level-setting-7bf2b99f4edd?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/7bf2b99f4edd</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Sun, 29 Dec 2019 19:10:35 GMT</pubDate>
            <atom:updated>2019-12-30T14:30:47.539Z</atom:updated>
            <content:encoded><![CDATA[<h3>How I Conduct Interviews: Technical Level Setting</h3><p>Technical interviews are often ambiguous and poorly structured without having explicit goals. The following outlines a methodology that I’ve used successfully over 30 interviews and 3 hires. Even though this isn’t a huge sample set I’m really happy with the signals that this methodology provides and I would like to share it. The methodology is called Invariants and Gradients, and this specific stage is focused on Level Setting (establishing that the candidate is at a certain technical level or proficiency). This is accomplished through asking open ended questions which encourage the candidates level, and current abilities, to emerge.</p><h3>What’s Even the Point of a Technical Interview?</h3><p>Technical interviews often act as a check and balance system to verify that candidates are making factual claims about their experience.</p><p>Consider a candidate that applies with nothing known about them compared to a candidate that is super well known within the industry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*JYeDHH7Yw5w1vXZkqkbwyw.png" /></figure><p>Most candidates fall somewhere in between not knowing anything at all about the candidate and having perfect information of their technical abilities. Normally, candidates present some sort of information about their backgrounds, projects they have worked on, and general technical indicators like open source examples. We don’t fully believe the candidate, because if we did we’d evaluate them technically based on the experience that they present.</p><p>Often the purpose of the technical interview is to:</p><ul><li>Establish that the candidate has the requisite experience to perform the functions of the job</li></ul><p>I have found there to be two major components to accomplish this:</p><ul><li>Materializing an understanding of the candidates holistic technical abilities. I call this <strong>Level Setting</strong> and its purpose is to ensure that the candidates level (their general technical capabilities) is understood, ie entry, junior, mid, engineer II, senior, etc.</li><li>Demonstrate understanding of specific technical proficiencies. This establishes that the candidate has specific technical abilities required of the position. If the code base they are working in is a distributed system and their is significant risk from incorrectness the candidate <em>needs</em> to demonstrate proficiency in distributed system. If a candidate is expected to understand a specific technology or language, they <em>need</em> to demonstrate proficiency in the language. There are many ways to do this, which are out of scope of this article.</li></ul><h3>Methodology Goal</h3><p>The goal is to create an exercise that provides high signal on a candidates level and minimizes the noise. There are many time constraints for interviews both on the candidate side and the company side. Candidates are most likely interviewing at many places, creating constraints of companies having to compete with other companies for offers. On the hiring side, interviews are extremely expensive and time consuming. The solution should minimize interview latency, while maximizing accuracy and signal fidelity by understanding a candidate’s level through exploring how they approach technical problems and create solutions.</p><h3>Implementation</h3><p>The solution creates an open ended scenario that encourages a candidates technical ability to emerge, meaning it allows the candidates competencies to emerge from the assignment. It’s necessarily something that the candidate can’t study for.</p><p>This uses a technical discussion (phone interview) or hands on task (coding exercise) as a foundation to determine, with high accuracy, where the candidate’s skills rank, right now. This is not for trying to guess what the candidate may be capable of in the future (that can be estimated from their past trajectory). This is also not for getting very specific technical information from them (which need to be demonstrated directly, either by asking or by demonstration).</p><p>All are open ended because all encourage emergent based on:</p><ul><li>The candidate will mention what they have the most experience doing. This indicates the way the candidate thinks and how they would actually approach the problem, because they aren’t something that can be prepared for.</li><li>The candidate will mention what they think is the most important based on second hand experience (read, observed, heard of, etc).</li><li>The candidate mentions what they think the interviewer wants to hear (I haven’t observed this yet in practice).</li></ul><h4>Invariants</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*HIsWCf6kQVg9CPcp8l3g_Q.png" /></figure><p>The foundation of the approach are Invariants. These are things that the candidate <em>must</em> display in order to qualify. These are a hard line in the sand.</p><p>Some of the general Invariants that I have used are:</p><ul><li>Asks clarifying questions</li><li>Verifies solution runs / works (for a coding exercise)</li></ul><p>It’s important that these are objective and can either be hit or missed. Consider some technical interviews that ask questions about specific technologies like:</p><ul><li>What are some common HTTP verbs and how are they typically used in REST API’s?</li><li>How would you request that a server return an API call results in JSON format?</li><li>Which request header is used to transmit credentials to a server?</li></ul><p>What would a candidate answering “I don’t know” to any of these questions indicate about the candidate? What if they were only able to list 2 of the request verbs? 5? Because of how expensive interviewing is, there’s a huge opportunity cost to asking questions like this during technical interviews. Invariants are used to prune candidates in an objective fair and high fidelity way.</p><h4>Gradients</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*plq0JHadNtkeYmiPohX4tg.jpeg" /></figure><p>Gradients define the signals that are important for the target role. The goal of gradients is to try and formalize the signals. For example Software Engineering, these are open ended and aim to have the candidates ability naturally emerge and place itself, the whole idea is for these to be things that people really can’t study for..</p><p>Gradients are questions that can’t be studied for which encourage immediate reflexive answers. There are no wrong answers here because the purpose is to place a candidate. I’ve found these discussions to be really fun and the candidate to be really engaged.</p><p>Examples are:</p><ul><li>How do you ensure that a solution functions correctly?</li><li>How do you tell if a deployment is successful?</li><li>Suppose that you’re creating an HTTP service and recently made a successfully deploy but some time later customers start to report a degradation of service, how do you go about figuring out the source of the issue?</li><li>Explain the steps of a request and response from curl to a web service like google.</li></ul><p>Each of the gradients may have one or more specific invariants. An invariant for How do you ensure that a solution functions correctly? may be:</p><ul><li>Must mention automated testing</li></ul><p>Because the question are so open ended the candidate usually has to choose a level of abstraction that they are most familiar with, which provides a strong signal in itself.</p><p>Gradient scoped invariants are where specific role requirements can be expressed. Consider the example of the HTTP Request/Response. If this question is asked to a backend we candidate and they fail to mention load balancing tiers. The goal of this methodology is to make these requirements explicit and increase objectivity of evaluation.</p><p>To illustrate how gradients may be evaluated, consider the question:</p><p>How do you ensure that a solution functions correctly?</p><p>Common answers may be:</p><ul><li>Unit Tests</li><li>TDD</li><li>Service Tests</li><li>CI/CD</li><li>Reliance on QA</li><li>Manual Verification</li><li>Monitoring</li></ul><p>Strong Level Setting indicators may be:</p><ul><li>Not Mentioning Tests — Entry</li><li>Mentioning: Run it and Watch with Logs — Entry</li><li>Tests &amp; No Monitoring — Mid</li></ul><h3>Conclusion</h3><p>I have found technical level setting to be a crucial step in materializing an understanding of candidates technical abilities. Technical Level setting is not a replacement for demonstrating specific technical requisites but can be a powerful fun interview step that provides low false positives and negatives for qualifying candidates for specific roles. Technical Leveling is based on ensuring candidates comply with invariants combined with questions that allow candidates abilities to emerge. A large number of the interviews that I gave ~5 / 30 (16%) I received direct feedback from the candidate describing these as fun!!! This stage is often a fun free feeling discussion. Happy Interviewing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bf2b99f4edd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/how-i-interview-technical-level-setting-7bf2b99f4edd">How I Interview: Technical Level Setting</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software Performance Tuning Methodology: Discover, Design, Measure & Refine]]></title>
            <link>https://medium.com/dm03514-tech-blog/software-performance-tuning-methodology-discover-design-measure-refine-e0866c0898b8?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/e0866c0898b8</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Tue, 17 Dec 2019 16:34:52 GMT</pubDate>
            <atom:updated>2019-12-17T20:29:31.035Z</atom:updated>
            <content:encoded><![CDATA[<p>Software performance tuning is often regarded as a dark art for low level hackers. In my experiences the majority of performance tuning is much more social and systems based, opposed to low level OS wizardry. This post outlines an approach to software performance tuning which incorporates performance into the very beginning of the software development lifecycle. This post was adapted from an internal talk.</p><h3>What is Software Performance? And Why Should We Care?</h3><p>Software performance is an overloaded term and can be used to mean a number of things:</p><ul><li>Responsiveness (latency)</li><li>Stability (availability)</li><li>-ilities (scalability, extensibility, maintainability, extensibility)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*IH3UOkIFv8T2VwOLj-Lmxw.png" /></figure><p>Without refining the definition yet, we should care because software performance:</p><ul><li><strong>Impacts</strong> Customer Experience</li><li><strong>Informs</strong> Scaling &amp; Provisioning</li><li><strong>Indicates</strong> Performance Regressions (relative changes)</li></ul><h4>What’s the Problem?</h4><p>Performance is (unfortunately) Emergent, meaning that individual systems performance may not inform the larger combined systems performance. Even if this is not strictly 100% true, it’s difficult to predict the performance of end user system until the system is fully put together and running on hardware that will be servicing the production workload. This has 2 important implications:</p><ul><li>Difficult to predict performance (Favor Observation)</li><li>System Specific (Local Doesn’t Predict Remote)</li></ul><p>Because of this, we often see the mantra: Make it Work then Make it Fast emerge. This suggests that we shouldn’t worry about performance until we have a working project. While I think this is better advice than the other end of the spectrum, there is plenty of low overhead work we can do to design performance in.</p><h4>Finally! Performance is:</h4><p>Defined by clients, and a proxy for their experience! Performance is the executable validation of a design. Performance will happen either Explicitly, under our control and guidance or implicitly, hitting against unforeseen user expectations or unforeseen physical limitations (such as the <a href="https://queue.acm.org/detail.cfm?id=1900007">speed of light</a>).</p><h3>Ensuring Code is Performant</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0usBxOjTuOu7iF-k8SvQA.png" /></figure><h3>Discover</h3><p>What are the known or discoverable constraints?</p><h4>Explicit Constraints</h4><p>These will often be viewed from a client’s perspective. These are often contractual ie a customer is paying for a certain level of service:</p><ul><li>Response must be served with 100ms</li><li>99.99% of requests need to be &lt; 1 second</li></ul><p>The two most common sources for these are:</p><ul><li>Contractual</li><li>Product Management</li></ul><h4>Implicit Constraints</h4><p>Implicit constraints are performance limits hidden in requirements. If a program is doing <em>daily</em> reports or <em>hourly</em> rollups, the interval of those actions may defined the expected limit. These are constraints that are often end user usability experience (UX) focused. For example if an action is blocking in the user path there is a <a href="https://www.nngroup.com/articles/powers-of-10-time-scales-in-ux/">~100ms limit for the site to feel “responsive”</a>.</p><h4>Physical Limitations of Systems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_Jxj7psjqOqQCLNULjA3g.png" /></figure><p>Physical limitations are often dictated by the minimum achievable performance in the physical world! If certain latencies are required between largely separated locations performance will run up against the speed of light! There are other performance critical constraints based on the current speed of hardware, seen in the <a href="https://gist.github.com/hellerbarde/2843375">“Latency Numbers Every Programmer Should Know”</a> infogram.</p><h4>Reasonable Ranges</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Va8Pn3F_J9cwCMXmSWAOdA.png" /></figure><p>Reasonable ranges are formed based on knowledge of physical system limitations and experience of observing systems in production. Suppose that there is a go service that accepts and HTTP connection, makes a get request to redis and then writes and closes the HTTP connection. The service is being tested locally by running both the service and redis and driving traffic using apache benchmark. Locally is a 2019 macbook pro with 6 core and 16GB of ram. On the first test the service is only able to handle 50 requests per second! Based on experiences this is unreasonable. This service should be able to push a couple hundred requests per second easy.</p><h3>Design</h3><p>Design phase is focused on materializing the structure of a concrete implementation. This is where discovered constraints are combined with implementation strategies. Most of the performance work should be done here, as this is where the solution is most malleable and has the least cost associated with changing.</p><h4>Measurement Strategy</h4><p>First step is to define a measuring strategy:</p><ul><li>What is being measured?</li><li>Where is it being measured?</li><li>How will it be collected?</li></ul><p>And finally what will we use to determine if this is “performant”? <a href="https://cloud.google.com/blog/products/gcp/building-good-slos-cre-life-lessons">Google’s strategies for choosing good SLOs highly align with this step</a>.</p><h4>Theoretical Performance</h4><p>Theoretical Design takes into account theoretical performance and runtime complexity of implementation algorithms and datastructures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*fKusI5kyN1DOND5WGvUPnw.png" /></figure><p>It also includes system specific implications of performance, like implications of schema choices on database query execution.</p><h4>Isolatable Components — Testable Design</h4><p>This brings us to the code level. There are a number of techniques that reduce <a href="https://medium.com/dm03514-tech-blog/you-are-going-to-need-it-using-interfaces-and-dependency-injection-to-future-proof-your-designs-2cf6f58db192">friction in isolating components</a> for individual benchmarks and in providing stub (highly controllable) implementations for performance testing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9aSHbFnG7e36al9Zfnhnxw.png" /></figure><ul><li>Interfaces &amp; Swappable Implementations</li><li>Dependency Injection</li><li>Exercise components in isolation using test harness</li><li>Ability bring up and exercise the service using test load</li></ul><h3>Measurement</h3><h4>Strategies</h4><p>Using the measurements defined above its time to start determining if performance has been “achieved”! Since this methodology puts client experience as first priority if performance meets the clients expectations then no more tuning is necessary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ampbxtKRzhI9Cj_8y4KNtQ.png" /></figure><h4>Execution : Generate Load</h4><p>Many different ways to load a system. Best to start with the higher level (customer representative metrics) and slowly work down into implementation details as more data becomes available. Many different test types:</p><ul><li><a href="https://en.wikipedia.org/wiki/Software_performance_testing">Performance</a></li></ul><p>Local (relative) vs Remote in a Prod-like environment (absolute)</p><ul><li><a href="https://en.wikipedia.org/wiki/Soak_testing">Soak/Endurance/Stress</a></li></ul><p>Remote Long running, verifies steady state</p><ul><li><a href="https://martinfowler.com/bliki/CanaryRelease.html">Canarying</a></li><li><a href="https://medium.com/dm03514-tech-blog/service-level-test-stacks-in-action-using-go-a77a0e808b1c">Service Test Harness</a></li><li>Micro benchmarks (such as <a href="https://golang.org/pkg/testing/#hdr-Benchmarks">go Benchmark</a>)</li></ul><h4>Establish Baseline</h4><p>Establishing baseline is crucial for seeing relative change in performance. Without a baseline it’s difficult to understand the impact of a change:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*6FCgLjwRsGfi54PlgyFLRg.png" /></figure><h4>Profile</h4><p>Profiling is used to determine where an application is spending its time or other resources (ie memory). This is often the <a href="https://medium.com/dm03514-tech-blog/sre-performance-analysis-tuning-methodology-using-a-simple-http-webserver-in-go-d475460f27ca">“hands-on” part of performance tuning</a>. Using low level tools to understand which resources a program is using. The goal is to generate empiric data for where an application is spending its time, and which resources its using:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*cZuZQgIFlwiCbZBiHJHcUA.png" /></figure><h4>Visualize!</h4><p>Once data is captured it’s important to put it in a form that can be easily understood. A common profile visualization technique is called Flame Graphs and was created by Brendan Gregg:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*GYLZ9_-aJL9nkGyS217CXA.png" /></figure><p>A detailed description on how it can be used to understand application <a href="https://queue.acm.org/detail.cfm?id=2927301">performance can be found on the acmqueue article here.</a></p><p>(<a href="https://queue.acm.org/detail.cfm?id=2927301">Image belongs to Brendan Gregg and was Published on acmqueue</a>)</p><h3>Refine</h3><p>Refining is the hands on act of combining the previous steps to generate data, observe the performance, create hypotheses, and execute experiments.</p><h4>Scientific Method in Action</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*6eXBZHOHFgU-zSXgpnZTKA.jpeg" /></figure><h4>Strategies</h4><p>Theory of Constraints (“a chain is no stronger than its weakest link”) is a common tuning strategy. It focuses on finding the largest bottleneck, or contributor to latency, to achieve desired performance. It identifies the bottleneck, removes the bottleneck, and then remeasures to identify the next bottleneck. In performance tuning there will always be a slowest operation. Because of this performance tuning provides diminishing returns. At some point it doesn’t make economic sense to continue increasing performance.</p><h4>Anti-Patterns</h4><p>Performance Engineering has a number of insidious anti-patterns:</p><p><strong>Identifying bottlenecks based on Intuition vs Facts</strong></p><p>Solution is to let reality (fact-based evidence) be the guide</p><p><strong>Using Local Performance to predict global</strong></p><p>Performance is emergent so the best local performance can tell is a relative performance increase/decrease based on previous local baselines</p><p><strong>Tuning without overarching client centric measurements</strong></p><p>Choose a client representative performance metric in order to understand the impact of performance tuning on the client’s experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*hN8lxWZcdxI1T1pjg8FDZw.png" /></figure><p><strong>Choosing low impact candidates</strong></p><p>If an operation is responsible for 5% of latency and can take 4 weeks to halve and another bottleneck is responsible for 10% of latency and takes 4 weeks to halve it should be intuitive to attack the 10% latency operation. This can be addressed by using fact based evidence combined with a methodology like Theory of Constraints.</p><p><strong>Not collecting data because it’s too difficult, low level, app wasn’t designed properly etc.</strong></p><p>There’s ways to measure pretty much everything (BPF) now!</p><p>(<a href="https://martinfowler.com/bliki/AntiPattern.html">antipattern image property of Martin Fowler</a>)</p><h3>Conclusion</h3><ul><li>Performance is as much Design as it is Execution</li><li>Performance is social</li><li>Performance is continuous</li><li>Performance is about uncovering what the program is actually doing by observing where it spends its time</li><li>Performance is application of Scientific Method</li></ul><p>Happy Performance Tuning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0866c0898b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/software-performance-tuning-methodology-discover-design-measure-refine-e0866c0898b8">Software Performance Tuning Methodology: Discover, Design, Measure &amp; Refine</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visions, Projects & Priorities: Using Temporal Graphs to Build A Reality]]></title>
            <link>https://medium.com/dm03514-tech-blog/visions-projects-priorities-using-temporal-graphs-to-build-a-reality-4e545fa5cdfa?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/4e545fa5cdfa</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Fri, 30 Aug 2019 19:16:15 GMT</pubDate>
            <atom:updated>2019-09-06T14:15:21.977Z</atom:updated>
            <content:encoded><![CDATA[<h3>Visions, Projects &amp; Priorities: Using Temporal Graphs to Execute an Actionable Vision</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*pqFnfF-vuVHHLfwE6__i5g.png" /></figure><p>There’s a huge problem in business right now: the best prioritizers, decision makers, visionaries, and “shapers” use their abilities to disproportionately benefit themselves. Deciding “what’s next” is one of the most difficult parts of leadership, but there are very few resources on making good mid and long term decisions, weighing decisions for priority and optimizing complex decision options. Without long term vision, and structured approach to creating a reality, companies are prone to fall into making decisions based on local optima: “What’s the best move right now” or making decisions optimized around short delivery intervals: “What’s the highest value thing to do in this interval”. This post explores the concept of temporal graphs and how they can be used to reason about reality and construct timelines in order to reason about long term initiatives.</p><h3>Temporal graphs</h3><p>Temporal Graphs provide a way to reason about causality. Every action made creates opportunity for new actions. Have you ever read a choose your own adventure book? Most pages are left with a choice, and each choice either opens up the possibility to make more choices or loses choices.</p><p>Consider a case where you had an option to walk to work ~30 minutes, or drive to work ~20 minutes unless you hit traffic. You choose to drive because you worked out extra hard last night.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*OzQpm5WORUWLopk3D7paRw.png" /></figure><p>In the walk scenario, getting delayed from traffic is nearly impossible; there are an almost infinite other ways to get delayed, but the chances of getting delayed because of traffic is minimal. At every moment in time there are a near infinite number of choices to make. Looking back over all choices that we made creates a path through a near infinite graph that we call our life. Where we are is a result of the decisions that we made, the decisions that other people made, and random luck. Successful projects and successful businesses also can be traced through a series of events; events that span conscious choices as well as strings of random events. Whatever the event’s driver, at each movement in time there was a set of possible events, one of which happened, opening up another set of possible events, until arriving at right now.</p><h3>End States</h3><p>End States are critical at the top level of business and projects for success. It’s unrealistic to hope to succeed as a by product of random luck or favorable events from other people. Because of this, leaders need to have clear end states, be able to explain those end states, and execute to achieve those states. A leader in this case is anyone with a stake at moving the company to a different state (ie anyone responsible for changing the company).</p><p>Let’s walk through an example of how end states can be thought about and visualized. In order to do this, pretend I have a growing family and I determine there’s a need for a mini-van.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2f6yncN-yF_hF0QhV27Xqg.png" /></figure><p>The vision is to get a minivan to provide more space for the family. The current state and end goal are very clear but there’s no defined path in between. While visualizing an end state is better than nothing, it provides very little value to have a dream but no path to it. Without a concrete path an end state is just a hope, a nice idea.</p><h3>Intermediary states</h3><p>Intermediary states are a series of causal events that can transform the current state into the desired state. At the highest level these are logical events and are notable events in the transformation of the current state to the desired.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*rzKxmF8KswlkjVoWA65ZUg.png" /></figure><p>Each one of these states lead to the end goal. The level of abstraction here is intentionally high level. For vision and roadmaps, the level only needs specific enough to clearly convey the thought process and explain and convince others how the current state can be transformed into the desired state.</p><h3>Levels of intermediary states (Abstraction)</h3><h4>Logical Strategy</h4><p>What we’ve been talking about up until this point is the Logical strategy. Logical strategy is a list of causal states that leads from one state to another but intentionally avoids getting too detailed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*8nE7pBZIaCw4Xb6ticvdTw.png" /></figure><h4>Execution Strategy</h4><p>Execution strategy is something that can be decomposed and handed out to people to begin to execute, the <strong>Sell Old Car</strong> state might have a temporal graph that looked like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*3qpbXRwIXiDYBidBkmDAGQ.png" /></figure><h4>Day to day (Executable) Strategy</h4><p>This level is focused on concrete work items. Executable strategy further breaks down the execution strategy from above. The “Prepare car” state from above might have an Executable Strategy of:</p><ul><li>Vacuum car</li><li>Schedule detailer</li><li>Fix muffler</li></ul><p>Likewise “List on Website 1” may require states:</p><ul><li>Create an account</li><li>Take photos</li><li>Write description</li><li>List</li></ul><p>These are scoped to the sorts of tasks an Independent Contributor would be doing in order to execute on the vision at the lowest actionable level.</p><h3>Goals</h3><p>Temporal Graphs are helpful at showing how actions lead to outcomes. One common that teams or individuals have goals: adopting a new technology or tool, learning a new language, saving for vacation. Without concrete events connecting the current state to the end state (the goal) there is no path to the goal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*pBdLrbIkIIw59hzc9jYMiA.png" /></figure><p>Hoping to learn a new language without a path to make it a reality (ie regular practicing) or hoping for a vacation without actually saving for it, are visions of the future without a concrete way to achieve those visions; Hopes and Dreams.</p><h3>Exploration Breadth First</h3><p>Establishing a vision may involve some steps that are unknown or need to be discovered. The goal of exploration is to open up favorable situations by learning about potential intermediary states that might lead to the end goal. There are many tasks that aren’t know beyond one or two steps these require exploration in order to get an idea of the possible paths , idea validation, product discovery, customer feedback all help create nodes on the graph and open up possibilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*F-HzEk5ZCUTKySl9bZAFLA.png" /></figure><p>Exploration is necessary when there are unknown intermediary states.</p><h3>Pruning and Probabilities</h3><p>We prune graphs all the time with the hopes of invalidating (fail fast) and removing parts of the graph that can’t help arrive at end state. If our goal is to arrive to work dry and we commute on foot or by bike, we consult the weather. If there is low likelihood of precipitation there’s no need to plan, pack or carry rain gear, in this case planning for rain (by carrying gear) was pruned from intermediary states based on the probability of rain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*qK_yDV5qH6nSSN7KziYzDA.png" /></figure><p>Each work item carries with it an opportunity cost. Pruning and probabilities helps to quickly prioritize events that we shouldn’t plan for or paths that shouldn’t be taken due to low likelihood or sub optimal expected outcomes.</p><h3>Inexperienced Leadership</h3><p>One pattern I regularly see are people in leadership either unwilling or unable to define and communicate a vision. Most of the time this is from in experience or not having certain leadership positions filled. It’s difficult to communicate a vision if a leader doesn’t have one. This manifests as moving away from the current state instead of towards an end state:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*Pppu-CsgnmXn7R47qFbhqQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*uF7AAP0eqzWuPmEy9zJpVQ.png" /></figure><p>Communicating a vision (end state) and a path for making it a reality is one of the <em>most</em> important things in leadership. Inexperienced leaders are often unable or unwilling to do this. A vision is a bold bet and requires confidence, research and commitment. It’s also often easier to incrementally improve the current state and label that as “Success” than painting a vision of the future and how to get there.</p><p>Another Leadership anti-pattern is communicating an end state without the intermediary states. Remember from the first step of the mini van. Having a vision without the concrete steps means its just a hopeful vision. One thing to keep in mind is that a leader will most likely not be filling in all the intermediary states, they orchestrate the reality not perform and define every single piece of work.</p><h3>Why?</h3><p>Temporal Graphs are significant because making decisions, prioritizing and executing is hard! Many times instead of moving towards and end state we’re just trying to incrementally improve the current one. There are surprisingly few heuristics or strategies about decision making in terms of roadmaps, priorities or execution. Shapers see an end goal and can motivate others to fill in the middle states. Project management is very interested in the middle states. Visionary leaders (<a href="http://www.richardhughesjones.com/visionary-leadership/">“Shapers”</a> as Ray Dalio refers to them) are able to do reason about the long term as well as the short term and execute on any level of temporal graphs.</p><p>Project management is interested in the end state and specifics of all in between states. Independent contributors are interested in Executable strategies. Project success requires understanding the logical vision, and understanding how each layer contributes to the overall success of the vision. I believe that understanding reality, and projects as a series of events that form a graph can help give a foundation for project success.</p><h3>Conclusion</h3><p>Project management and priority is one of the most difficult skills required for leadership. Thinking of projects as a graph of states can help to ground those complexities, and give a framework to reason about how to change the current state into desired state. Not understanding and conveying causality severely handicaps projects. Where are you going and how are you getting there?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e545fa5cdfa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/visions-projects-priorities-using-temporal-graphs-to-build-a-reality-4e545fa5cdfa">Visions, Projects &amp; Priorities: Using Temporal Graphs to Build A Reality</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang: Introduction to Race Conditions for the Web Engineer]]></title>
            <link>https://medium.com/dm03514-tech-blog/golang-introduction-to-race-conditions-for-the-web-engineer-738513ab2ec2?source=rss----9135414d39d---4</link>
            <guid isPermaLink="false">https://medium.com/p/738513ab2ec2</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[dm03514]]></dc:creator>
            <pubDate>Wed, 28 Aug 2019 19:08:56 GMT</pubDate>
            <atom:updated>2019-09-03T19:09:22.621Z</atom:updated>
            <content:encoded><![CDATA[<p>Go provides a first-class primitive for concurrency called a Goroutine. Goroutines allow end users to easily spawn concurrent tasks and synchronize those tasks. Since Goroutines are so foundational to the language most non-trivial programs end up exposing the engineer to concurrent programming. While go does provide some tools for preventing <a href="https://en.wikipedia.org/wiki/Race_condition#Data_race">race conditions</a>, it is still ultimately up to the end user to verify that their program is safe for concurrent execution (<a href="https://en.wikipedia.org/wiki/Thread_safety">thread-safe</a>). This post explains what concurrency is in the context of web programming, what data races are, why they are tricky to detect, and how they can be detected using go’s -race detector. At the end of it the reader should understand how to define a data race, how to write a test to exercise a concurrent function and how to analyze the test using the -race detector!</p><h3>Concurrent Web Service</h3><p>A concurrent web service (in this case webserver) is able to service a number of client connections at the same time. For the purposes of this article we are talking about concurrency with regard to the clients. Go’s concurrency model coupled with its built in HTTP frameworks make very compelling for cost effective, performant web services. This makes it popular with traditional web language communities (ruby, python, php) and creates an environment where many experienced web engineers are thrown into concurrent programming environment, resulting in race conditions and bugs.</p><p>In order to illustrate concurrency, consider a web server that is not concurrent. It accepts a single request processes it until finished and then waits for another request. If a client makes a request while a current request is in flight they are either dropped or queued. This is a synchronous system with respect to the clients, meaning it can only process a single request at a time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_wkiX07CNwUFcpkhR9whcg.png" /></figure><p>Traditional web environments act just like this, but instead of running a single script they start multiple single scripts! In <a href="https://httpd.apache.org/docs/2.4/mod/prefork.html">Apache</a> and other environments (unicorn, <a href="https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#adding-concurrency-and-monitoring">uwsgi</a>) this means specifying a number of processes to start (the concurrency level) and then each one of those is able to handle connections as they come in (worker pool). While this model is easy to reason about and insulates engineers from the complexities of concurrency it does mean that concurrent connections scale linearly with respect to hardware resources (ie processes/threads):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*fa5r7jkUICzCP48V8SDnng.png" /></figure><p>The image above shows a synchronous service with multiple instances enabling concurrency with respect to clients. One thing that is interesting here is that if the concurrency level is greater than the total number of instances the service becomes saturated (it has reached capacity because it has more requests than servers able to handle those requests) at which point connections either queue or are dropped.</p><p>What we didn’t discuss here are asynchronous runtimes like node.js that support multiple concurrent requests but can guarantee that only a single statement executes at any moment removing the possibility of data-races. In this case concurrency is handled in the runtime instead of from the webserver orchestrating processes.</p><h3>Concurrency In Go</h3><p>In go, concurrency is achieved through <a href="https://gobyexample.com/goroutines">Goroutines</a>. Goroutines are just normal functions <a href="https://www.ardanlabs.com/blog/2018/12/scheduling-in-go-part3.html">executed concurrently by using the </a><a href="https://www.ardanlabs.com/blog/2018/12/scheduling-in-go-part3.html">go statement</a>! The issue is that if two goroutines access the same memory and at least one of them is writing to the memory this results in undefined behavior: a panic, nothing or a strange interleaving of program. This is called a <a href="https://medium.com/dm03514-tech-blog/golang-candidates-and-contexts-a-heuristic-approach-to-race-condition-detection-e2b230e70d08#0252">Race Condition</a>. Shared memory can be a global variable, a shared map, or a pointer that is shared between goroutines.</p><h3>Go HTTP Concurrency</h3><p>The builtin <a href="https://golang.org/pkg/net/http/">go HTTP</a> server <a href="https://eli.thegreenplace.net/2019/on-concurrency-in-go-http-servers/#appendix-where-net-http-goes-concurrent">invokes each incoming HTTP request using a Goroutine</a>. Concurrency with respect to the client connection is the default which means all HTTP code is concurrent and needs to be safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*Z2hhqo6XILPiTT_J8SOsWg.png" /></figure><h3>In practice</h3><p>What this means in practice is close attention needs to be paid whenever shared memory is accessed inside of an http handler (ie goroutine). In order to remove boilerplate and focus on identifying the data race the test below is executing go func() directly instead of going through HTTP handlers. Since HTTP handlers are executed inside of gostatements this should be equivalent. The test below simulates a “webserver” with a handler go func() and 5 concurrent client requests. Each one of these requests mutate shared memory m:</p><pre>package main<br><br>import (<br>        &quot;fmt&quot;<br>        &quot;testing&quot;<br>)<br><br>func Test_RaceCondition(t *testing.T){<br><br>        m := make(map[string]string)<br><br>        for i := 0; i &lt; 5; i++ {<br>                go func() {<br>                        kv := fmt.Sprintf(&quot;hi_%d&quot;, i)<br>                        m[kv] = kv<br>                }()<br>        }<br>}</pre><h3>Undefined Behavior</h3><p>One of the challenges of race conditions are that they result in undefined behavior. The code below executes the race condition test listed above using 5 Goroutines, and it “passes”:</p><pre>for i := 0; i &lt; 5; i++ {</pre><p>And</p><pre>$ go test -run Test_RaceCondition<br>PASS<br>ok      github.com/dm03514/test 0.019s</pre><p>This test execution randomly got lucky, for whatever reason there were no interleaving of writes m[kv] = kv. Even though it says PASS there is still a race condition.</p><p>Increasing the concurrency to 50 and re-running the test reliably triggers the race condition:</p><pre>$ go test -run Test_RaceCondition<br>fatal error: concurrent map writes<br>fatal error: concurrent map writes<br><br>goroutine 20 [running]:<br>runtime.throw(0x113bf62, 0x15)<br>        /usr/local/Cellar/go/1.12.7/libexec/src/runtime/panic.go:617 +0x72 fp=0xc00003df00 sp=0xc00003ded0 pc=0x102aa32<br>runtime.mapassign_faststr(0x11112c0, 0xc0000962a0, 0xc0000f0010, 0x4, 0x1)<br>        /usr/local/Cellar/go/1.12.7/libexec/src/runtime/map_faststr.go:291 +0x40f fp=0xc00003df68 sp=0xc00003df00 pc=0x101138f<br>github.com/dm03514/test.Test_RaceCondition.func1(0xc0000a0088, 0xc0000962a0)<br>        /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:15 +0xbd fp=0xc00003dfd0 sp=0xc00003df68 pc=0x10f0d9d<br>runtime.goexit()<br>        /usr/local/Cellar/go/1.12.7/libexec/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00003dfd8 sp=0xc00003dfd0 pc=0x10570a1<br>created by github.com/dm03514/test.Test_RaceCondition<br>        /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:13 +0x6e<br><br>...<br><br>goroutine 23 [running]:<br>runtime.throw(0x113bf62, 0x15)<br>        /usr/local/Cellar/go/1.12.7/libexec/src/runtime/panic.go:617 +0x72 fp=0xc00003f700 sp=0xc00003f6d0 pc=0x102aa32<br>runtime.mapassign_faststr(0x11112c0, 0xc0000962a0, 0xc0000180b8, 0x5, 0x1)<br>        /usr/local/Cellar/go/1.12.7/libexec/src/runtime/map_faststr.go:291 +0x40f fp=0xc00003f768 sp=0xc00003f700 pc=0x101138f<br>github.com/dm03514/test.Test_RaceCondition.func1(0xc0000a0088, 0xc0000962a0)<br>        /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:15 +0xbd fp=0xc00003f7d0 sp=0xc00003f768 pc=0x10f0d9d<br>runtime.goexit()<br>        /usr/local/Cellar/go/1.12.7/libexec/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00003f7d8 sp=0xc00003f7d0 pc=0x10570a1<br>created by github.com/dm03514/test.Test_RaceCondition<br>        /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:13 +0x6e<br>exit status 2<br>FAIL    github.com/dm03514/test 1.017s</pre><h3>Race detector</h3><p>Go provides a built in tool called the race dector that helps to identify race conditions. While -race should be used it can result in false negatives because it needs to observe concurrent accesses. For example, if a test suite passed in -race flag but doesn&#39;t spawn go routines than there&#39;s nothing the race detector can do. In the case of the race condition above the race detector works perfectly. Even at 5 concurrent requests its able to detect the race condition:</p><pre>$ go test -run Test_RaceCondition -race<br>==================<br>WARNING: DATA RACE<br>Read at 0x00c00001a160 by goroutine 7:<br>  github.com/dm03514/test.Test_RaceCondition.func1()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:14 +0x3c<br><br>Previous write at 0x00c00001a160 by goroutine 6:<br>  github.com/dm03514/test.Test_RaceCondition()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:12 +0xb8<br>  testing.tRunner()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:865 +0x163<br><br>Goroutine 7 (running) created at:<br>  github.com/dm03514/test.Test_RaceCondition()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:13 +0x94<br>  testing.tRunner()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:865 +0x163<br><br>Goroutine 6 (running) created at:<br>  testing.(*T).Run()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:916 +0x65a<br>  testing.runTests.func1()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:1157 +0xa8<br>  testing.tRunner()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:865 +0x163<br>  testing.runTests()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:1155 +0x523<br>  testing.(*M).Run()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:1072 +0x2eb<br>  main.main()<br>      _testmain.go:42 +0x222<br>==================<br>==================<br>WARNING: DATA RACE<br>Write at 0x00c000086330 by goroutine 9:<br>  runtime.mapassign_faststr()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/runtime/map_faststr.go:202 +0x0<br>  github.com/dm03514/test.Test_RaceCondition.func1()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:15 +0xd8<br><br>Previous write at 0x00c000086330 by goroutine 7:<br>  runtime.mapassign_faststr()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/runtime/map_faststr.go:202 +0x0<br>  github.com/dm03514/test.Test_RaceCondition.func1()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:15 +0xd8<br><br>Goroutine 9 (running) created at:<br>  github.com/dm03514/test.Test_RaceCondition()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:13 +0x94<br>  testing.tRunner()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:865 +0x163<br><br>Goroutine 7 (finished) created at:<br>  github.com/dm03514/test.Test_RaceCondition()<br>      /Users/danielmican/go/src/github.com/dm03514/test/main_test.go:13 +0x94<br>  testing.tRunner()<br>      /usr/local/Cellar/go/1.12.7/libexec/src/testing/testing.go:865 +0x163<br>==================<br>--- FAIL: Test_RaceCondition (0.00s)<br>    testing.go:809: race detected during execution of test<br>FAIL<br>exit status 1<br>FAIL    github.com/dm03514/test 0.535s</pre><p>Whereas previously the test could pass with 5 concurrency from random bad luck, the race detector is able to record the accesses to the shared memory and see that those accesses are coming from 2 separate go routines!</p><h3>Fixing the Data Race</h3><p>The final step is to fix the Data race. One way of doing this is to use primitives (mutex) to ensure that only a single goroutine is able to write to the map at a time.</p><pre>package main<br><br>import (<br>        &quot;fmt&quot;<br>        &quot;testing&quot;<br>        &quot;sync&quot;<br>)<br><br>func Test_RaceCondition(t *testing.T){<br>        m := make(map[string]string)<br>        mu := sync.Mutex{}<br><br><br>        for i := 0; i &lt; 500; i++ {<br>                go func() {<br>                        kv := fmt.Sprintf(&quot;hi_%d&quot;, i)<br>                        mu.Lock()<br>                        m[kv] = kv<br>                        mu.Unlock()<br>                }()<br>        }<br>}</pre><p>Even at 500! concurrency the code is now safe because it guarantees that no concurrent accesses can be performed on the map.</p><pre>$ go test -run Test_RaceCondition<br>PASS<br>ok      github.com/dm03514/test 1.755s</pre><h3>Conclusion</h3><p>Go exposes concurrency as a first class citizen and makes it dead simple to use. This comes at the cost of reasoning about concurrent correctness and data races. On top of this, many of the popular web environments, have runtimes and deployment models that remove the need. Careful consideration must be given to concurrent memory accesses and shared memory when programming in go. It’s critically important to understanding: “Will this code be executing concurrently?” and if so “Is this code thread-safe”? Once this is understood go provides great testing utilities (-race) to help gain confidence in concurrent correctness. In addition to -race I have also had a ton of success detecting race conditions using a Structured analysis technique named <a href="https://medium.com/dm03514-tech-blog/golang-candidates-and-contexts-a-heuristic-approach-to-race-condition-detection-e2b230e70d08#0783">Candidates &amp; Contexts</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=738513ab2ec2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dm03514-tech-blog/golang-introduction-to-race-conditions-for-the-web-engineer-738513ab2ec2">Golang: Introduction to Race Conditions for the Web Engineer</a> was originally published in <a href="https://medium.com/dm03514-tech-blog">Dm03514 Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>