<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piyush Verma on Medium]]></title>
        <description><![CDATA[Stories by Piyush Verma on Medium]]></description>
        <link>https://medium.com/@meson10?source=rss-fe36fbe48c68------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IzRwDN1U1bWsHZJKB_0wGQ.jpeg</url>
            <title>Stories by Piyush Verma on Medium</title>
            <link>https://medium.com/@meson10?source=rss-fe36fbe48c68------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 17:45:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@meson10/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sample vs Metrics vs Cardinality]]></title>
            <link>https://medium.com/last9/sample-vs-metrics-vs-cardinality-ffb160a45fae?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffb160a45fae</guid>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[reliability]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Sun, 21 Aug 2022 19:21:23 GMT</pubDate>
            <atom:updated>2022-08-22T17:47:17.330Z</atom:updated>
            <content:encoded><![CDATA[<p>When dealing with Time Series databases, I always got confused with Sample vs Metrics vs Cardinality. Here’s an explanation as I have understood it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Vxa1Yja5UC72HBbIDa6ww.jpeg" /></figure><blockquote>A sample is the most atomic record of observability that cannot be explored beyond its attributes.</blockquote><p>A sample has a few key attributes.</p><ul><li><strong>Timestamp</strong>: Of the observation.</li><li><strong>Value</strong>: Of the observation.</li><li><strong>Name</strong>: What is being recorded?</li><li><strong>Duration</strong>: Time between two recordings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Od88tbWmlz4DGWHe" /><figcaption>Dimensions are City: Austin, State: Texas, Country: USA, ZIP: 94043 Metric name is `total_water_consumed` The period is 19:00–19:10 on 20th January 2022</figcaption></figure><h3>Key things</h3><ul><li>The data is not AT the time of observation, but the duration between two observations. Ex. <strong>17th Jan 2022, 18:05</strong> till <strong>17th Jan 2022, 18:06</strong></li><li>Some call this <strong>Period</strong>.</li><li>No matter your service, the data is always an aggregate for a minimum time range.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f0hW-cWKQDDWKeJ4" /></figure><p>As your Observability grows, you face three key challenges.</p><ul><li><strong>More metrics</strong> are to be observed. Also called <strong>Coverage</strong>.</li><li>Same metric for <strong>more entities</strong>. Also called <strong>Cardinality</strong>.</li><li>Save metrics for a <strong>longer duration</strong>. Also called <strong>Retention</strong>.</li></ul><h3>Retention</h3><p>As you start saving data for longer, at a given time, you have more historical data that can be used for trend analysis. But like any Data system, this adds additional strain to keep the system reliable for a larger volume of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YgP_nDV5dxgxUTiR" /><figcaption>We were earlier saving data for only 10 minutes. When we increase the retention to 30 minutes, 2 more samples will be preserved.</figcaption></figure><h3>Cardinality</h3><p>It is the number of unique attributes observed for a given metric. Cardinality is the most formidable challenge because the computation and memory needed to tackle queries grow exponentially.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qaWKWZrJrZjZFALO" /><figcaption>We were monitoring total_water_consumed for a particular zip code, say we need 2 additional zip codes to be monitored, It would mean 2 more samples for the same metric + duration.</figcaption></figure><h3>More Observability</h3><p>Coverage increases when you observe more metrics or more dimensions. But, Saving more metrics does not mean better Observability; it may also just mean more unused data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZrigrsFTPaZv-nJt" /><figcaption>For the same duration AND for all the 3 ZIP Codes, we decide to observe total_power_consumed in addition to the total_water_consumed. Therefore, we will save 3 more samples, 1 for each zip_code AND timestamp.</figcaption></figure><h3>Growth in Real Data</h3><p>Modern Time Series systems doesn’t have to grow along a single axis of Cardinality, Coverage, or Retention alone. Instead, the rate of ingestion and exploration warrants an expansion on all three axes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YDj3Nt-w4hJbU-sf" /></figure><p>It’s crucial to understand how samples get involved while querying and the scale and performance limitations.</p><p>Say, I want water AND power consumption for 20 minutes but only for <em>{austin, texas, USA, 94043}</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*34STjHwEjeg-bCru" /><figcaption>It will only involve one Dimension, two Coverage, and two Time units.</figcaption></figure><p>Say, I want ONLY water consumption for 20 minutes, but for <em>{austin, texas, USA, 94043}</em> AND <em>{austin, texas, USA, 95058}</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*84Gx9jt1vB_YyaQD" /><figcaption>It only involves one Coverage, two Dimensions, and two Time units.</figcaption></figure><p>Conversely, If I want Power AND Water consumption across <em>{austin, texas, USA, 94043}</em> AND <em>{austin, texas, USA, 95058}</em> but ONLY for 10 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wdpM5ycqpHT9xNSf" /><figcaption>It will involve one Time, two Dimensions, and two Coverage units.</figcaption></figure><p>Whereas, If I want Water and Power consumed by all Females of <em>{austin, texas, USA, 94043}</em>, <strong>I won’t find any results.</strong></p><blockquote>Remember: A sample is the most atomic record of observability that cannot be explored beyond its attributes.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*65GbP6oHRV6Q80Mt" /></figure><h3>Concluding</h3><ul><li>For better observability, you need to have more exemplary attributes. Resulting in more samples.</li><li>For better coverage, you need more metrics. Resulting in more samples.</li><li>For historical knowledge, you need more retention. Resulting in more samples.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LadSKoIYzxMn90AH" /><figcaption>A typical time-series database is subjected to various workloads, all requesting different kinds and ranges of data.</figcaption></figure><p>Found this interesting? Got a different PoV? Reach out to me on twitter — <a href="https://twitter.com/realmeson10">@realmeson10 </a>. Oh, also, you should check out <a href="https://medium.com/last9.io">last9.io</a>; we’re building reliability tools to make running systems at scale, fun, and <strong><em>embarrassingly easy. </em></strong>Check us out :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffb160a45fae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/last9/sample-vs-metrics-vs-cardinality-ffb160a45fae">Sample vs Metrics vs Cardinality</a> was originally published in <a href="https://medium.com/last9">last9</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Much that we’ve gotten wrong about Site Reliability Engineering]]></title>
            <link>https://medium.com/last9/much-that-weve-gotten-wrong-about-site-reliability-engineering-b4b74142b25a?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4b74142b25a</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Thu, 19 Nov 2020 02:33:49 GMT</pubDate>
            <atom:updated>2020-11-19T02:33:49.289Z</atom:updated>
            <content:encoded><![CDATA[<h3>Much that we have gotten wrong about SRE</h3><p>An illustrated summary of <strong>Developers</strong> -&gt; <strong>DevOps</strong> -&gt; <strong>SRE</strong></p><h4>1. Developers wanted to ship their produce</h4><p>To the other side</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xI9vJAHlskEm3pLwPMA9nQ.jpeg" /></figure><h4>2. Production never matches the development environment. It resembles, but cannot match.</h4><p>So they deployed people on the other side</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*US-khkej4EjoyBBWeV2gyQ.jpeg" /></figure><h4>3. But this process was slow, they wanted to deploy faster</h4><p>So they deployed <strong>Continuous-Deployment</strong> (CI/CD)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r77GWXxJF6y5XVEvnykYxw.jpeg" /></figure><h4>4. To improve reliability, we got SRE to do this</h4><p>SREs’ first job was to hold this ship, but that’s all where they got stuck at</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbjP9GGPU2yqU8yzE8IuFA.jpeg" /></figure><h4>5. What Site Reliability Engineers should’ve built is</h4><p>SREs should’ve been *engineering* and *observing* the bridge, but instead they became the bridge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q876JFPzn7ZdFSqc-xokiA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4b74142b25a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/last9/much-that-weve-gotten-wrong-about-site-reliability-engineering-b4b74142b25a">Much that we’ve gotten wrong about Site Reliability Engineering</a> was originally published in <a href="https://medium.com/last9">last9</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infrastructure-as-code-as-Software]]></title>
            <link>https://medium.com/last9/infrastructure-as-code-as-software-a5e4b2b93e8e?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5e4b2b93e8e</guid>
            <category><![CDATA[software-design-patterns]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Mon, 16 Nov 2020 16:40:23 GMT</pubDate>
            <atom:updated>2020-11-16T16:40:23.685Z</atom:updated>
            <content:encoded><![CDATA[<h3>We ran a poll on <a href="https://twitter.com/last9inc/status/1293182598030585858">Twitter</a></h3><blockquote><em>“Do you care about the quality of your infrastructure code?”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Uj3cc96Qst9QVbhTHQ6Uw.png" /></figure><h3>And on <a href="https://www.reddit.com/r/sre/comments/i8gfxv/do_you_care_about_the_quality_of_your_infra_code/">Reddit</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y8_bJ_NZC4_-7cTcTNUM7Q.png" /></figure><p>That’s an approximate and staggering <strong>60–30–10</strong> split.</p><blockquote>What do you think will the response be if the poll was — “Do you care about the quality of your product code?”</blockquote><h3>Reasons</h3><p>We asked a follow up question to reason why ~30% are in the Somewhat but mostly no category and gleaned these reasons from <a href="https://twitter.com/last9inc/status/1293548686925078530">Twitter</a> and <a href="https://www.reddit.com/r/sre/comments/i8gfxv/do_you_care_about_the_quality_of_your_infra_code/">Reddit</a>:</p><ol><li>Someone manually created the legacy infrastructure. No one questioned the practice or broke the tradition.</li><li>Organizations at a small enough scale might feel that it is faster to deploy infrastructure from the cloud provider console than to codify it.</li><li>Infrastructure-as-code’s <a href="https://en.wikipedia.org/wiki/Domain-specific_language">DSL</a> approach makes one think that software development practices don’t apply to it.</li><li>Lack of visibility on the importance and benefits of treating infrastructure code at the same level as product code.</li></ol><p>The last two reasons suggest a mindset gap between how developers approach infrastructure code v/s product code.</p><p>Developers are willing to invest time and effort to refactor product code and make it readable, modular, easy to maintain, etc. But often, these practices are not applied to infrastructure-as-code.</p><h3>Ambition</h3><p>The most common design principles that helped me get better at Software Engineering apply almost 1:1 to Infrastructure-as-code as well.</p><ul><li>📒 Readability</li><li>🍴Separation of Concern</li><li>🔗Loose Coupling</li><li>🤼 Conway’s Law</li><li>📡IPC (Inter Process Communication)</li><li>♻️ Reusability and Abstraction</li><li>🔒 Dependency and Version management</li><li>📄Static Code checkers</li></ul><p>The ambition of this post is to build the mindset that software is software. Be it infrastructure or product, the principles of — modularity, ease of use, reusability, maintainability apply to the “DevOps” code alike.</p><blockquote>To seasoned practitioners, some of these thoughts may sound duh obvious. Maybe the similarity doesn’t appeal to you enough. In that case, I certainly hope that it is something you can refer others to. If the application of these principles resonate with you, bookmark this post and pass it along the next time you have to review that Pull Request and save an explanation of the Idea behind Infrastructure-as-code.</blockquote><h3>Enough talk!</h3><p>The example case is simple: <strong>Deploy a minimal REST API</strong> and evolve its code from the easiest (read: ugliest) to the what we think is production ready.</p><p><em>And, a pretty diagram. Because a pic or it didn’t happen</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/0*GHBTUAk32BdT_EXp.png" /></figure><ul><li><a href="https://aws.amazon.com/api-gateway/">AWS API gateway</a> backed by <a href="https://aws.amazon.com/lambda/">AWS Lambda</a></li></ul><pre>$ curl http://$api_gw_url/test/breakfast | jq</pre><pre>{&quot;breakfast&quot;: {&quot;time&quot;: &quot;9 AM&quot;, &quot;menu&quot;: [&quot;poha&quot;, &quot;chai&quot;]}}</pre><h3>Setup</h3><p>You must have the following installed/cloned on your system:</p><ul><li><a href="https://www.terraform.io/downloads.html">Terraform version 0.12.X</a></li><li><a href="https://stedolan.github.io/jq/">jq</a></li><li><a href="https://www.gnu.org/software/make/">make</a></li><li><a href="https://github.com/last9bot/tf_best_practices_infra_code">tf_best_practices_infra_code</a></li><li><a href="https://github.com/last9bot/tf_best_practices_sample_module">tf_best_practices_sample_module</a></li></ul><p>Each stage in <a href="https://github.com/last9bot/tf_best_practices_infra_code">tf_best_practices_infra_code</a> repo creates the same infrastructure but with code evolved from the previous step. Hence it is necessary if you destroyed infrastructure at the current phase before moving on to the next.</p><blockquote>It would be worthwhile to clone the repositories to see the principles in action as the code evolves. We will go from <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/01_big_main_tf">01_big_main_tf</a> to <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/05_multi_env">05_multi_env</a>, explaining software engineering principles backing up each step in the evolution process. Some seasoned Terraform practitioners may skip the hands-on part and just read through the post to see how each stage builds upon the previous.</blockquote><h3>0/8 🛹 Make it work</h3><p>Code: <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/01_big_main_tf">01_big_main_tf</a></p><pre>  ├── README.md<br>  ├── lambda.py<br>  ├── lambda.zip<br>  └── main.tf</pre><p>All the code is dumped in one main.tf. It may seem unusual to a few, but I have seen many a projects with such structure. Remember:</p><ul><li>When was the last time you wrote your product frontend, backend and controller logic in one big file and shipped that product to your customers? Is your favourite Python module written as one big main.py?</li><li>How will multiple developers collaborate on this code base?</li><li>What happens when one more infrastructure component gets added?</li><li>Even the Javascript world has moved on to <a href="https://github.com/facebook/create-react-app/issues/2206">splitting their js files</a></li></ul><h3>1/8 📒Readability</h3><blockquote><em>We thought of ending the post here and advise the reader to write all their Terraform code in one file.</em></blockquote><p>Then 1998 called — they wanted their design principles back.</p><p>Code: <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/02_split_main_tf">02_split_main_tf</a></p><pre>    ├── README.md<br>    ├── apigw.tf<br>    ├── init.tf<br>    ├── lambda.py<br>    ├── lambda.tf<br>    ├── lambda.zip<br>    ├── outputs.tf<br>    └── variables.tf</pre><p>This code layout reads better than the previous version. We can link the code layout to the architecture diagram and understand that lambda.tf deploys the lambda code and apigw.tf deploys the API Gateway code.</p><p>There are still some limitations:</p><ul><li>We didn’t really improve the code — we just split one big file into multiple ones. <em>One big pizza sliced into six pieces will still have the same amount of calories.</em></li><li>The file split surfaces an important learning: there are groups of related components that should be identified and bundled separately.</li></ul><h3>2/8 🍴Separation of Concern</h3><p>Splitting codebase into multiple files ≠ separation of concern. The following components are tightly coupled, yet they reside in the same folder.</p><p>lambda.py - application code</p><p>lambda.tf - deploys the application code</p><p>apigw.tf - deploys the API Gateway.</p><p>If you had to <strong>redeploy</strong> just the lambda code, you would also need to tinker/no-op the API Gateway along. A software development parallel drives the idea home — every time your application code is updated — you do not redeploy the Load balancer or the database.</p><p>Bundling all infrastructure components together also has a <strong>runtime cost</strong>. If your application deployment code is in the same folder as your database infrastructure code — every small application deployment code change will also lead to checking database code. Some changes to AWS components like the AWS API gateway, <a href="http://onsole.aws.amazon.com/console/home">AWS RDS</a> take north of <strong>10 minutes!!</strong></p><p><em>All you had to do was update an environment variable! It reminds me of no better image to describe the situation.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*3EEI0ZRAlEksrWRR.png" /></figure><p>“My infrastructure’s deploying” is the new “My code’s compiling.”</p><h3>3/8 🔗Loose Coupling</h3><p>Let’s take the first swipe at decoupling the infrastructure code and application code.</p><p>Code: <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/03_split_infra_and_app">03_split_infra_and_app</a></p><pre>  ├── README.md<br>  ├── apigw.tf<br>  ├── init.tf<br>  ├── lambda.tf<br>  ├── outputs.tf<br>  └── variables.tf</pre><p>lambda.py <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/app_code">app_code</a></p><pre>  ├── README.md<br>  ├── create-bucket.sh<br>  ├── delete-bucket.sh<br>  ├── lambda.py<br>  ├── lambda.zip<br>  ├── package.sh<br>  ├── upload.sh</pre><p><strong>What is missing here</strong>:</p><ul><li>The deployment process is now split between <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/app_code">app_code</a> and <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/03_split_infra_and_app">03_split_infra_and_app</a></li><li><a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/app_code">app_code</a> creates and deletes its infrastructure via create-bucket.sh and delete-bucket.sh respectively. The lambda code is uploaded via upload.sh.</li><li><a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/03_split_infra_and_app">03_split_infra_and_app</a> <strong>assumes</strong> that the bucket is created and lambda zip file is uploaded.</li></ul><p>The assumption code is shown <a href="https://github.com/last9bot/tf_best_practices_infra_code/blob/main/03_split_infra_and_app/lambda.tf#L24">here</a> and hard coded as follows:</p><pre>s3_bucket     = &quot;test-tf-lambda-s3&quot;<br>s3_key        = &quot;lambda.zip&quot;</pre><blockquote><em>If the app code changes the bucket location, the infrastructure code will have no way of knowing that.</em></blockquote><h3>4/8 🤼 Conway’s law and Service Oriented Design</h3><blockquote><em>Conway’s law states that organization design systems mirror their own communication structure.</em></blockquote><p>The division of teams handling different components is most prevalent in the infrastructure domain e.g. networking teams, database teams, product teams, etc. These teams work independently, and communicate with each other to identify integration points, e.g. Database team speaks with the networking team to know which subnets to use when creating a private network database. More often than not, configuration changes like these are a major contributing factor in repeat issues.</p><p>To map our sample setup infrastructure to teams:</p><ul><li>Application code — business logic code. <strong>Team</strong>: Product development team.</li><li>AWS Lambda — deploys the application code artifact. <strong>Team</strong>: Product development team.</li><li>AWS API Gateway — REST API endpoints definition and integration with AWS Lambda. <strong>Team</strong>: API management.</li></ul><p>Let us evolve our infrastructure code accordingly.</p><pre>But before we move ahead, In a service-oriented design, it’s incredibly crucial to establish:</pre><pre>1. The exchange contracts<br>2. The mode of communication.</pre><h3>5/8 📡Inter Layer Communication</h3><p>Code: <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/04_split_infra_and_app_remote_state">04_split_infra_and_app_remote_state</a></p><p>Terraform resources can access infrastructure information in two ways:</p><ol><li><a href="https://www.terraform.io/docs/configuration/data-sources.html"><strong>data sources</strong></a> For example, if layer one creates a database, another layer can use <strong>aws_rds</strong> data source to query information for the created RDS like the RDS endpoint.</li><li><a href="https://www.terraform.io/docs/providers/terraform/d/remote_state.html"><strong>remote state</strong></a> Instead of going to the last mile of the created AWS component, make a query to a centrally located Terraform remote state file. There are <a href="https://www.terraform.io/docs/backends/types/remote.html">multiple remote backends</a> to choose from, and we have used the most widely used option of <a href="https://www.terraform.io/docs/backends/types/s3.html">AWS S3</a>.</li></ol><p><strong>Stale Pro Tip</strong>: Use S3 + DynamoDB for remote state storage with locking. Without locking, there is always a possibility of multiple developers updating the same state file in parallel and overwriting each other’s changes.</p><p><a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/04_split_infra_and_app_remote_state/01-base">01-base</a> does this work for us. It creates the remote state infrastructure using the <a href="https://github.com/saurabh-hirani/terraform-aws-state-mgmt">terraform-aws-state-mgmt</a> module.</p><p>With remote state addressed, let us revisit the folder layout and understand how it addresses the separation of concerns between different layers.</p><pre>  ├── 01-base<br>  ├── 02-app<br>  └── 03-infra</pre><ul><li>01-base - <strong>base layer</strong> - Foundational infrastructure for storing state - creates S3 state bucket and DynamoDB lock table - <strong>least</strong> frequently updated.</li><li>03-app - <strong>app layer</strong> -Infrastructure which deploys <a href="https://github.com/last9/tf_best_practices_infra_code/tree/master/app_code">app_code</a> in an AWS Lambda - uses base layer infrastructure to store state. <strong>Most</strong> frequently updated as application logic can change without impacting API Gateway infra.</li><li>02-infra - <strong>infrastructure layer</strong> - Infrastructure for API Gateway - uses base layer infrastructure to store state and app layer to link API endpoints with Lambda backend. Updated more frequently <strong>than</strong> base layer but less regularly than app layer.</li></ul><blockquote>Just like a burger where one team builds the bottom bun, another team perfects the patty and offers multiple varieties, and a third team perfects the dressings.</blockquote><h3>⚽ Reading State the hard way.</h3><ul><li>The base layer creates an S3 bucket and a DynamoDB table, which is used by the app and infrastructure layer. Unfortunately, Terraform does not permit variable interpolation while setting state, so we have to initialize the app and infrastructure layer by passing in the bucket name and DynamoDB table name in the command line.</li></ul><pre>terraform init -backend=true \\<br>-backend-config=&quot;bucket=test-tf-state-bucket-1&quot; \\<br>-backend-config=&quot;key=app.tfstate&quot; \\<br>-backend config=&quot;dynamodb_table=test-tf-state-lock-table-1&quot;</pre><ul><li>The infrastructure can refer to the lambda attributes created in the app layer by using aws_lambda_function <a href="https://www.terraform.io/docs/configuration/data-sources.html">Terraform data source</a>:</li></ul><pre>data &quot;aws_lambda_function&quot; &quot;test&quot; {<br>    function_name = var.lambda_name<br>}</pre><pre>function_name = data.aws_lambda_function.test.function_name<br>uri = data.aws_lambda_function.test.invoke_arn</pre><p><strong>What is missing here?</strong></p><ul><li>Every data source lookup involves one or more network calls to lookup the specific AWS resources, extending the terraform plan time. The technique is sufficient only if you have a few resources to refer to, but as the dependent resources increase, the number of Cloud API calls will increase.</li><li>Data source lookups are limited by the implementation — what if you wanted to look up the list of tags applied to your S3 bucket and use the same for tagging your EC2 instance? No data source does that.</li><li>One cannot use data source lookups cannot be used conditionally. A search that yields no resource results will result in an exception.</li></ul><blockquote><em>I have been unfortunate to witness AWS Rate limits exceeding just by querying the data sources, resulting in retries and too sluggish terraform plans</em>.</blockquote><p>One can improve Inter-Layer communication by moving from data.aws_lambda_function.* data source direct AWS component calls to data.terraform_remote_state.* remote state lookup calls.</p><p><strong>Before</strong></p><pre>uri = data.aws_lambda_function.test.invoke_arn</pre><pre>function_name = data.aws_lambda_function.test.function_name</pre><p><strong>After</strong></p><pre>lambda_name = data.terraform_remote_state.app.outputs.lambda_function_name</pre><pre>lambda_invoke_arn = data.terraform_remote_state.app.outputs.lambda_function_invoke_arn</pre><h3>6/8 ♻️Reusability and Abstraction</h3><p>We have come a long way from one big file to using a remote state for state management and lookup between infrastructure layers. However, we still fall short on these questions:</p><ul><li>If you have to deploy the sample infrastructure in different environments e.g. dev, stage, prod — how much of the AWS API Gateway and Lambda code would you duplicate?</li><li>Suppose this wildly original AWS API Gateway + Lambda pattern took the infrastructure world by storm and different teams in your organization wanted to deploy the same API Gateway + Lambda infrastructure for their products. Should they be concerned with the gnarly internals of AWS API Gateway and AWS Lambda intricacies?</li></ul><p><a href="https://www.terraform.io/docs/modules/index.html">Terraform Modules</a> <br>Modules are to Terraform what libraries as to programming — reusable components that can provide uniform functionality.</p><p>In our setup, we will leverage modules to deploy to dev and stage environments.</p><ul><li>Code repo and deployment steps: <a href="https://github.com/last9bot/tf_best_practices_infra_code/tree/main/05_multi_env">05_multi_env</a></li><li>Terrform module repo: <a href="https://github.com/last9bot/tf_best_practices_sample_module">tf_best_practices_sample_module</a></li></ul><p>The caller code layout, along with dev and stage folders, has another folder called common.</p><p>This folder holds default values which can be overriden by each environment specific folder e.g. <a href="https://github.com/last9bot/tf_best_practices_infra_code/blob/main/05_multi_env/common/lambda.tfvars">common/lambda.tfvars</a> has</p><pre>lambda_timeout = 60<br>lambda_memory_size = 128</pre><p>which is overriden at <a href="https://github.com/last9bot/tf_best_practices_infra_code/blob/main/05_multi_env/dev/02-app/input.tfvars">dev/02-app/input.tfvars</a></p><pre>lambda_timeout = 30<br>lambda_memory_size = 256</pre><h3>7/8 📄Static Code Checking</h3><p>A good developer uses code linters, static analysis and document generation tools to bolster her application codebase. Both the <a href="https://github.com/last9bot/tf_best_practices_infra_code">tf_best_practices_infra_code</a> and <a href="https://github.com/last9bot/tf_best_practices_sample_module">tf_best_practices_sample_module</a> use <a href="https://github.com/antonbabenko/pre-commit-terraform">pre-commit-terraform</a> which wraps the following tools:</p><ul><li><a href="https://github.com/terraform-docs/terraform-docs">terraform-docs</a> — Generate documentation from variable descriptions.</li><li><a href="https://github.com/terraform-linters/tflint">tflint</a> — Terraform linter.</li><li><a href="https://github.com/liamg/tfsec">tfsec</a> — Terraform static analysis security checker.</li></ul><p>If you are using MacOS, installing pre-commit via brew upgrades your python to 3.8 - check this issue <a href="https://github.com/antonbabenko/pre-commit-terraform/issues/133">here</a>.</p><h3>8/8 🔒 Dependency Locking</h3><p>Can you imagine releasing your Ruby module without <a href="https://bundler.io/rationale.html">Gemfile.lock</a>? Terraform’s landscape is no different. Along with git based module version locking, you can have version locking on:</p><ul><li>Terraform binary — we use <a href="https://github.com/tfutils/tfenv">tfenv</a> to lock dev machine versions. To enforce specific versions in our codebase we update the terraform block. For example - to lock Terraform version &gt;= 0.12 but &lt; 0.13 and 0.1 we add:</li></ul><pre>terraform {<br>  required_version = &quot;&gt;= 0.12, &lt; 0.13, &lt; 1.0&quot;<br>}</pre><p>This level of locking can be tightened further by specifying exact Terraform versions also.</p><ul><li>Terraform provider locking — Terraform uses <a href="https://www.terraform.io/docs/providers/index.html">providers</a> to expose interfaces to different cloud provider APIs. These should also be locked by using the required_providers block:</li></ul><pre>terraform {<br>  required_version = &quot;0.12.3&quot;</pre><pre>  required_providers {<br>    aws = &quot;2.62.0&quot;<br>  }<br>}</pre><h3>🏁 Conclusion and Secrets</h3><p>That’s about it for the code evolution process.</p><p>Using software engineering principles to make infrastructure code more robust and reliable is a learnable skill and we hope this post will help you achieve that. However, no infrastructure codebase is perfect or has one right way of solving a problem. Our sample repositories are no different. If you think our codebase can be evolved further and have suggestions for improvement, please engage with us in the comment section.</p><blockquote><em>ba dum tish</em></blockquote><p>Most of the stages in <a href="https://github.com/last9bot/tf_best_practices_infra_code">tf_best_practices_infra_code</a> have hidden bugs which don’t reveal themselves in basic testing. See if you can smoke them out. We will do the big reveal in the much awaited sequel — <strong>Terraform from the trenches — Brunch after dinner?</strong></p><p><a href="https://last9.io/"><em>Last9</em></a><em> is a Site Reliability Engineering (SRE) Platform that removes the guesswork from improving the reliability of your distributed systems.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5e4b2b93e8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/last9/infrastructure-as-code-as-software-a5e4b2b93e8e">Infrastructure-as-code-as-Software</a> was originally published in <a href="https://medium.com/last9">last9</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Root Cause Analysis For Reliability: A Case Study]]></title>
            <link>https://medium.com/last9/root-cause-analysis-for-reliability-a-case-study-8a987ed3a31c?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a987ed3a31c</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[fmea]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[root-cause-analysis]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Tue, 03 Nov 2020 02:57:25 GMT</pubDate>
            <atom:updated>2020-11-03T02:57:25.621Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_uc0KsrHJjw7vkMvKdVbeA.png" /></figure><p>Wikipedia defines Root Cause Analysis (RCA) as “a method of problem-solving used for identifying the root causes of faults or problems.”</p><p>Essentially, root cause analysis means to dive deeper into an issue to find what caused a non-conformance. What’s important to understand here is that Root Cause Analysis does not mean just looking at superficial causes of a problem. Rather, it means finding the highest-level cause- the thing that started a chain of cause-effect reactions and ultimately led to the issue at hand.</p><p>Root cause analysis methodology is widely used in IT operations, telecommunications, healthcare industry, etc. In this post, I’ll take you through how to use RCA for making your system more reliable using an experience I had.</p><p>However, before discussing the case study, a note on the importance of RCA in Site Reliability Engineering.</p><h3>Why should you use Root Cause Analysis for Reliability?</h3><p>A good Root Cause Analysis looks beyond the immediate technical cause of the problem and helps you in finding the systemic root cause of the issue. And then you work on eliminating it. RCAs along with documenting the core issue and steps to resolve it — also lends itself to the very next question an SRE will ask — is this happening somewhere else? Automating the solution and deploying it to affected areas ensures that not only is the problem’s root cause taken care of but also problems of similar nature are permanently fixed.</p><p>That being said, a not-so-thorough RCA makes the entire process futile. Here’s an incident I encountered a few years back and how my team went about doing a Root Cause Analysis for it.</p><h3>How Things Went Down</h3><h3>Elasticsearch wakes up Pagerduty wakes up Oncall</h3><p>It was one of those times for me where I thought everything was in order, working fine. And yet, a small slip led to an unexpected breakdown. It was around 7:30 AM, almost 25 hours before a country launch and PagerDuty went off. Buzzers kept going on that something’s breaking. So, we went to Elasticsearch and found five or six 5XX requests.</p><h3>Pagerduty auto resolves — good news or bad news?</h3><p>There were a lot of logs coming in around 1 Mbps. There wasn’t a correlation ID so we couldn’t actually isolate from the tons of logs where the issue was exactly happening. Precisely 5 minutes later, HTTP 500 error stopped and Pager duty auto resolved. We thought it’s probably not a major issue and de-prioritised it.</p><h3>Pagerduty wrath unleashes in rinse repeat mode</h3><p>Five minutes later, exactly sharp by the clock, Pingdom starts alerting PagerDuty again that the public API is unreachable. We had set up all the tools we could possibly think of. There was Grafana, which was sending us some 5xx requests. The alerts were going to Sentry, through the standard route Elastisearch&gt; Elastalert&gt; Sentry. The issue keeps going on until exactly 7:45 AM, five minutes later. 500 stops and the Pagerduty is auto-resolved. This keeps repeating.</p><p>The first thought we had was whether the issue was due to a new deployment. It’s often the easiest answer to these problems. The release manager as well the on-call SRE deny any new deployment. Not just that, all other tools were doing okay. Grafana looked okay, Sentry was doing its job, Prometheus and APM were doing okay. We even checked the firewall to figure out whether there was a drop in traffic, but there wasn’t.</p><h3>But what really happened?</h3><p>Nearly <strong>20 hours</strong> of struggling to find the problem, we realised that there was a simple mount command which hadn’t run on one of the database shards. And because of that, data was being written in-memory. When the system rebooted, data was wiped. Now this machine wasn’t supposed to be commissioned, but we had fixed the machine. So we put it into circulation.</p><p>However, Ansible did not run one of its commands as it should have.</p><p><em>So one machine was left out, one command did not run and the machine rebooted.</em></p><p>What we noticed here is that only a certain section of the data was gone. So only a certain type of request started failing, which disconnected the faulty node from the load balancer as an unhealthy upstream.</p><p>Now that we had figured it out, the question was how to fix this. But before going there, two more important questions needed to be answered:</p><ul><li>If there’s a mistake in one place, it’s quite likely the mistake was repeated in other places. Where else was this failing?</li><li>How am I going to avoid the situation ever from happening again?</li></ul><blockquote><strong>Systems are designed by culture.</strong></blockquote><h3>Digging Deeper: The Root Cause Analysis</h3><p>We found out that this was a gap in our first bootstrap run by Ansible. We used Ansible to run commands. But when the (faulty) machine was brought into circulation, one of the Ansible runs hadn’t happened on it. It was a newly provisioned machine and service discovery did not pick it up.</p><p>We were using Nomad to schedule dynamic workloads. This meant that request could go to any of the deployed machined, faulty one’s included.</p><p>We could have gone old school, used none of these, and avoided the situation. But that wasn’t really an option. We had dynamic workloads and needed a clustered scheduler like Nomad.</p><p>But our RCA didn’t end there. A good root cause analysis, identifies the business loss as well. Was it data loss? Did it cause a significant amount of embarrassment, if not financial loss? Was there financial loss that couldn’t be measured because the system was down?</p><h3>Probable Root Cause 1: Raghu forgot to execute the mount command</h3><p>A common outcome of RCA is that we end up blaming individuals. We did mention this: Raghu forgot to execute the mount command. What we must realize is that individuals cannot be reasons for failures. So the conclusion of our RCA <strong><em>cannot</em></strong> be that an individual failed to do something. It must be more actionable and concrete.</p><h3>Probable Root Cause 2: Infrastructure team forgot to execute the mount command</h3><p>We took another stab at it. We said that the infrastructure team forgot to execute the mount command. But this RCA, too, is not actionable.</p><blockquote><strong>Is improving the Infrastructure Team’s memory a probable resolution?</strong></blockquote><p>Another thing to consider with the above two causes is the <a href="https://en.wikipedia.org/wiki/Bad_apple"><strong>bad apple theory</strong></a>. Change the existing team or replace the team members? The Bad Apple theory states that if you take out a bad apple from a basket, what remains is a basket of really good apples. Most of the time changing people doesn’t really solve anything. We have to be pragmatic and cognizant of the fact that skill and ownership always go hand in hand. You have to have the right skills in people and you have to give them the right ownership. Along with the right tools. This solution would, therefore, have not reaped the right results. <em>Human failures are inevitable.</em></p><h3>Probable Root Cause 3: Remote Login via SSH Allowed</h3><p>We then went a step deeper and asked the question: Why was it possible to SSH into the system? If SSH was the issue, a probable resolution could be to not allow SSH. But that would actually hinder work.</p><p>There’s a reliability team, there’s a DevOps team and there’s a development team working on the system. They are invariably going to SSH into the system.</p><blockquote><strong>Preventing somebody from doing their job is not going to make the system reliable.</strong></blockquote><h3>Probable Root Cause 4: Right Tools Were Not Present</h3><p>We then asked another question: why don’t we have a tool which actually matches alerts on a configuration mismatch? What if all my systems in production, had a way to match their configurations and raise an error every time there was a mismatch? That would be the best thing, right?</p><p>This way people are free to automate. Are not burdened. And can actually make decisions freely because the mundane work of configuration validation has been offloaded to a computer. As long as there are humans in charge of the system, there will be mistakes.</p><p>So the real issue here was that we didn’t have the right tools to catch the smaller details a human could easily overlook.</p><blockquote><strong>Anything that you’re not paying attention to is going to fail at some point or the other.</strong></blockquote><h3>The final root cause analysis</h3><h3>System configuration validator was absent</h3><p>There was no system to actually validate the state of the machines every now and then.</p><p><em>Solution</em>: We built it. It became one of our bread and butter tools. Everytime we had a launch, we actually started running those configuration validators to see if everything looked okay. We could check if all ports could actually scan all machines and if the connectivity was fine. And the scope kept increasing. One of the greatest tools that fit here is <a href="https://osquery.io/"><strong>osquery</strong></a>.</p><p>It is a great tool by Facebook where you can actually make SQL queries to systems. Then, you can match them across systems and see which one’s are working fine, if there’s any difference and so on.</p><h3>FMEA did not exist</h3><p><a href="https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis"><strong>Failure mode effective analysis</strong></a> is a slightly out-dated term borrowed from the industrial sector. The cool kids these days call it chaos engineering. FMEA means we run an application in Failure Mode and see its impact. If we could actually run these systems against failure scenarios in our test environments by using fault injection techniques, we would have probably caught the issue.</p><h3>Non-Latent Configuration Validator was not available</h3><p>Non-latent configuration validation is an approach which advocates early detection of invalid data to ensure fast failure and prevent the situation where an invalid configuration causes bigger damage. For example, if a bad URL is detected during system initialization, there’ll be far less damage than if a customer workflow invokes it.</p><p>We realized that a non-latent configuration validator was missing.</p><blockquote><strong>Almost 72% of the failures could have been reduced if configuration validation was not latent!</strong></blockquote><p>These three we identified as the root cause of this situation. And when we rectified these, that is, when we added the necessary tools we did not encounter similar issues again.</p><blockquote>That’s the power of a strong RCA. And to do it, you need to look beyond individuals, teams, and superficial causes and find actionable solutions.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a987ed3a31c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/last9/root-cause-analysis-for-reliability-a-case-study-8a987ed3a31c">Root Cause Analysis For Reliability: A Case Study</a> was originally published in <a href="https://medium.com/last9">last9</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Percentiles are incorrect P99 of the times.]]></title>
            <link>https://medium.com/last9/your-percentiles-are-incorrect-p99-of-the-times-11436c97d524?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/11436c97d524</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[latency]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Tue, 03 Nov 2020 02:44:01 GMT</pubDate>
            <atom:updated>2020-11-21T07:13:32.422Z</atom:updated>
            <content:encoded><![CDATA[<p>In an earlier Post, I talked about how <a href="https://medium.com/last9/slos-that-lie-128db934d59b">SLOs can be misleading</a>, and the Service Level Indicator in consideration was Uptime. There is another SLI which is almost impossible to be accurate about, Latency.</p><p>Like Uptime is measured as % and aggregated over a month/year/week, based on time window choice, Latency is for a unit of time (ms and s.), and the preferred aggregate is <strong>percentile</strong>.</p><p>The purpose of this post is to debunk common mistakes that I did while dealing with Percentiles.</p><blockquote><em>Why is it important to understand percentiles in depth? Because one of the critical Indicators of software performance, Latency, is measured using Percentiles. Somebody cannot deal with something as clinical as performance without understanding the behavior of its yardstick.</em></blockquote><h3>But I use &lt;foo&gt;_tool that calculates the percentiles. Should I care?</h3><p>All of these tools, solutions do the same things. They assume a user to have a statistical understanding of these constructs. They are not wrong, nor have they been lying. That is just how math works <strong>¯\<em>(ツ)</em>/¯.</strong></p><h3>What is a Percentile?</h3><p>In everyday conversations, Percentage, Percentile, and Percentile Rank are invariably confusing terms. Partially because the only time you encountered them before SRE became a fad was during Management or Engineering entrance examination results.</p><blockquote><em>Given a series of records, a percentile is a value greater than that many percentages of the records.</em></blockquote><p>Example: 20th Percentile is a value greater than 20% of the values.</p><p>Alternatively, In a room of 20 people, if you are the fourth tallest person, then your height is the 80th percentile, i.e., 80% people == 16 people are shorter than you.</p><h3>Some common confusions with Percentiles</h3><h3>What is P100?</h3><p>We use this one fairly liberally, but in reality there <strong>cannot be a 100th percentile</strong>.</p><p>The percentile value is supposed to be greater than 100% of the values, including the value. Hence there cannot be a P100.</p><p>Say there are 100 distinct records, the last value is better than 99 records hence, P99 but not P100. In a sample-set of 10000 distinct values, the highest value is greater than 9999 others. Hence P99.99 but not P100.</p><p>If you are the tallest person in a room of 20 people, you are NOT taller than 100% of the people.</p><blockquote>P100 is often confused with MAX. Using MAX is fine. Labeling it as P100 is a misnomer.</blockquote><h3>What is P0?</h3><p>The 0th percentile is way more confusing than the 100th percentile.</p><p>Suppose you had a scale from 0 to 100. Including both 0 and 100, <strong>there would be 101 values</strong>!</p><p>Imagine there are 100 distinct records; they can either be 0 to 99 or 1 till 100. The simplest formula for percentile is that the nth record in N records denotes the Pth percentile</p><blockquote>As P100 is often used interchangeably with MAX, P0 is used with MIN.</blockquote><h3>What about P95?</h3><p>Say there are 100 distinct records, 95th percentile is the one which covers 95% of the records. So 95th value.</p><p>But in 10 distinct records, the 95th percentile cannot a discrete value. So what would the result be?</p><blockquote>Why is this important?<br>When was the last time your application gave your exactly multiple of 100 records when you wanted to calculate P90!</blockquote><p>There are <strong>two ways</strong> to calculate the value of percentile when your record-set is not a multiple of 100:</p><ul><li><strong>Discrete (Nearest Rank)<br></strong>In the absence of a discrete data point, use the nearest value using a rounding technique. Example: In a record-set of 10 records, since you would never find 95, you can round off to the 9th or 10th value, but neither is the authentic P95.</li><li><strong>Interpolated<br></strong>In the absence of a real data point, one may use an appropriate <a href="https://en.wikipedia.org/wiki/Curve_fitting">Curve-Fitting</a> technique to fill the missing values. (<a href="https://www.dummies.com/programming/big-data/data-science/how-to-use-curve-fitting-in-predictive-analytics/">Another</a> good read)</li><li>One can achieve Curve fitting with one of these techniques:<br>1. By finding an <strong>exact fit</strong> for every data point (a process called <strong>interpolation</strong>)<br>2. By <strong>drawing trends</strong> out of the data<br>3. By employing <strong>data smoothing</strong> to come up with a function that represents the smoothed graph</li></ul><h3>Discrete vs. Interpolated Percentiles</h3><pre>What’s the missing value?</pre><pre>1,2,3,4,5,6,?,8</pre><p>Chances are, you will say 7. Which may or may not be incorrect, but it is the most <em>reasonable</em> fill.</p><p>For a demonstration of these samples, I will often use standard SQL as an example, but the principles are similar to a data processing tool/framework of your choice. Be it Pandas, or Excel too!</p><blockquote>You would be surprised how often you consume Interpolated percentiles, more than the discrete percentiles.</blockquote><p>To demonstrate, I will show an example of how percentiles are computed in Postgres. Assume a table which has the following data</p><pre>SELECT * from test;                                                                                                                                           <br>| a | b |<br>| 1 | a |<br>| 2 | a |<br>| 3 | a |<br>| 4 | a |<br>| 5 | a |</pre><p><strong>Nearest Rank (Discrete)</strong></p><pre>SELECT<br>percentile_disc(0.8)<br>WITHIN GROUP (ORDER BY a)<br>FROM test;<br>                                                                                           <br>|-----------------|<br>| 4               |</pre><p><strong>Curve Fitting (Continuous)</strong></p><pre>SELECT<br>percentile_cont(0.8)<br>WITHIN GROUP (ORDER BY a)<br>FROM test;<br>                                                                                              <br>|-----------------|<br>| 4.2             |</pre><p>You would expect both the percentiles to be the same, but continuous interpolation, while may incorrect to the naked eye, is far more useful in almost all cases.</p><p>For the discrete to be accurate, your record-set has to have a perfect multiple of the desired percentile number of records, which will rarely be the case.</p><blockquote><em>Either way, the percentile is a rounded-off value OR an interpolated value. Both of them will be approximated and never accurate.</em></blockquote><h3>How are Percentiles limiting in context of Operations</h3><p>In the context of SLIs (Service Level Indicators), data needs to be computed from multiple requests that arrives in multiple bursts. For cognitive ease, let’s say for a system that receives ten requests/second, ten events are happening per second, which will emit data worthy of being processed.</p><p>For every request that comes in, you need to</p><ul><li>Provide throughput, using request COUNT</li><li>Provide latency, using AVERAGE response time,</li><li>Provide Latency degradation using 90th percentile, and maybe others!</li></ul><p>Here’s a pictorial representation. Every point, <em>Tx</em>, brings in some features, namely status, and request_time. How do these get converted into metrics and dashboards?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zn2K1Zxe5G3U0X5D" /></figure><blockquote>CSV is just for the illustration of the data. It does not imply that we should save data as CSV.</blockquote><p>Data, per request</p><pre><strong>server,timestamp,status,latency</strong><br>a,1599962400,200,0.5<br>a,1599962401,200,0.45<br>a,1599962401,304,0.2<br>a,1599962401,500,0.8</pre><p>Needs to be <strong>folded</strong> into something like this</p><pre><strong>server,timestamp,throughput,2xx,3xx,4xx,5xx,avg_latency</strong><br>a,1599962400,2,3,6,0,0,0.5<br>a,1599962460,7,6,10,1,2,0.48</pre><p>(per-minute aggregation)</p><blockquote><em>Either the LoadBalancer, the web-server, or the reverse-proxy performs these aggregations for you and emits an interval value or they spit out raw log lines and allow you to extract this Information.<br>Either way, an aggregation is needed.</em></blockquote><p>To club the requests arriving together, a brief introduction to <strong>Aggregation</strong> will help</p><h3>A quick rundown of aggregations</h3><p>Given the record-set, One can apply different aggregation functions on the attributes.</p><ul><li><strong>Throughput</strong> For a given time range, a SUM / COUNT of the number of requests is enough</li><li><strong>Average Latency</strong> For the same time range, Summation of the response times / <strong>throughput</strong></li></ul><p>So far so good. For a given time range we can apply these simple functions to pull the desired metric value.</p><p>Now how about,<br>What was the average latency from 7:30 - 7:45 PM 3 weeks back?</p><p>There are <em>ten events/sec</em>, around <em>106060/hr</em>, around <em>250K/day</em>, about <em>1.5m/week</em>, and about around <em>6 million/month</em>. Per source!<br>Assuming there are 10 services, each query will aggregate and filter <strong><em>6 million * 10 = 60 million rows</em></strong>.</p><p>We know, we surely cannot retain every event to recompute each time. Hence, We need to look elsewhere for an answer.</p><p>Two concepts required to be understood before we hunt that further</p><ul><li>Streaming Aggregation</li><li>Downsampling</li></ul><h3>Streaming Aggregation</h3><p>It’s impossible to query the ENTIRE data range to produce an aggregate. A simpler techniques will be to Incrementally aggregate the Incoming data by using functions that save some state from the previous run, and resume without caring for the entire record-set.</p><p>Calculate the throughput (SUM)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*obMKNyDDJWHm_hnp" /></figure><p>Or, <strong>Moving Average</strong> for a stream of input</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5OJEn5Eo-iqIbAmb" /></figure><h3>Down-sampling</h3><p>All granular analysis requires only recent data. Whereas older information is used to compute trends.</p><p>Downsampling is the process of reducing the resolution of data. In our example data there are 6 million records per month. If we decide to keep data per minute instead of per second, it will reduce the number of records by 60x. 6 million rows to 100K rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lTzh4DOTsqVwYf0r" /></figure><p>As information gets stale, queries usually shift towards trends compared with Needle-in-haystack search. So it’s wise to start folding the data. <strong>This process is called downsampling.</strong></p><p>As obvious as it may seem, downsampled data lose all visibility into the distribution of data within the sample.</p><p><strong>Example:</strong> If downsampling is performed for a record-set of 5 seconds (1, 2, 1, null, 1) using the SUM every 5 seconds, Sum will be 5.</p><p>Also, no longer you can tell that 4th second didn’t produce any data. It’s a trade-off between accuracy vs. performance. If you pay attention, at any severe scale, it is just inevitable to get accustomed to approximations.</p><p>Downsampling needs two components</p><ul><li><strong>Interval(s)</strong> Clubbing data across the Time range using the function. ****</li><li><strong>Aggregation Function</strong> Very much like Streaming aggregation, a mathematical function that can merge records.</li></ul><p>Example: To downsample status_2xx and status_5xx, you could use the mathematical function <strong>SUM</strong></p><p><a href="https://www.notion.so/37b783d5ca8f47c5a3189adeb5777431">I</a>nput</p><pre>| timestamp | status_2xx | status_5xx |<br>|-----------|------------|------------|<br>|         0 |          5 |          1 |<br>|        10 |          5 |          2 |<br>|        20 |            |          1 |<br>|        30 |          1 |          2 |<br>|        40 |         10 |          4 |<br>|        50 |         20 |          2 |<br>|        60 |          5 |          4 |</pre><p>Downsampled every minute</p><pre>| timestamp | status_2xx | status_5xx |<br>|-----------|------------|------------|<br>|         0 |         41 |         12 |<br>|        60 |          5 |          4 |</pre><p>Downsampling can be applied across multiple intervals using the same aggregate function. Example: Ancient data can be downsampled to a day, using the same mathematical function that was is used to downsample second → minute → hour</p><p>Downsampling works well when using mathematical aggregation functions like SUM, MIN, MAX, COUNT, AVG, and MEAN.</p><p>But what about Percentile?</p><h3>Can Down-sampling be applied to Percentiles?</h3><p>Say we only save P50, P75, and P90 of records</p><p>Saved</p><pre>| timestamp | requests | p50  |  p75  |  p90  |<br>|-----------|----------|------|-------|-------|<br>|         0 |      100 | 50ms | 100ms | 500ms |<br>|        60 |      200 | 30ms | 50ms  | 200ms |</pre><blockquote><em>What would the resultant P50 (or P75) when you club these two records together?</em></blockquote><p>You cannot tell. You would need the ENTIRE record-set to re-compute the percentiles.</p><p><strong>Why do you need the entire record-set?</strong></p><p>P50 wants the latency for 50% of both the samples. i.e 100+200/2 = 150 requests.</p><pre>- 50 requests from first sample are ≤ 50ms<br>- 100 requests from the second sample are &lt; 30ms.</pre><p>Now these two may add up to make the lowest 150 requests of the ENTIRE sample set, in which case <br><strong>P50 will be 50ms</strong></p><p>However, another distribution where</p><pre>- 10 requests from first sample are ≤ 20 ms<br>- 100 requests from second sample are ≤ 30ms<br>- Another 40 samples from the second sample are &lt; 45ms.</pre><p>In this case, the lowest 150 requests of the entire sample set came mostly from the 2nd sample and weighed ≤ 45ms. In this case the<br><strong>P50 will be 45ms</strong></p><p>You see what happened here. Without knowing the entire distribution, p50 across multiple sets will be an uneducated guess.</p><p>Any attempt to deduce this, like weighted-average, mean, or median, will be smart but inaccurate. So we need the <em>ENTIRE</em> data set, but we concluded earlier that it&#39;s infeasible to use the <em>ENTIRE</em> data set to re-compute on each request.</p><p><strong>So what’s the solution?</strong></p><blockquote><em>You approximate. Because, </em><strong><em>your Percentiles are incorrect P99 of the times!</em></strong></blockquote><pre>Now that we have sufficient foundation of percentiles, as the go-to metric for latency, the question has to be asked — Is there a better way to approach latency calculations? There is and we will soon be adding our findings on <a href="https://blog.last9.io/">blog.last9.io</a>. </pre><pre><a href="https://blog.last9.io/sre-tooling-clever-hans-fallacy/#stories-from-the-world-of-sre-delivered">Subscribe to our newsletter</a> and keep that popcorn on standby.</pre><p><a href="https://last9.io/"><em>Last9</em></a><em> is a Site Reliability Engineering (SRE) Platform that removes the guesswork from improving the reliability of your distributed systems.</em></p><p>Still having second thoughts about subscribing to our newsletter? Maybe <a href="https://blog.last9.io/sre-tooling-clever-hans-fallacy/">this</a> post on how number crunching horses relate to SRE tooling will change your mind. Also, did we mention that we are hiring to build the next gen SRE observability platform? If you are interested reach out to us on <a href="https://twitter.com/last9inc">twitter</a> or email us at hello@last9.io</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11436c97d524" width="1" height="1" alt=""><hr><p><a href="https://medium.com/last9/your-percentiles-are-incorrect-p99-of-the-times-11436c97d524">Your Percentiles are incorrect P99 of the times.</a> was originally published in <a href="https://medium.com/last9">last9</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SLOs that lie]]></title>
            <link>https://medium.com/last9/slos-that-lie-128db934d59b?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/128db934d59b</guid>
            <category><![CDATA[availability]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[uptime]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Tue, 03 Nov 2020 02:34:08 GMT</pubDate>
            <atom:updated>2020-11-03T02:34:08.680Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jz98uB8izQKEnowZpM9MPA.jpeg" /></figure><p>SLO is an acronym for Service Level Objective. But before I explain SLO, you need one more acronym <strong>SLI</strong> (Service Level Indicator)</p><p>An SLI is a quantitative measurement of a (and not the) quality of a Service. It may be unique to each use-case, but there are certain standard qualities of services that practitioners tend to follow.</p><ul><li><strong>Availability</strong> The amount of time that a service was available to respond to a request. Referred to as <strong>Uptime</strong></li><li><strong>Speed</strong> How fast does a service responds to a request. Referred to as <strong>Latency</strong></li><li><strong>Correctness</strong> Response alone isn’t good enough. It also matters whether it was the right one. Referred to as <strong>ErrorRatio</strong></li></ul><p>SLOs are boundaries of these measurements where you would begin to worry about your service. Example: If the speed of delivery at a restaurant falls below 5dishes per hour. You know customers are going to have longer waiting periods. Having the right SLOs help you take better decisions (More in <strong>this</strong> blog)</p><h3>Uptime</h3><p>One of the primary indicators and objectives is <strong>Uptime</strong>. If the site isn’t serving, let alone the speed and accuracy. Also, why uptime is usually desired to be towards 100%.</p><blockquote>If you are a restaurant that sees footfall throughout the day, why shut and lose business.</blockquote><p>How do you compute uptime? Fairly straightforward. The time that you were up / total time.</p><blockquote>x = Up / ( Up + Down ) %</blockquote><p>99% uptime means out of <strong>365</strong> days * <strong>24</strong> hours * <strong>3600</strong> seconds you have <strong>1</strong>% allowance to not be available. Over a year that is 3.7 (~4)days.<br>But would you be OK remaining shut for consecutive 4 days from 25th December — 28th December?</p><p>If the answer is NO, then this is not a ration that you can keep accumulating. It’s more like a periodic allowance that you get a refill on, too bad if you couldn’t spend it.</p><p>Based on each bucket, the downtime allowance may seem more palpable.</p><pre>┌─────────┬────────────┬───────────┬───────────┬───────────┐<br>│   9s    │  per Day   │ per Week  │ per Month │ per Year  │<br>├─────────┼────────────┼───────────┼───────────┼───────────┤<br>│ 99%     │ 14.4 Mins  │ 1.7 Hrs   │ 7.3 Hrs   │ 3.7 Days  │<br>│ 99.9%   │ 1.4 Mins   │ 10.1 Mins │ 43.8 Hrs  │ 8.7 Hrs   │<br>│ 99.99%  │ 8.6 Secs   │ 1 Mins    │ 4.4 Mins  │ 52.6 Mins │<br>│ 99.999% │ 0.864 Secs │ 6.1 Secs  │ 26.3 Secs │ 5.3 Mins  │<br>└─────────┴────────────┴───────────┴───────────┴───────────┘</pre><p><strong>How do you tell that service is up?</strong></p><p>A request comes in, the request was served. Your service is <em>Up</em>. There are plenty of tools you can use for this including <a href="https://prometheus.io/">Prometheus</a>, <a href="https://cloud.google.com/products/operations">Stackdriver</a>, etc. These tools piggyback on the software components to emit a logline or a metric.</p><p><strong>How do you tell the service is down?</strong></p><p>A request comes in, but it wasn’t served. For a request that wasn’t served, there won’t be a logline or a metric value emitted.</p><blockquote>In a store that is supposed to be up 24x7, the staff bunks randomly, how do you measure how long were they away for?</blockquote><h3>Option 1: SDK (Measure at each caller)</h3><p>We could use an SDK that tracks every outbound request to our service. Depending on the design, one may be using <a href="https://envoy-mobile.github.io/">https://envoy-mobile.github.io/</a> or a segment.com to emit constant metrics. But,</p><ul><li>In a world where <a href="https://www.mulesoft.com/infographics/api/rise-application-programming-interface">everything is becoming an API</a>, the control you may exercise over the callers is diminishing.</li><li>There will be so many senders!! Before concluding from that pattern, whether it was a problem with the sender or one of your receivers, one would have to wait for the data to arrive and wait for it to be collated.</li><li>What if the sender’s network is jittery. Say 1% of the senders had an ISP fault and the stats never make it to your aggregates.</li></ul><blockquote>Waiting on a response to be aggregated across 100% of callers, with their own delay, where the goal is a 99.99% uptime SLO with only 4 minutes of downtime available, you may have already lost the uptime target.</blockquote><p>Clearly, using this method will weaken the definition of uptime, where 99.99% will feel like a joke.</p><pre><strong>Firewall   --&gt;   Load Balancer   --&gt;   L7 Proxy   --&gt;   Handler</strong></pre><p>In reality, the request never really reaches your server directly. There will be a Firewall upstreaming to a Load Balancer upstreaming to a L7 proxy upstreaming to your servers.</p><blockquote>SLOs is an aggregate of all layers underneath</blockquote><p>We should be talking about the SLO of each layer before the service as a whole. A breach in <strong>Uptime</strong> SLO of a backend service would impact NOT the uptime but the <strong>ErrorRatio</strong> of the calling L7Proxy. Similarly, if the L7Proxy is down, Load Balancer’s <strong>ErrorRatio</strong> will increase and not the <strong>Uptime</strong>. All the way till you reach the CDN, probably.</p><blockquote>So, Uptime (as customer experiences) is best measured at the layer which is closest to the Customer and farthest from your code.</blockquote><p>Each such layer, we are probably monitoring the uptime of a component which is not our core business unit and outside our control. In the -&gt; LB -&gt; CDN -&gt; … journey, we have probably lost the uptime essence of the actual code deployed.</p><blockquote>It’s possible that the business calls the SLO <strong>Uptime</strong>, but instead what you are addressing is the <strong>ErrorRatio!!</strong></blockquote><h3>Option 2: Uptime (actually Downtime) Monitors</h3><p>This is where you introduce uptime monitors and downtime checkers. Simple services that have existed forever, but extremely crucial. Outsource the trust of uptime to these services where some poor bot(s) have been assigned the mundane work of periodically hitting your service endpoint.</p><blockquote>Uptime = timeUp / ( timeUp + timeDown ) %</blockquote><p>This introduces two compromises:</p><ul><li>Uptime is now as the monitor sees it, <strong>Not</strong> how the real customer sees it. Those 1% of ISP faults may still be affected.</li><li>How about the reliability of the uptime monitor? If we have trouble staying up 100% of the time, sure they cannot be up 100% of the time.</li></ul><blockquote>Say, uptime monitor guarantees an uptime of 99.99%, What if those 4 minutes don’t overlap? For the 4 minutes that the uptime monitor was down for, your service may be up or down, the monitor would not know.</blockquote><p>You don’t keep all your eggs in the same basket. We introduce a multi-geography downtime monitor. Say a downtime monitor requests from 4 Geos.</p><ul><li>Should a failure across 1 Geo- be called a downtime?</li><li>What if one of the Geos is highlighting a downtime for a geographical region?</li><li>Also, <strong>CAP</strong> and Network failures.</li></ul><p>Downtime monitors aren’t holier-than-thou. They go through glitches too. They need to retry too. Say if a request failed, wisdom says retry. Wisdom says Hystrix. But what about the failed attempt? Was it a failure or was it counted as success?</p><p>Before we proceed, there is a frequency too. How often do you check? We cannot go per second. We cannot go per minute. It’s a balance that you pick.</p><ul><li>The faster you check, the shallower will be your health check.</li><li>The slower you check the deeper can be your health check.</li></ul><p>Depth of a health check is basically a trade-off you make. A shallow check only checks for a static response. A deeper health check will check for a DB operation in and out. The DB call obviously will take considerably longer and eat a transaction. You can’t have that come in every 10 seconds.</p><p><em>The frequency vs depth is an argument that doesn’t have one right answer. And like other things, you may just need both.</em></p><h3>Option 3: State-based Monitors</h3><p>Because the uptime is not requesting every ms, we count the time between states of the service.</p><p>We need states. <strong>OK</strong>, <strong>Unknown</strong>, and <strong>Error</strong>.</p><p>Error is confirmed down. The Unknown could be any emission that was interrupted or pending for retry or the duration that was in-between maintenance.</p><ul><li>Time spent between <strong>OK</strong> and <strong>Unknown</strong> is <strong>OK</strong> since you cannot tell for sure.</li><li>Time spent between <strong>Unknown</strong> and <strong>Error</strong> is <strong>Error</strong>.</li><li>Time spent between <strong>Error</strong> and <strong>Unknown</strong> is <strong>Error</strong> since you cannot tell whether service is Up or not.</li><li>Similarly, the time between <strong>Error</strong> to <strong>OK</strong> is considered down, since it was down.</li></ul><p>This in itself is the first step of aggregation.</p><p>Let’s take these two situationss</p><ol><li>Counted as <strong>OK</strong> for 20. Uptime = 100%</li></ol><pre>┌────────┬──────────┬──────────┬──────────┬──────────┐<br>│ Status │    OK    │    OK    │ Unknown  │    Ok    │<br>├────────┼──────────┼──────────┼──────────┼──────────┤<br>│ Time   │ 10:00:01 │ 10:00:10 │ 10:00:11 │ 10:00:21 │<br>└────────┴──────────┴──────────┴──────────┴──────────┘</pre><p>2. This will be counted as <strong>OK</strong> for 10 secs and <strong>Down</strong> for 10 secs. Uptime: 50%</p><pre>┌────────┬──────────┬──────────┬──────────┬──────────┐<br>│ Status │    OK    │ Unknown  │   Down   │    Ok    │<br>├────────┼──────────┼──────────┼──────────┼──────────┤<br>│ Time   │ 10:00:01 │ 10:00:10 │ 10:00:20 │ 10:00:21 │<br>└────────┴──────────┴──────────┴──────────┴──────────┘</pre><p>You can keep these rules to be configurable per service. But these are rules and subject to Interpretation. The absolute 99.99% is long gone!</p><blockquote>I have not discussed the condition where downtime monitor’s sleep period overlaps with your actual downtime. A unique situation where every hit that comes in is only a fraction of the actual flapping status. The uptime figures that will come, are going to be really skewed from reality.</blockquote><h3><strong>Conclusion</strong></h3><ul><li>There is not one single SLO. They are formed at layers, and uptime SLO of one could be error SLO of another.</li><li>The uptime number is massively aggregated, and always approximate.</li><li>As your uptime reaches the higher 9s, the support structure and the mindset needs to shift towards proactive efforts, since waiting on an outage and then reacting to bring it up will not always work.</li></ul><p><a href="https://last9.io/"><em>Last9</em></a><em> is a Site Reliability Engineering (SRE) Platform that removes the guesswork from improving the reliability of your distributed systems.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=128db934d59b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/last9/slos-that-lie-128db934d59b">SLOs that lie</a> was originally published in <a href="https://medium.com/last9">last9</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Processes]]></title>
            <link>https://medium.com/trusting-social-engineering/engineering-processes-629e05880ffb?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/629e05880ffb</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Fri, 16 Nov 2018 08:37:25 GMT</pubDate>
            <atom:updated>2018-11-17T08:21:51.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yze3mrsSOVgyiJPzGa5YDw.jpeg" /></figure><p>Engineering teams like to move fast. Multiple products with a variety of projects, deployments happen almost every day. Ensuring that the chaos does not take over sanity and lead to multiple failures requires carefully thought out processes. And while most of the teams have processes in place — right from the dev stage to the production stage — ensuring that the processes are followed becomes a challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*wSXS8JfSs9dwavDSpoMakA.png" /></figure><p>There are checklists in place before a new update is pushed to the production stage. Following the checklist mitigates the chances of things going wrong and establishes that basic hygiene factors were taken care of.</p><p><a href="https://gist.github.com/meson10/55e9ac01049c352185822b63a8e6ffd1">Here’s a sample checklist we’ve created for deployment.</a></p><p><strong>Types of Deployment</strong></p><p>Deployment usually falls under two buckets:</p><ol><li><strong>Infrastructure Change — </strong>includes things like NAT change, VPN Change, Backup, Maintenance, adding more infrastructure etc.</li><li><strong>Code Change — </strong>includes bug fixes, feature updates and new additions to the product code base.</li></ol><h3>The Challenge: Ownership and Accountability</h3><p>Each type of deployment happens under the purview of a stakeholder. Product changes are usually presided over by Product Owners whereas Infrastructure update happens under the purview of the Infrastructure Stakeholder.</p><p>The final accountability of the updates lie with the stakeholders. But because the teams are big, and communication can be tedious — there can be cases when the stakeholder is not completely aware of a new update going to production.</p><p>And if there are multiple stakeholders involved for a certain update — everyone needs to be equally accountable for a certain update. Emails and messages can be ‘missed out’.</p><h3>The Solution</h3><p>To ensure that no update goes out without authorisation from the Stakeholders, we devised and created a 2FA token process built into each deployment. What this means is that without entering the correct tokens — which are only known to the Stakeholders of the project — the deployment would not go ahead.</p><p>Every project has a unique set of tokens. These tokens should be the same within a single stakeholder group. So if there are 3 people acting as project managers for a certain update — either of them could authenticate the deployment by sharing the token. If there are multiple stakeholder groups for a certain project — each group would have a unique token which would be needed to run the deployment process. For example, a deployment that falls under the purview of both the product as well as the infrastructure team would need tokens from both the stakeholder groups to go ahead.</p><h3>How Does it Work</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FhydL4XC32ux38zC" /><figcaption>QR Code to be scanned by Stakeholders</figcaption></figure><p>Tesselate is used for Infrastructure Deployments and Nomad is used for product changes. We configured Tesselate to ask for a token from the Infrastructure stakeholder before any update could be pushed to production, and we configured Nomad to ask for a token from both Product Stakeholders as well as Infrastructure Stakeholders to push a software update.</p><p>For every new project, the Stakeholder group generates a QR code and shares it amongst themselves. Using a regular 2FA token app, something like a Google Authenticator or Authy, the QR code is scanned and tokens for the project are generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*b98Qc1WG2qZv247RvvnfgA.png" /></figure><p>These tokens are only known to the Stakeholder groups. Whenever a change needs to be deployed, the engineers pushing the update to production need to obtain the token from the stakeholders, enter it and only then can the deployment go ahead. The Stakeholders before handing out the token can then ensure that the hygiene checklist was followed by the engineers on call, thereby reducing the chances of downtime and unforeseen errors on production.</p><h3>Result</h3><p>Downtime was a common phenomenon after a deployment. Something or the other would crash, causing on-call engineers great frenzy. Since the implementation of the 2FA token requirement, the system has had 0 downtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kuKbMSFuZxaBKFTZjV95eQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Rk3R8JM2zahZR2wT0fjPOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*5PpfSqCrYQ0f-QFWeuLXrw.png" /></figure><h3>Projects/Codes:</h3><p>Generate QR Codes via CLI that can be shared within a Stakeholder group.</p><p><a href="https://github.com/tsocial/ts2fa">tsocial/ts2fa</a></p><p>Tessellate Changes:</p><p><a href="https://github.com/tsocial/tessellate/blob/7246f515fa1739f9cc436f96b321f0a8ac359e5e/server/middleware/twofa.go">tsocial/tessellate</a></p><p>Nomad-Proxy</p><p><a href="https://github.com/tsocial/nomad-proxy/blob/b534ee8e79b2d9e9ae67312674682d5404bed17c/client-proxy/main.go">tsocial/nomad-proxy</a></p><p><strong>Sample configurations:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f53469226a125f7c7b8aefa6504f012f/href">https://medium.com/media/f53469226a125f7c7b8aefa6504f012f/href</a></iframe><p>Tessellate Command:</p><pre>./tessellate — totp-config config.json</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73979700a66d9b024d0398319c7fb61c/href">https://medium.com/media/73979700a66d9b024d0398319c7fb61c/href</a></iframe><p>Nomad Command:</p><pre>./nomad_serverproxy — totp-config config.json</pre><p>2FA config is a JSON hash that has the following structure.</p><pre>json<br>{<br>  &quot;Namespace/Ident&quot;: {<br>    &quot;Operation&quot;: [&quot;array&quot;, &quot;of&quot;, &quot;tokens&quot;]<br>  }<br>}</pre><p>Example: If you wish to Protect Nomad Job called “hello-world”:<br>- Allow Get<br>- A new deployment should have two stakeholders’s OTP<br>- Stop a Job could do with just one</p><p>The totp config would look like:</p><pre>json<br>{<br>  &quot;hello-world&quot;: {<br>    &quot;GET&quot;: [],<br>    &quot;POST&quot;: [&quot;GA4DGMQ4TKZMFTDSNBUDEMZYMYYA&quot;, &quot;GA4DGMQ4TKZMdedeqeSNBUDEMZYMYYA&quot;],<br>    &quot;DELETE&quot;: [&quot;GA4DGMQ4TKZMFTDSNBUDEMZYMYYA&quot;]<br>  }<br>}</pre><p>Verification library is independent of Nomad and Tessellate and can be utilized for any purpose which can leverage the same configuration structure.<br>The library is open-source and available at</p><p><a href="https://github.com/tsocial/ts2fa/blob/master/otp/Readme.md">tsocial/ts2fa</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=629e05880ffb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trusting-social-engineering/engineering-processes-629e05880ffb">Engineering Processes</a> was originally published in <a href="https://medium.com/trusting-social-engineering">Site Reliability Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Reliability: From Operations to Engineering]]></title>
            <link>https://medium.com/trusting-social-engineering/sre-intro-116f9b8bac8?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/116f9b8bac8</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Sat, 06 Oct 2018 13:42:07 GMT</pubDate>
            <atom:updated>2018-10-06T13:42:07.742Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*DndFmOhuzcEO3mo4ZvB7pg.png" /></figure><p>Site Reliability Engineering is the new fad. It’s not the Docker that you don’t need, it’s not the Kubernetes that you don’t need. It’s also not the Blockchain that you don’t need. Or well, maybe it <em>is</em>.</p><p>Question on running operations at scale, like the manpower or cost involved. Or what is the cost associated with each 9 in the <a href="https://en.wikipedia.org/wiki/Five_nines">Five Nines</a> are being talked about more often. What does it take to run things smoothly.</p><p>Here are a few pillars, or keys to success, of running things reliably:</p><ol><li>Have no silos of information</li><li>Measure everything</li><li>Culture, not tools</li><li>Accidents happen</li><li>Changes should be gradual</li></ol><p>There are multiple interpretations of these pillars.</p><p>In this series of posts, we’ll talk about how we interpret these guidelines and go about reliable systems at TrustingSocial.com.</p><p>While defining Site Reliability Engineering is probably beyond the scope of this post, we will take the liberty of quoting Google’s definition <em>(</em><a href="https://landing.google.com/sre/"><em>https://landing.google.com/sre/</em></a><em>)</em> of SRE here:</p><p><em>SRE is what you get when you treat operations as if it’s a software problem. Our mission is to protect, provide for, and progress the software and systems behind all of Google’s public services — Google Search, Ads, Gmail, Android, YouTube, and App Engine, to name just a few — with an ever-watchful eye on their availability, latency, performance, and capacity.</em></p><p>A friend, Nishant, usually says this with a lot of pun and sarcasm:</p><blockquote><em>If nothing ever stopped working, was it never supposed to break Or did the reliability team do a wonderful Job?</em></blockquote><p>The current technology and product evolution can seem a little counter-intuitive. Shipping fast and breaking things is the accepted norm. Or that networks are going to fail. And at the same time we’re seeing Site Reliability Engineering picking up great momentum.</p><p><em>Why couldn’t the dev teams just be a little more careful about service?</em></p><p>In hindsight, it’s quite logical actually. Responsibility and focus.</p><p>Let me elaborate. Having a separate team to worry about the uptime and reliability allows the product teams to completely focus on the business logic and feature building. This razor sharp focus grants product teams the ability to release every minute, while the reliability team wants to not even have a minute of downtime. The business as a whole moves really fast but does <em>not </em>break things (well, most of it).</p><p>This responsibility division and ownership also brings a different dynamics into play. Opinions collide which sometimes erupt into arguments, as would between any passionate team. And as long as each of the teams can own the authority and responsibility of being available with a mutually agreed upon commitment of their deliverables, it works out great!</p><p>Most people think its DevOps++, but here’s the difference. While traditionally operations teams run the code for product teams, we are talking about providing product teams with a tooling that allows them to take code to production and run it reliably. A badly written code without UAT wont automagically fix itself, but the reliability toolchain can certainly ping the developer, every time their deployed application fails.</p><p>Don’t give them a fish, give them the tools to fetch a fish, reliably.</p><p>In a way, silos are good. Silos of responsibility and authority, not information. And as long as each of the teams can own the authority and responsibility of running their service with a mutually agreed upon commitment of reliability, it works out great!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4oQgUVU4XKWCzf4-1snlng.png" /><figcaption>Just because people don’t complain, doesn’t mean parachutes always work.</figcaption></figure><p>Speaking of Reliability, these follow up questions come to mind:</p><ul><li>Why should a service be reliable to begin with?</li><li>How do you measure Reliability?</li><li>How <em>Reliable</em> is Reliable?</li></ul><p>These questions are usually met with a raised eyebrow.</p><p>And here I quote my friend, Nishant again:</p><blockquote>The Best code ever is the one that I never wrote.</blockquote><p>There is a lot of wisdom in that statement. The Best code is also probably the most reliable code, because it never breaks.</p><p>History and statistics show that every time there is a change, something breaks. The need of uptime of a service (<em>product</em>, when speaking from the perspective of a business) is and should only be defined by the product’s owner. This is the real essence of a <strong>Service Level Objective (SLO).</strong></p><p>We let the product owner define the uptime for their service and they usually answer: Always, <em>obviously</em>.</p><p><strong>What kind of service should always be running?</strong></p><p>A pacemaker is a service that should have 100% uptime, but you don’t ship updates to it. I did a quick google search to find that out, and here’s what I got.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8HILr71klVFacDxa6d41RA.jpeg" /><figcaption>The Irony</figcaption></figure><p>If 100% uptime is a myth. So, what is an acceptable uptime?</p><p>The only real way to answer this would be through a quick calculation. If your service was down for x hours a day how much would it cost.</p><p>Let’s say you are running a Stock Exchange. The trading hours range from 9 am to 5 pm. You simply can’t afford a downtime during the trading hours, and a few hours before and after. But a few minutes of downtime at night might not affect your business. So, more than the uptime, the window of uptime matters here.</p><p>A sample Service Level objective could read something like:</p><p><em>100% uptime during 8 AM — 6 PM</em></p><p>But again, 100% uptime even during a window of time, forever, is not completely possible. Hence, we introduce the notion of <strong>Error Budget</strong>. The uptime agreed upon keeping the operations, on-call people, testing, and development costs in mind. So if the agreed upon uptime is 99%, that gives us an Error Budget of 1%. This is your window of not being responsible. You could choose to do anything else, clear technical debts, binge on pizzas or simply choose not to react. <em>As long as issues remain under 1%</em>.</p><p>This brings us to the next question: How much reliable is Reliable? And <em>how </em>do you measure Reliability?</p><p>You know how they say, it’s not about being first but being the best, but you also don’t want to come fifth and win a toaster instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*OXneqFBZ8RjvkGRf2yLIkQ.png" /></figure><p>A server that takes forever to respond, but eventually does, is reliable; it’s probably just not very useful. Reliability is not just about doing a thing reliably, but also doing it in an acceptable time frame.</p><p>Measuring Reliability is a painful ask and brings out the topic of Service Level Indicators (SLI).</p><p>Service Level Indicators could be one or many of the following:</p><ul><li><em>Latency</em><br>Was a given request served within the agreed upon time? Given you have no control over the network Infrastructure used by the requester, this is usually the time between a request being received and returned.</li><li><em>Error Rate</em><br>Number of failed requests. Definition of failure, however, is subjective. For an HTTP service this could be all requests with status code 500 and above.</li><li><em>Throughput</em><br>Number of valid requests that can be served within a time frame.</li><li><em>Availability</em><br>Overall uptime of the system within an aggregated window.</li><li><em>Correctness</em><br>A system could be doing all of the above, but the data being served is incorrect. Example a service that computes PI to some obscure places. The value is served within the SLA but the precision is off.</li><li><em>Freshness</em><br>Think of another system that does all of the above, but a cache busting logic algorithm seems to be malfunctioning, so it continues to serve stale data.</li><li><em>Elasticity</em><br>The system behaves perfectly when there are low concurrent requests but breaks apart the moment concurrent requests exceed a certain value.</li></ul><p>Service Level Indicators have two aspects to them: Specification and Implementation.</p><p>Specification is provided by Product teams while Implementation is driven by the Reliability Team.</p><p>A key thing to note is that Indicators cannot be measured absolutely. For example, you cannot have Error rate since Inception. You need an agreed quantum of time in which the metrics are <strong>aggregated</strong>. The choice of aggregation function is unique to the product. Say, one could take an average response time or the mean response time. Google and many other companies prefer percentile.</p><p><em>99% of requests were served within 100ms whereas 98% were served within 10ms.</em></p><p>Also, larger the window, the more complex the storage gets. An aggregate over past two years would require two years worth of data points to be <em>reliably </em>retained. Recursive reliability, anyone?</p><p><strong>Interval</strong> is another important point of consideration. The interval at which these Indicators are captured has to be just low enough. Going too low would cause <strong>Artificial Load</strong> on the service and going too high would miss important system heartbeats. Here’s an example: an I/O wait check interval of 5 minutes would only get the state every 5 minutes, and probably fail to capture any intermittent disk subsystem wait. But, you do not want to query top 5 seconds either, which may result in top suggesting 100% CPU utilization because of top itself, that is busy doing a stat of /proc/&lt;pids&gt;.</p><p>One of the final considerations is that Indicators behave differently under different degree of loads. So one of the agreed upon criterions should be the load or Elasticity of the system. Not all indicators would degrade linearly as load increases. It is important to measure Degradation as well.</p><p>In the upcoming posts, we will cover the technology and Codes (Released under OSS by Trusting Social Engineering) that help us achieve the Indicators and Objectives.</p><p>Here’s a sample form that we ask Product Owners to fill out for services that qualify to be reliable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F1KzT4m0juOoSfFdzsWy6g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=116f9b8bac8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trusting-social-engineering/sre-intro-116f9b8bac8">Scaling Reliability: From Operations to Engineering</a> was originally published in <a href="https://medium.com/trusting-social-engineering">Site Reliability Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang: Singletons]]></title>
            <link>https://medium.com/@meson10/golang-singletons-762181b05d0b?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/762181b05d0b</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Wed, 29 Jun 2016 18:22:38 GMT</pubDate>
            <atom:updated>2016-06-30T04:33:02.411Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<p>I was recently writing an application in Golang which required some Database interaction. The db library I was using had inbuilt Pooling so I didn’t have to bother about connection recycling and reusing, as long as I could initialise a DbPool and continue to call DbPool.new(). Having a module level Singleton object of DbPool would do this trick. However the problem with Singletons is that, in a multi threaded environment, the initialisation must be protected to prevent re-initialisation. I will discuss a few common ways to achieve this, along-with the shortcomings of each approach.</p><h3>Module init()</h3><p>Most common approach I have come across is to define an init() functions in module files. These module level construtors perform operations like DB Pool initialisation or caches. It is guaranteed that this code runs once-and-only-once at startup of your program. Looks good.</p><p>I have two problems with this approach:</p><ol><li>Import Order: The import order is defined by the order in which the files show up in your code, and unless you rename your files obscurely there is no way to control this sequence.</li><li>Implicit Calls: These inits are automatically called at startup and there is no way to invoke it explicitly. This makes it quite a challenge to test such codes. Like if you wanted to test a part of the code which depends on a DB state pre-initialised, you cannot easily mock the connection by seeding that value from within the test suite.</li></ol><h4>Import Order Problem</h4><p>Let’s say you have a directory structure that looks something like this:</p><pre>.<br>├── abc<br>│   ├── one.go<br>│   └── two.go<br>├── main.go<br>└── pack<br>    ├── one.go<br>    └── two.go</pre><p>Where one.go looks has the following init method:</p><pre>func init() {<br>	log.Println(&quot;&lt;package_name&gt; - One&quot;)<br>}</pre><p>And two.go’s init method looks like this:</p><pre>func init() {<br>	log.Println(&quot;&lt;package_name&gt; - Two&quot;)<br>}</pre><p>And your main.go had a very simple code which looks like this:</p><pre>package main</pre><pre>import (<br>	&quot;log&quot;<br>	&quot;./abc&quot;<br>	&quot;./pack&quot;<br>)</pre><pre>func main() {<br>	log.Println(pack.PackOne, pack.PackTwo, abc.AbcOne, abc.AbcTwo)<br>}</pre><p>Output will always be:</p><pre>2016/06/29 21:34:53 Abc - One<br>2016/06/29 21:34:53 Abc - Two<br>2016/06/29 21:34:53 Pack - One<br>2016/06/29 21:34:53 Pack - Two<br>2016/06/29 21:34:53 hello world<br>2016/06/29 21:34:53 1 2 1 2</pre><p>Since package abc appears ahead of package pack (alphabetically), and both of them are included in the main, there is no way you can alter the init order without renaming the package to something else.<br>Also, If pack.PackOne had to be seeded with a mock value while testing, it cannot be done because there is no way of invoking the init method explicitly. And while testing, Database connectors is something that you more-often-than-not have to mock.</p><h3>sync.RWMutex</h3><p>Alternate approach to do this is to use a Module level cache variable with embedded Read-Write Mutex to ensure synchronisation across multiple go-routines.</p><p>An explicit method can then be used to acquire a ReadWrite Lock to check and return if the value had already been initialised, or initialise it with a value and return that otherwise.</p><p>A sample code for such an approach would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4404a603e3a717b6032137b50231b0ae/href">https://medium.com/media/4404a603e3a717b6032137b50231b0ae/href</a></iframe><p>On carefully examining the output of this code you will observe a problem that the code tries to attain locks even after the first initialisation is complete.</p><pre>2016/06/29 21:22:57 lock 5<br>2016/06/29 21:22:57 Initializing GetInt<br>2016/06/29 21:22:57 lock freed 5<br>2016/06/29 21:22:57 &amp;{1}<br>2016/06/29 21:22:57 lock 3<br>2016/06/29 21:22:57 lock freed 3<br>2016/06/29 21:22:57 &amp;{1}<br>2016/06/29 21:22:57 lock 1<br>2016/06/29 21:22:57 lock freed 1<br>2016/06/29 21:22:57 &amp;{1}<br>2016/06/29 21:22:57 lock 2<br>2016/06/29 21:22:57 lock freed 2<br>2016/06/29 21:22:57 &amp;{1}<br>2016/06/29 21:22:57 lock 0<br>2016/06/29 21:22:57 lock freed 0<br>2016/06/29 21:22:57 &amp;{1}<br>2016/06/29 21:22:57 lock 4<br>2016/06/29 21:22:57 lock freed 4<br>2016/06/29 21:22:57 &amp;{1}</pre><p>After 5 was initialised; 1, 2, 3, and 4 should have been free to run in Parallel. Since the access to the cached value is bound by a ReadWrite Lock and only one goroutine would have that at a time, they pretty much execute in a sequence.</p><p>There should be a way to better to tackle this.</p><h3>sync.Once</h3><p>By Definition: Singleton is a design pattern that restricts the instantiation to one object. It would be lot more efficient if there was a way to lock JUST the first initialisation. Thereafter, any piece of code should be free to access the value without having to bother aout Locking and inevitably Blocking other resources.</p><p>sync.Once allows you to do exactly that, where Once is an object that will perform exactly one action.<br>You can read more about the documetation here: <a href="https://golang.org/pkg/sync/#Once">https://golang.org/pkg/sync/#Once</a></p><p>The same code, as demonstrated in the last method, when moved to sync.Once pattern will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91a425fda086fc646923d157dd0fa57a/href">https://medium.com/media/91a425fda086fc646923d157dd0fa57a/href</a></iframe><p>While the output of this code will be:</p><pre>2016/06/29 21:26:42 No lock 5<br>2016/06/29 21:26:42 No lock 0<br>2016/06/29 21:26:42 No lock 2<br>2016/06/29 21:26:42 No lock 4<br>2016/06/29 21:26:42 No lock 3<br>2016/06/29 21:26:42 Initializing GetInt<br>2016/06/29 21:26:42 No lock return 5<br>2016/06/29 21:26:42 &amp;{1}<br>2016/06/29 21:26:42 No lock return 0<br>2016/06/29 21:26:42 &amp;{1}<br>2016/06/29 21:26:42 No lock return 2<br>2016/06/29 21:26:42 No lock 1<br>2016/06/29 21:26:42 No lock return 1<br>2016/06/29 21:26:42 &amp;{1}<br>2016/06/29 21:26:42 No lock return 4<br>2016/06/29 21:26:42 &amp;{1}<br>2016/06/29 21:26:42 &amp;{1}<br>2016/06/29 21:26:42 No lock return 3<br>2016/06/29 21:26:42 &amp;{1}</pre><p>Do observe that after the first initialisation of 5, goroutines do not block each other and pretty much run at random. This by-passes the locking and still provide you the flexibility of being able to invoke it explicitly. The only down side of this method is that you would need a separate Once object for each such cached variable in your code. It also requires a promise that the value is not going to change through the lifecycle of the code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=762181b05d0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform, null_resources and Azure ASM]]></title>
            <link>https://medium.com/@meson10/terraform-and-azure-api-cheats-cdba54cc7035?source=rss-fe36fbe48c68------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdba54cc7035</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[ssl-certificate]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Piyush Verma]]></dc:creator>
            <pubDate>Fri, 17 Jun 2016 14:25:21 GMT</pubDate>
            <atom:updated>2016-06-30T18:33:50.869Z</atom:updated>
            <content:encoded><![CDATA[<h4>Preface</h4><p>Recently, I have been trying to bring up virtual machines in Microsoft Azure but ran into this interesting &amp; annoying problem of not being able to upload SSH keys via the terraform DSL. There is a provision to provide a <strong>ssh_key_thumbprint</strong> but sadly no way to upload what you would call a KeyPair in AWS jargon.</p><p>While terraform does not support this operation via its DSL, It is possible to achieve this using some less-explored features of terraform.</p><h4>Solution</h4><p>I am using OS X, so my code samples might include some OS X specific commands. However it should be fairly easy to carry out these operations on other operating systems too.</p><p>First, the azure cli must be installed. Easiest way to do that is using brew:<br><strong>$: brew install azure-cli</strong></p><p>Post installation you will have to authenticate the azure cli. But that’s fairly easy. All you have to do is <strong>$: azure login</strong> and subsequent instructions on the screen will handhold you through the process.</p><p>Next, generate a SSL certificate that meets the following requirements:</p><ul><li>The certificate must contain a private key.</li><li>The certificate must be created for key exchange, exportable to a Personal Information Exchange (.pfx) file.</li><li>The certificate must use a minimum of 2048-bit encryption.</li></ul><p>A SSH keypair requires to be associated with an azure service. So you can create a service.json with the following contents:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bec5cb15b0ce98816ce1c2825cebfc3/href">https://medium.com/media/5bec5cb15b0ce98816ce1c2825cebfc3/href</a></iframe><p>Here’s how you can generate a certificate, a .pfx file and upload it to Azure portal.</p><pre>$service=&quot;domain-gamma&quot;</pre><pre>openssl req -x509 \<br>  -key $service-deployer.key \<br>  -nodes \<br>  -days 1365 -newkey rsa:2048 \<br>  -out /tmp/$service-deployer.pem \<br>  -subj ‘/CN=domain.com/O=Domain Inc./C=US’</pre><pre>openssl x509 \<br>  -outform der \<br>  -in /tmp/$service-deployer.pem \<br>  -out /tmp/$service-deployer.pfx</pre><pre>azure service cert create $service /tmp/$service-deployer.pfx</pre><p>Azure API also provides a way to fetch the list of all certificates uploaded and attached to it’s services.</p><pre>piyush:azure master λ azure service cert list</pre><pre>info: Executing command service cert list<br>+ Getting cloud services<br>+ Getting cloud service certificates</pre><pre>data: Service Name Thumbprint Algorithm<br>data: domain-gamma 4F2AUA9ADF39830CDEHAJAND553DEANAJNAD8C8F sha1</pre><pre>info: service cert list command OK</pre><p>The recently uploaded certificate has started showing up with a corresponding thumbprint, that can be used to provision new Azure machines.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd558e0d8195b186b4846a66a83acc4f/href">https://medium.com/media/cd558e0d8195b186b4846a66a83acc4f/href</a></iframe><h4>Catch</h4><p>So while the above example works well, it does not yet have an automatic essence to it. I am still responsible for the grunt work of checking if the certificate has been uploaded and if not, create one key pair, upload the .pfx and then save the thumbnail corresponding to that service, and all of this before running the terraform plan. Thing can be definitely be done better.</p><h4>Explanation</h4><p>You mainly have to observe these four things in the above example:</p><ul><li>depends_on</li><li>null_resource.ssh_key</li><li>ssh_key_thumbprint: ${file(“./ssl/ssh_thumbprint”)}</li><li>ssl/cert.sh</li></ul><h4>depends_on</h4><p>While most dependencies in Terraform are implicit; i.e Terraform is able to infer dependencies based on usage of attributes of other resources, Sometime you need to specify explicit dependencies. You can do this with the <strong>depends_on</strong> parameter which is available on any resource.</p><p>I recommend reading more about Terraform dependencies <a href="https://www.terraform.io/intro/getting-started/dependencies.html">here</a>.</p><p>By injecting a depends_on we can defer the responsibility of assurance of a thumbprint to another resource, but that should be done before an Instance is created.</p><p><strong>Note (FAQ)</strong>: Using a local-exec provisioner approach will <strong>not </strong>work here, because local-exec is done AFTER the resource has been created and not before. Also local-exec provisioner on any previous operation doesn’t guarantee re-run if the resource itself does not change.</p><p>Read on, for the solution.</p><h4>null_resource</h4><p>The null_resource is a resource that allows you to configure provisioners that are not directly associated with a single existing resource.</p><p>null_resource is like a dummy stub that you can use to insert a node that encapsulates provisioners between two existing stages of the graph. The position is determined by refering to this resource via a depends_on from the child resource. In this case, null_resource will be called from the azure_instance resource.</p><p>You can read more about terraform’s null_resource <a href="https://www.terraform.io/docs/provisioners/null_resource.html">here</a>.</p><p>Say we delegate all the duties to a standalone Bash script, we can invoke the script as a local-exec provisioner from the null_resource.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc9759d472a5f1c3d70a939b54c69386/href">https://medium.com/media/cc9759d472a5f1c3d70a939b54c69386/href</a></iframe><h4>triggers</h4><p>But what if someone deletes the ssh_thumbprint file? Every subsequent terraform run would panic and crash. Solution lies in triggers attribute of a null_resource. triggers is amapping of values which should trigger a rerun of this set of provisioners. Values are meant to be interpolated references to variables or attributes of other resources.</p><p>In this case it’s a file that is being read from the filesystem. So any changes forces the resource to be re-trigerred eventually forcing a re-converge on the instances that depend on this null_resource.</p><h4>ssl/cert.sh</h4><p>Putting together the bash script, which accepts the service name and tries to locate an existing uploaded certificate for that service. If not, it generates a new .pfx using the above mentioned techniques, fetches the ssh_key_thumbprint and saves it to a common file where terraform instance resource can read it from.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4930e8d180588f17df09ed871310860/href">https://medium.com/media/b4930e8d180588f17df09ed871310860/href</a></iframe><p>Now, you should be able to provision a SSH only VM and use the generated .pem file to login to your freshly created Virtual Machine. Yay!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdba54cc7035" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>