<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bruce Dominguez on Medium]]></title>
        <description><![CDATA[Stories by Bruce Dominguez on Medium]]></description>
        <link>https://medium.com/@bruce_25864?source=rss-d5553ac6322e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sMMVbH8vZ-kl6NLM1BT1HA.jpeg</url>
            <title>Stories by Bruce Dominguez on Medium</title>
            <link>https://medium.com/@bruce_25864?source=rss-d5553ac6322e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:38:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bruce_25864/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The ROAD to SRE]]></title>
            <link>https://medium.com/@bruce_25864/the-road-to-sre-ad4c73df78b8?source=rss-d5553ac6322e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad4c73df78b8</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Bruce Dominguez]]></dc:creator>
            <pubDate>Mon, 20 Dec 2021 12:11:24 GMT</pubDate>
            <atom:updated>2021-12-20T12:11:24.019Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nxc1QhBlNrcezh3P" /><figcaption>Photo by <a href="https://unsplash.com/@jerrykavan?utm_source=medium&amp;utm_medium=referral">Jaromír Kavan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There are many ways to introduce Site Reliability Engineering practices to your organisation, but it can be confusing where you should start. Do you start with introducing <a href="https://sre.google/sre-book/service-level-objectives/">Service Level Objectives</a> (SLO’s) into your organisation so that you can measure what matters most to your customers? But you need observability (o11y) right? This gives us visibility into our code, highlighting bottlenecks, errors and bad designs as well as allowing us to build Service Level Indicators (SLI’s) for our SLO’s. But what happens if you do have a customer facing incident, surely you need a tight incident response process so that we can recover or remediate quickly? Or do you ensure you can deploy both code and infra repeatedly and rapidly? The answer is all of these! But what do you work on first? And how do we know what areas need to level up?</p><p>I work in a well established SRE team for a cloud native company where our goal is to help improve product reliability and stability. We achieve this in a number of ways by providing coaching and guidance to our engineering team to improve service observability, infrastructure automation and architecture. This year we began to plan out what the landscape of the SRE practice could look like in the future and it soon became clear that we had difficulty clearly articulating what the guiding principles of Site Reliability Engineering and what it meant for our team. Without a clear tangible definition of Site Reliability Engineering we could be in danger of turning into another ops or platform team.</p><p>What we needed was a set of clearly defined principles that would guide us in defining key competencies. Now, there is no shortage of resources out there, you only have to google SRE and you get 43,000,000 results. Yikes! In the past we leveraged the many videos and blog posts from google. However we found quickly that not everything could be applied the same way. We were not Google’s scale, nor did we have that kind of workforce resource available as we run 1 (sometimes 2) SRE’s per Business Unit. So we set out to define what “SRE means to us” and define our core principles of our SRE practice. In this article I will discuss the 4 key principles that we use to guide what define as SRE in terms of our organisation and provides clarity on areas we need to focus on. In subsequent articles we will break down each of our principles into competencies that form our “SRE skill tree”.</p><h3><strong>Defining SRE</strong></h3><p>First we defined Site Reliability Engineering, as <em>“a discipline that incorporates aspects of software engineering and applies them to infrastructure and operations problems to create scalable and highly reliable software systems”</em>. This sounds great, but it actually doesn’t describe what the SRE team does. To address this gap, we defined 4 key principles that would form the focus of how we would build out our SRE practice. The principles are broken down into the following areas of <strong>R</strong>esponse, <strong>O</strong>bservability, <strong>A</strong>vailability and <strong>D</strong>elivery. And when stacked together, form the <strong><em>ROAD</em></strong><em> </em>to SRE (pun intended).</p><p><strong>Principle 1: Response — </strong><em>The effective response to failure events through incident processes, runbooks and disaster recovery events.</em></p><p>Why is this important? Well, services and infrastructure can fail at any time, and often during the early hours of the morning, so we need a robust response process for handling an incident. And when they do occur, we should have an effective remediation process so that we continue to learn and avoid repeating the same mistakes. This principle is broken down into the following 3 subsections:</p><ul><li><strong>Respond: </strong>Improving how we handle an incident when one occurs.</li><li><strong>Measure: </strong>Tracking key metrics like MTTD (mean time to detection), MTTA (mean time to acknowledge), MTTR (mean time to resolution) and MTBF (mean time between failure) to identify improvement opportunities in our process.</li><li><strong>Remediate: </strong>Improving how quickly we can fix issues when they arise with clear runbooks or automation or better yet, prevent them in the first place. This also includes blameless post mortems on incidents to apply learnings gained.</li></ul><p><strong>Principle 2: Observability — </strong><em>The ability to understand the health of services and surface events through telemetry that impact the availability and performance of product services.</em></p><p>I probably do not need to preach why good observability (o11y) is important, but without observability you are flying blind during an incident, at best you can only make educated guesses based on inferred knowledge and domain experience. Who wants that stress at 3 am in the morning?</p><p>Instrumenting observability into your services is critical and gives you the ability to ask arbitrary questions of service and pinpoint performance problems much quicker. Being able to quickly find a root cause of an error makes for a much happier development and SRE team. This principle is broken down into the following 3 subsections:</p><ul><li><strong>Instrument</strong> : Implement logs, traces, metrics and deployment markers.</li><li><strong>Monitor: </strong>Ensuring that we monitor not only what we instrument but also ensuring we have synthetics, and ideally that can test our SLI/SLO (more later on this).</li><li><strong>Alert: </strong>Ensure that all alerts are actionable and ideally linked to an <a href="https://sre.google/sre-book/service-level-objectives/">SLO.</a></li></ul><p><strong>Principle 3: Availability &amp; Reliability — </strong><em>Product services are both available (able to fulfil its intended function) and Reliable (confidently service requests) as governed by the Service Level Objective (SLO) and measured by the Service Level Indicator (SLI).</em></p><p>Understanding how your services handle failure (gracefully or not) and understanding it’s dependencies helps inform the availability of your services. This can guide how many <a href="https://nobl9.com/resources/do-you-really-need-five-nines/">9’s</a> your SLO can promise to the customers or if we need to halt feature work and focus on making our services more resilient. This principle is broken down into the following 3 subsections:</p><ul><li><strong>Service Level Objectives</strong> : Defining meaningful Service Level Indicators that measures availability and reliability from our customer’s perspective with appropriate Service Level Objectives. For more information on what SLI’s and SLO’s are you can check out this great blog from <a href="https://cloud.google.com/blog/products/devops-sre/sre-fundamentals-sli-vs-slo-vs-sla">google</a>.</li><li><strong>Performant: </strong>Understanding our system bottleneck and capacity to better support Service Level Objectives.</li><li><strong>Failure Management: </strong>Identifying and testing the failure scenarios of our workloads to improve resiliency.</li></ul><p><strong>Principle 4: Delivery — </strong><em>Consistent provisioning and deployment of product services and operational dependencies such as Infrastructure as Code (IAC), Continuous Integration &amp; Delivery (CI/CD) and Requests for Change (RFC’s).</em></p><p>The Build, Provision and Deployment of our services are repeatable, not only as an outcome of an automated build pipeline, but also that they are <a href="https://reproducible-builds.org/">reproducible</a>, meaning that we can verify the path from code to binary is clean from any malicious code. We also want to enable the Product teams to deliver faster by progressively delivering features to production using techniques such as blue/green and canary deployments. This principle is concerned with providing the capability to go fast, deploy rapidly but ensure it is repeatable and secure. Again, this principle is broken down into the following 3 subsections:</p><ul><li><strong>Build: </strong>Application artefacts are built and tested in an automated and secure build pipeline.</li><li><strong>Provision: </strong>All infrastructure and configuration is defined as code for consistent and repeatable provisioning of cloud infrastructure.</li><li><strong>Deploy: </strong>All services deployments are automated and use progressive deployment strategies like canary and blue/green deployments.</li></ul><h3><strong>Conclusion</strong></h3><p>By establishing a set of core principles (<strong>R</strong>esponse, <strong>O</strong>bservability, <strong>A</strong>vailability and <strong>D</strong>elivery) aka our<em> “ROAD to SRE”</em>, we now have clarity on what areas we expect our SRE team should be focusing on and avoiding a common pitfall of becoming another platform or Ops team. We also now have a yardstick that we can use to measure progress, highlighting new capabilities and identifying the areas we are lacking. In the upcoming articles we will begin to flesh out how we break down each subcategory of our principles into competencies, in the form of a skill tree, and map them to the 5 phases of <a href="https://cloud.google.com/blog/products/devops-sre/the-five-phases-of-organizational-reliability">organisational reliability</a> of Reactive, Proactive, Strategic and Visionary.</p><h3><strong>Acknowledgements</strong></h3><p>Co-creator Marc Armstrong and reviewed by Lee Campbell and Rhys Campbell.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad4c73df78b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The SRE Incident Response game]]></title>
            <link>https://medium.com/@bruce_25864/the-sre-incident-response-game-db242fff391c?source=rss-d5553ac6322e------2</link>
            <guid isPermaLink="false">https://medium.com/p/db242fff391c</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Bruce Dominguez]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 01:56:05 GMT</pubDate>
            <atom:updated>2020-06-16T01:56:05.113Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BdYo7vOSQAHJc-Z2" /><figcaption>Photo by <a href="https://unsplash.com/@clintbustrillos?utm_source=medium&amp;utm_medium=referral">Clint Bustrillos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Shall we play a game?…</h3><p>A sustainable on-call duty is critical for any Incident response, providing the human filter for the reliability and availability of services that are under the watchful care of the engineering team. The key word is sustainable. Burnout from sleep peppered with PagerDuty alerts, complex and adrenaline pumping incidents can take its toll. Establishing an on-call rotation where an engineer feels underprepared, stressed and likely ready to quit is something we want to avoid, not only because it’s harmful to your engineers but also to maintaining the availability of your services. So let’s set the scene.</p><p>It’s 3am, you and your family are fast asleep. Then BAM! You are woken by a PagerDuty alarm, ripping you from your dreams into what could be potentially a major incident. Your adrenaline is spiking, yours eyes are still focusing and you haven’t even had a cup of coffee yet. Are you ready to jump in and triage the page? Do you know where to start? Have you seen this error before? Are you even familiar with that particular service?</p><p>So how do we become comfortable with the uncomfortable? This takes practice and experience responding to incidents until it becomes muscle memory. However repetition alone isn’t going to build this capacity or improve the teams performance for responding to incidents. K. Anders Ericsson, a leading psychologist in expert performance, describes elite performers practice differently from everyone else. They engage in deliberate practice which has specific goals with a defined focus. Critical to this practice is immediate feedback, allowing for lessons to be learnt from mistakes which are made in a safe environment.</p><p>A great way to direct this practice and cultivate the learning culture is by running regular disaster role playing games. The RPG game created by SRE’s, often called “Wheel of Misfortune’’ or “Walk the plank”, is an exercise that allows junior SRE’s or on-call engineers to walk through an incident and their response to the incident in a safe environment with feedback and hopefully with some fun thrown in. There are many ways you can run these games, check out these resources if you want to know more:</p><p><a href="https://landing.google.com/sre/sre-book/chapters/accelerating-sre-on-call/">https://landing.google.com/sre/sre-book/chapters/accelerating-sre-on-call</a></p><p><a href="https://cloud.google.com/blog/products/management-tools/shrinking-the-time-to-mitigate-production-incidents">Using SRE and disaster recovery testing principles in production | Google Cloud Blog</a></p><p>I am going to go through the variation we use to up skill our on-call engineers we called “The Kobayashi Maru”, the name we borrowed from the Star Trek training exercise to test the character of Starfleet cadets.</p><h3><strong>What do you need:</strong></h3><h4><strong>The Players</strong></h4><p><strong>Games Master</strong> — You are responsible for guiding the scenario, this includes triggering the incident in PagerDuty, describing the scenario and providing context, as well responding to questions from the on-call engineers.</p><p><strong>Incident commander</strong> — You are the escalation point for the incident. Your directive is to:</p><ul><li>Stem the bleeding</li><li>Communicate to stakeholders</li><li>Resolve the Major incident</li></ul><p><strong>Primary On-Call</strong> — You are the primary point for all alerts. Your directive is to:</p><ul><li>Triage the incident</li><li>Fix the incident if possible</li><li>Escalate if to a Incident Commander</li></ul><p><strong>Secondary On-Cal</strong>l — You are learning the ropes of being on-call. Your directive is to:</p><ul><li>Assist the Primary On Call to resolve the issue</li><li>Aid in collecting information for future forensics</li></ul><p>The primary and secondary on-call players will choose 3 Action cards and 2 Special ability cards. The incident commander also receives 3 Action cards and the Games master receives 5 Play Now cards.</p><p>Each player must have their phone on them with PagerDuty to receive pages.</p><h3><strong>The Cards</strong></h3><p><strong>Action Cards:</strong></p><p>These are used by the player to perform a response to the incident, with each one having a buff (+1) and a hit (-1) depending on your special ability. An example is “check the logs” action. A player can play this card, but if they don’t have the “Runbook” special ability card then they take a 30 second hit while they dig into the logs. Here are a few examples of action cards we use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pbh4lunbHzjmA38jnAxhVg.png" /></figure><p>The incident commander also has action cards for them to play, in the event that the incident is escalated. Here are some examples we use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftTHZFBvF9eoHtFJjP4CrQ.png" /></figure><p><strong>Special Ability Cards:</strong></p><p>These cards are used as a bonus ability for the player, giving them abilities like the “Runbook” card. This provides the player with rapid recovery paths or even the “IAC” card which allows the player to deploy infrastructure using code rather than ClickOps. Here are some examples that we use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3usPjOBLEoIByhOIakNU4g.png" /></figure><p><strong>Play Now Cards:</strong></p><p>These cards are used by the Games master that can be played anytime to help move the incident along or add some heat to the incident. Some examples are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etRge2hn2aGI0Gv-gQN3eQ.png" /></figure><h3><strong>The goal</strong></h3><p>With a countdown of 5 mins you have to:</p><ul><li>Triage the incident using only your action cards and special ability cards.</li><li>You may escalate to a Major Incident via the Incident Commander (if required)</li><li>Resolve the incident.</li><li>Go back to bed.</li></ul><p>The Primary and secondary on-call players take it in turns to respond to the incident using their action cards, and escalate as a major incident to the Incident Commander if required to do so. If the timer runs out, this indicates your shift has finished and you must hand over to the next team and the clock restarts.</p><p><strong>Starting the game:</strong></p><p>To kick off the game you will need to do some prep beforehand, ideally grabbing some past incidents with a mixture of major incidents and noisy alerts to keep it as realistic as possible. We use PagerDuty and preload the incidents with the information so that it is ready to push out during the game. Once you’re ready to go and have assigned all of the roles and everyone understands their responsibilities, you can start the game.</p><p>Trigger the first incident in PagerDuty, assigning it to the person with the Primary on-call role and start the timer when they have acknowledged the page. To add a little bit of pressure, we have a 5 minute timer counting down that is visible for all players during the game.</p><p>The Primary on-call will review the page and their action cards to determine what they can do, with the Games master providing some context to the alert, such as “a deployment of X services took place 5 minutes ago” or “Multiple synthetics are also failing for service Y”. The primary and secondary take turns and attempt to identify the issue, triage and resolve while talking through decisions being. At any time the Games master may use their “Play Now” card to spice up the game or force a change in direction.</p><p><strong>The game postmortem</strong></p><p>At the end of each game we run a mini postmortem to identify what went well, what didn’t go so well and what can we learn that can be carried over to future rounds. It’s also important to get immediate feedback on the teams performance so actions and improvements can be made.</p><p><strong>Don’t forget to have fun!</strong></p><p>There are lots of ways you can run a Game session and put your flavour on it, but a few things to keep in mind that help us.</p><ol><li>Have fun — You will get more engagement if you can keep some fun and humour in the games.</li><li>Encourage a culture of learning and feedback — Any decisions you made in the game are the right ones for the context of the situations. Use the mini-postmortem at the end to provide some immediate guidance for improvements and also celebrate the team’s successes!</li><li>Keep a regular cadence of sessions — The more times you run these sessions the more practice your team gets, and more comfortable they will become with the processes and practices of being on-call.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db242fff391c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set up Vault with Vagrant and Ansible]]></title>
            <link>https://medium.com/@bruce_25864/set-up-vault-with-vagrant-and-ansible-c90cefe77381?source=rss-d5553ac6322e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c90cefe77381</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[vagrant]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <dc:creator><![CDATA[Bruce Dominguez]]></dc:creator>
            <pubDate>Fri, 19 Apr 2019 11:42:41 GMT</pubDate>
            <atom:updated>2019-04-19T11:49:52.884Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8HXHm3H9MIiQocMJ" /><figcaption>Photo by <a href="https://unsplash.com/@artemkovalev?utm_source=medium&amp;utm_medium=referral">Artem Kovalev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the DevOps space, we take for granted that we can now easily scale our infrastructure outside of the boundaries of our safe and secured data centers, spin up as many instances that we could possibly want, and all with a touch of a button (or maybe a few keystrokes). But with great power comes great responsibility (thanks uncle Ben!). Are we managing our secrets and sensitive data in a way that we can tightly control access and audit who has accessed it? We have all been guilty using the one SSH key on multiple machines or even saving that long string token/password in a text file on your local machine to copy paste for later use. But what if your Azure or AWS credentials got accidentally commit in plain text to your GitHub repo and now become a free service for running bitcoins? How do we reduce the blast radius of the impact of such an incident?</p><p><a href="https://www.hashicorp.com/products/vault/">HashiCorp Vault</a> is becoming a very popular open source secret management solution that will securely store your secrets (password, database, cloud access keys and any other key value you want) and can be accessed once authenticated, an example of this is providing dynamic AWS credentials or even one-time SSH passwords. Using dynamic ephemeral credentials reduces the risk of credentials that are logged to disk or otherwise exposed because they will be rotated automatically or have a short time to live. Vault also offers encryption as a service which fortifies your data at rest and in transit, and PKI as a service allowing you to dynamically generate signed TLS certificates on demand. Access to the data in Vault is tightly controlled, with many authentication methods available such as LDAP, GitHub, Username/Password, token etc.</p><p>This guide will help you get up and running with a local Dev environment using Vagrant and Ansible and we will get our hands dirty with an exercise to create an SSH one-time password issued by Vault to log into a server.</p><p>What are we going to build:</p><ol><li>A Vault server — in Dev mode</li><li>A Consul server — This will be our backend storage and works natively with Vault.</li><li>Client — Ubuntu server</li><li>Server — Ubuntu server, we will use this to demonstrate Vault provisioning a one-time Password.</li></ol><p>What you are going to need.</p><ol><li>Vagrant installed — Get it <a href="https://www.vagrantup.com/docs/installation/">here</a></li><li>Virtualbox installed — Get it <a href="https://www.virtualbox.org/wiki/Downloads">here</a></li><li>Ansible — if you are using Mac you can run `brew install ansible` if you have homebrew installed.</li><li>Clone of this <a href="http://git@github.com:brucedominguez/article-vault-vagrant.git">Git Repo</a></li></ol><p>Let’s take a look at what we have in the cloned directory.</p><h3><strong>Vagrant file</strong></h3><p>Our Vagrant file declares what infrastructure we want to provision. We are going use ansible to perform the installation and configuration of Vault and Consul on our “Vault server” and set up our “client” server and “OTP” server.</p><p>The servers we are going to provision are:</p><p><strong>Vault/Consul server</strong> — IP address 192.168.50.100 and using ansible to provision it using the install.yml file under the vault directory.</p><p><strong>Client</strong> — IP address of 192.1268.50.101 and using the install.yml file under the client directory.</p><p><strong>OTP server </strong>— IP address of 192.1268.50.102 and using the install.yml file under the OTP directory.</p><h3><strong>The Ansible files</strong></h3><p><strong>The Vault/Consul Server (install.yml)</strong></p><p>The install.yml file used by ansible will:</p><ul><li>Create a Vault User</li><li>Download Consul and configure it as a service in -dev mode</li><li>Download Vault and configure it as a service in -dev mode using Consul as it’s backend.</li><li>Unseal Vault and save the root token locally under the /rootKey folder</li></ul><p><strong>The Client (install.yml)</strong></p><p>The install.yml file is used by ansible to download Vault and use it to access the Vault Server</p><p><strong>The SSH-OTP Server (install.yml)</strong></p><p>This install.yml file is used by ansible to download and configure the Vault SSH Helper to allow the server to consume the One-Time Password created by Vault.</p><p>Now let’s get our environment up and running. To do this run vagrant up and provision your 3 servers.</p><p><strong>Note</strong> if you have issues with ansible creating files to store the vault keys then you can run sudo vagrant up.</p><pre>Bringing machine ‘vault’ up with ‘virtualbox’ provider…<br>Bringing machine ‘client’ up with ‘virtualbox’ provider…<br>Bringing machine ‘otp’ up with ‘virtualbox’ provider..<br>.....</pre><pre>TASK [Download Consul] *********************************************************<br>changed: [vault]</pre><pre>TASK [Unzip Consul] ************************************************************</pre><pre>changed: [vault]</pre><pre>TASK [Make Consul a service] ***************************************************<br>changed: [vault]</pre><pre>TASK [Ensure config directory exists] ******************************************<br>changed: [vault]</pre><pre>........</pre><pre>TASK [Write unseal keys to files] **********************************************<br>changed: [vault -&gt; localhost] =&gt; (item=[0, ‘3f6299696c198070b0c0d8fd935705f0e53a4bd95a587a55705abc0846b405b32f’])<br>changed: [vault -&gt; localhost] =&gt; (item=[1, ‘e85f00e1103cc2c0aa309aa08d6e139bca7baf44bf2842df489d4dbc40f11a1c83’])<br>changed: [vault -&gt; localhost] =&gt; (item=[2, ‘d23f09eaee91da83ed5491743f7113c1f0954e0dbe23a582f9601cb2ea7f394df6’])<br>changed: [vault -&gt; localhost] =&gt; (item=[3, ‘0ace41f5005a7d7a49371400c8cff264b4e26ab50ba04433346558e99102023d6b’])<br>changed: [vault -&gt; localhost] =&gt; (item=[4, ‘88ba279b850996ac6398878d48c86f690a7afce625f20110e873153b6cec1f55d5’])</pre><p>Now that our infrastructure is up and running we can SSH into our Vault server by running vagrant ssh vault (or if you need to use sudo, then sudo vagrant ssh vault).</p><p>Set your environment variable so the server knows where to find Vault.</p><pre>export VAULT_ADDR=’<a href="http://127.0.0.1:8200/">http://127.0.0.1:8200</a>&#39;</pre><p>Login into Vault with your token found in /vault/rootKey/rootKey.txt(in production you would not use the root key!)</p><pre>Vault login s.a0fJ1fYMwvFGbFaSoqf3P6E0 (remember to use your token)</pre><pre>Key Value<br>— — — — -<br>token s.a0fJ1fYMwvFGbFaSoqf3P6E0<br>token_accessor JR3OMCNehciCWp74VDDXdjwl<br>token_duration ∞<br>token_renewable false<br>token_policies [“root”]<br>identity_policies []<br>policies [“root”]</pre><p>And now you&#39;re in, so let’s take Vault for a test drive.</p><h3><strong>One Time SSH Password</strong></h3><p>A great use case for Vault is the One-Time SSH Password (OTP) SSH secrets engine that will allow Vault to issue a One-Time Password every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification. This solves a few problems, ensuring that we are restricting access for anyone to SSH into our servers and make config changes (keeping with the DevOps immutable ethos), and limiting the “one pem key to rule them all” scenario. The authentication flow will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*-COmyT8GRovzpygr" /></figure><ol><li>From the “client” server we are authenticating to vault with a token and request an SSH one-time password.</li><li>Vault will check our access and then provide us password to use to ssh to our “otp” server.</li><li>From the “client” server we will ssh into the server with our one-time password.</li><li>The Vault SSH agent will check with vault to see if the password is valid and allow us to use it. Once used it can not be used again.</li></ol><p>First, we need to set up the SSH backend on the Vault server (logged in to the vault server)</p><pre>vault secrets enable ssh</pre><p>Now create an<strong> </strong>otp role with a key type<strong> </strong>as OTP and the default ssh username as vagrant.</p><pre>vault write ssh/roles/otp_key_role key_type=otp default_user=vagrant cidr_list=0.0.0.0/0</pre><p>We now need to set up a method for our “client” to authenticate to Vault. We are going to use the User/Password method.</p><pre>vault auth enable userpass</pre><p>Then we create a policy to allow for a one-time password :</p><pre>vault policy write admin -&lt;&lt;EOF<br>path “sys/mounts” {<br>capabilities = [“list”,”read”]<br>}<br>path “ssh/*” {<br>capabilities = [“read”,”list”,”create”,”update”]<br>}<br>path “ssh/creds/otp_key_role” {<br>capabilities = [ “update” ]<br>}<br>EOF</pre><p>Time to create a new user that client server will log in as. We are going to create “admin” as the user who will use the OTP.</p><pre>vault write auth/userpass/users/vault password=vault policies=admin</pre><p>Now open up a new terminal window and go to the cloned directory and turn vagrant ssh client or sudo vagrant ssh client. Login as the new user you created.</p><pre>vault login -method=userpass username=vault password=vault</pre><p>Request a one-time password from Vault with the “otp” server IP address.</p><pre>vault write ssh/creds/otp_key_role ip=192.168.50.102</pre><pre>Key Value<br>— — — — -<br>lease_id ssh/creds/otp_key_role/u9AuTq4m5i4hcu6fL5XqhOQI<br>lease_duration 768h<br>lease_renewable false<br>ip 192.168.50.102<br>key 0f79a06c-3458–53c3–1282–0be992f3560b<br>key_type otp<br>port 22<br>username vagrant</pre><p>Now issue the following command</p><pre>ssh vagrant@192.168.50.102</pre><pre>The authenticity of host ‘192.168.50.102 (192.168.50.102)’ can’t be established.<br>ECDSA key fingerprint is SHA256:G1J5ek/T6KuCCT7Xp2IN1LUslRt24mhmhKUo/kWWVrs.<br>Are you sure you want to continue connecting (yes/no)? yes<br>Warning: Permanently added ‘192.168.50.102’ (ECDSA) to the list of known hosts.<br>Password:</pre><p>Now paste the key from earlier as the password. The SSH connection should successfully log you in. Any further attempts will fail, as the password is good for one use only!</p><p>Once you have finished you can tear down your environment with a simple command</p><pre>vagrant destroy -f or sudo vagrant destroy -f</pre><h3>Conclusion</h3><p>HashiCorp Vault is a great answer to the problem of secrets management in a very elegant way and I’ve seen many clients that could benefit from the centralization, dynamic provisioning and rich auditing that Vault provides. All of Vault’s capabilities can also be accessed via the HTTP API as well as CLI and UI, which comes in handy when access use Vault to store our passwords and access them in our CI/CD pipeline.</p><p>I hope this helps you get started using Vault!</p><p><strong><em>PS: If you liked the article, please support it with claps. Cheers</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c90cefe77381" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using feature toggles to recover an EC2 server in Terraform]]></title>
            <link>https://medium.com/@bruce_25864/using-feature-toggles-to-recover-an-ec2-server-in-terraform-69342f9f25ab?source=rss-d5553ac6322e------2</link>
            <guid isPermaLink="false">https://medium.com/p/69342f9f25ab</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp]]></category>
            <dc:creator><![CDATA[Bruce Dominguez]]></dc:creator>
            <pubDate>Sat, 09 Feb 2019 10:06:44 GMT</pubDate>
            <atom:updated>2019-02-09T22:02:17.961Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Feature Toggles with Terraform</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MkYU-wNPVwt3QVBe" /><figcaption>Photo by <a href="https://unsplash.com/@juskteez?utm_source=medium&amp;utm_medium=referral">Juskteez Vu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article will describe how you can use feature toggling for creating Terraform resources with a single feature flag.</p><p>For context I hadn’t realised I needed one until I blew away my dev Jenkins EC2 instance, luckily I had automated snapshots created by our nightly Lambda function. So the problem was how can I easily restore the server back to its former glory using snapshots and in an automated way that was repeatable using Terraform. I also wanted to build a more resilient Terraform script to ease restoration in the future by simply typing Terraform Apply.</p><p>After reading a great article <a href="https://medium.com/capital-one-tech/building-feature-toggles-into-terraform-d75806217647">building feature toggles into terraform</a> by Chris Pisano I had the inspiration to us a feature flag that would determine if an EC2 instance is to be restored or a new instance created. So here are the steps I used to do this.</p><h4><strong>Restoring from snapshot</strong></h4><p>Before I can create a feature toggle I needed to code the restore and I broke it down into these steps:</p><ol><li>Find the ebs snapshots for restore.</li><li>Create an AMI (a bit odd but you will see why).</li><li>The restore the instance using the new AMI.</li><li>Associate the new instance with the existing elastic ip.</li></ol><h4><strong><em>Find the ebs snapshot</em></strong></h4><p>Using the data <a href="https://www.terraform.io/docs/providers/aws/d/ebs_snapshot.html">aws_ebs_snapshot </a>data source you can find the snapshot you need to restore from based on certain filters such as name and volume size for example.</p><pre>data &quot;aws_ebs_snapshot&quot; &quot;root&quot; {<br>most_recent = true<br>owners = [&quot;self&quot;]<br>filter {<br>name = &quot;volume-size&quot;<br>values = [&quot;64&quot;]<br>}</pre><pre>filter {<br>name = &quot;tag:Name&quot;<br>values = [&quot;${var.snapshot_name}&quot;]<br>}<br>}</pre><p>In the example above I am filtering on the size of my volume (root) and the name of my snapshot which I have set as a variable <strong>var.snapshot_name</strong> as to not hardcode it in. This would be needed to be done for all snapshots required to be restored.</p><h4><strong>Create an AMI</strong></h4><p>Next, I need to create an AMI that I would use to launch the new restored server. It might seem an unnecessary step, however, if I just attached the snapshots under the aws_instance it would only present as additional volumes and not use the root snapshot to restore all of the previous config and state. By creating an AMI from the snapshot I can overcome this issue.</p><pre>resource &quot;aws_ami&quot; &quot;restore&quot; {<br>count = &quot;${var.restore}&quot;<br>name = &quot;from-${data.aws_ebs_snapshot.root.snapshot_id}&quot;<br>virtualization_type = &quot;hvm&quot;<br>root_device_name = &quot;/dev/xvda&quot;</pre><pre>ebs_block_device {<br>device_name = &quot;/dev/xvda&quot;<br>snapshot_id = &quot;${data.aws_ebs_snapshot.root.id}&quot;<br>volume_size = &quot;${data.aws_ebs_snapshot.root.volume_size}&quot;<br>}<br>}</pre><p>The above creates the AMI with the snapshot defined as root. You will notice the count parameter, I’ll get into why we are using that very soon.</p><h4><strong><em>The restored instance</em></strong></h4><p>Now to define the instance to be restored using the new restore AMI I just created, again you will notice the count parameter, I’ll get to that soon.</p><pre>resource &quot;aws_instance&quot; &quot;restore&quot; {<br>count = &quot;${var.restore}&quot;<br>ami = &quot;${aws_ami.restore.id}&quot;<br>instance_type = &quot;t2.medium&quot;</pre><h4><strong><em>Associate the elastic IP</em></strong></h4><p>So that we don’t overwrite the existing elastic ip we would use the <a href="https://www.terraform.io/docs/providers/aws/r/eip_association.html">aws_eip_association</a> resource.</p><pre>resource &quot;aws_eip_association&quot; &quot;restore&quot; {<br>count = &quot;${var.restore}&quot;<br>instance_id = &quot;${aws_instance.restore.id}&quot;<br>allocation_id = &quot;${aws_eip.this.id}&quot;<br>}</pre><p>This would also need to be added to the original resource in the same fashion. For example:</p><pre>resource &quot;aws_eip_association&quot; &quot;this&quot; {<br>count = &quot;${1- var.restore}&quot;<br>instance_id = &quot;${aws_instance.this.id}&quot; <br>allocation_id = &quot;${aws_eip.this.id}&quot;<br>}</pre><h4><strong>The Terraform Count Parameter</strong></h4><p>As you would have spotted throughout the examples I had a count parameter set on each of the restore resources with a variable of <strong>var.restore</strong>. The count parameter tells Terraform how many of resources it should create with zero being an acceptable value. With zero as the value, no resources are created. Also taking advantage that Terraform converts 1 and 0 as to a boolean true or false we can use it as a feature of:</p><pre># This is just a pseudo code. It will not work in Terraform</pre><pre>if ${var.restore} {<br>  resource &quot;aws_ami&quot; &quot;restore&quot;,<br>  resource &quot;aws_instance&quot; &quot;restore&quot;,<br>  resource &quot;aws_eip_association&quot; &quot;restore&quot;<br>}<br>else {<br>  resource &quot;aws_instance&quot; &quot;this&quot;,<br>}</pre><p>This great but I need to create a restored resource and not create a brand new Jenkins server. To do this I needed to take advantage of the fact Terraform also does simple math in interpolations. By adding a -1 to the count it would do the -1 + 1 = 0 (false). For example:</p><pre>variable &quot;restore&quot; {<br>description = &quot;Used to restore the jenkins server&quot;<br>value = true<br>}<br>variable &quot;snapshot_name&quot; {<br>description = &quot;snapshot name&quot;<br>}</pre><pre>resource &quot;aws_instance&quot; &quot;restore&quot; {<br>count = &quot;${var.restore}&quot;<br>......</pre><pre>resource &quot;aws_instance&quot; &quot;this&quot; {<br>count = &quot;${1- var.restore}&quot;<br>....</pre><p>The above would create the restored resource but not the Jenkins instance.</p><h4>Creating a Terraform toggle.</h4><p>With my var.restore variable now controlling the creation of either my restored or brand new Jenkins server, I needed to be able to toggle this feature on or off when needed. To do this I used the *<strong>.auto.tfvars </strong>file (I named restore.auto.tfvars, but you can name it whatever you want). If a <strong>*.auto.tfvars</strong> is present in the current directory, it is picked up by Terraform and is read at runtime passing any variables and values required.</p><pre># restore.auto.tfvars<br># restore key controls restore toggle<br># Snapshot_name controls what snapshot to restore.<br>restore = false<br>snapshot_name = &quot;&lt;insert snapshot name here&gt;&quot;</pre><p>My <strong>variables.tf</strong> now only passes on the variable to the <strong>restore.auto.tfvars</strong> which now controls the restore toggle.</p><pre>variable &quot;restore&quot;{<br>description = “Used to restore the jenkins server&quot;<br>}<br>variable &quot;snapshot_name&quot; {<br>description = “snapshot name<br>}</pre><h4>Related:</h4><p>Thanks, <a href="https://blog.gruntwork.io/@brikis98">Yevgeniy Brikman</a> from Gruntwork for writing <a href="https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9">Terraform tips &amp; tricks: loops, loops, if-statements, and gotchas</a> and thanks to <a href="https://medium.com/@chrispisano">Chris Pisano</a> for writing <a href="https://medium.com/capital-one-tech/building-feature-toggles-into-terraform-d75806217647">Building Feature Toggles into Terraform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69342f9f25ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Windows 2016 server AMI on AWS with Packer and Terraform. Part 2]]></title>
            <link>https://medium.com/@bruce_25864/deploying-a-windows-2016-server-ami-on-aws-with-packer-and-terraform-part-2-3a7dca0caef2?source=rss-d5553ac6322e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a7dca0caef2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Bruce Dominguez]]></dc:creator>
            <pubDate>Thu, 17 Jan 2019 13:04:36 GMT</pubDate>
            <atom:updated>2019-01-18T04:24:21.785Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8CB1gp7sKmuxqf04" /><figcaption>Photo by <a href="https://unsplash.com/@benelwood?utm_source=medium&amp;utm_medium=referral">Ben Elwood</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my <a href="https://medium.com/@bruce_25864/deploying-a-windows-2016-server-ami-on-aws-with-packer-and-terraform-part-1-8a2f771ecabf">previous article</a>, we conquered WINRM to build our golden AMI using Packer, installed our custom applications using chocolatey and overcame connectivity issues with RDP. I now had a working AMI in AWS that I could use to spawn multiple copies on demand. It was time to automate the provisioning of the server so that it could be predictably provisioned and destroyed cleanly. My requirement was to keep the configuration of IaC in source control, for this I opted to use Terraform from HashiCorp. I won’t go into too much detail on the benefits of using Terraform, but it is a very powerful tool to deploy, teardown and codify your infrastructure.</p><p>Breaking up my Terraform Script to deploy my infrastructure I wrote 3 files:</p><p><strong>Main.tf </strong>— This would hold the meat and potatoes of what I was building, my server and configuration.</p><p><strong>Variables.tf </strong>— This holds all the variables used by my Main.tf to use, allowing me to change just a value in one file instead of sifting through my main.tf</p><p><strong>Output.tf</strong> — This is where I specify any outputs that I need (more on this later).</p><p>You can check out my terraform scripts <a href="https://github.com/brucedominguez/Windows-2016svr-Packer-Terraform/tree/master/Terraform">here</a>.</p><p>So to kick off my Main.tf file I need to first let Terraform know that I want to provision to AWS. To do this just add the below:</p><pre>provider “aws” {</pre><pre>region = “${var.aws_region}”</pre><pre>profile = “${var.aws_profile}”</pre><pre>}</pre><p>I use variables <strong>${var.aws_region} </strong>to reference my AWS region in variables.tf and <strong>${var.aws_profile}</strong> to also reference my AWS CLI named profile. Using the CLI named profile makes it easier to have multiple AWS accounts e.g. Dev, Prod etc. This easily set using <strong>aws configure — profile Prod</strong>. <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html">Check</a> here for more info.</p><p>Next I need to dynamically identify the correct subnet to deploy to using Tags I have set up in my AWS VPC.</p><pre># — — Get VPC ID — — -<br>data “aws_vpc” “selected” {<br> tags = {<br> Name = “${var.name_tag}”<br> }<br>}</pre><pre># — Get Public Subnet List<br>data “aws_subnet_ids” “selected” {<br> vpc_id = “${data.aws_vpc.selected.id}”</pre><pre>tags  = {<br> Tier = “public”<br> }<br>}</pre><p>Using <strong>data”aws_vpc”</strong> resource in Terraform to identify which one of my subnets are Prod or Staging environment, then using good old interpolation syntax in the <strong>data “aws_subnet_ids” </strong>resource to pull out my public subnet.</p><p>I also use another data resource to help identify an already existing security group that I had defined. Of course, you can always create your own specific security group for this, which I did end up doing to keep everything contained for my project.</p><pre>data “aws_security_group” “selected” {<br> tags = {<br> Name = “${var.name_tag}*”<br> }<br>}</pre><p>Now before I define my EC2 instance I need to do a few more steps. I needed to make sure I could RDP to my instance, which I managed to overcome in Part 1 by setting the userdata script to be used by my EC2 instance at boot.</p><pre>data “template_file” “user_data” {<br> template = “/scripts/user_data.ps1”<br>}</pre><p><strong>The fourth problem — Dynamically create and store Key pairs on S3.</strong></p><p>As part of this project I wanted to also dynamically create the instance key pairs, register them and store them on S3 for later use. This took a bit of head scratching and some googling to the solution that worked cleanly, mainly the storing on S3 part. During the research (googling) I came across a great Terraform Module by the guys at <a href="https://cloudposse.com/">Cloud Posse</a> that had the perfect Module for this.</p><pre>module “ssh_key_pair” {<br>source = “git::https://github.com/cloudposse/terraform-aws-key-pair.git?ref=master&quot;<br> namespace = “example”<br> stage = “dev”<br> name = “${var.key_name}”<br> ssh_public_key_path = “${path.module}/secret”<br> generate_ssh_key = “true”<br> private_key_extension = “.pem”<br> public_key_extension = “.pub”<br>}</pre><p>Check the module at the Cloud Posse Github repo <a href="https://github.com/cloudposse/terraform-aws-key-pair">here</a>.</p><p>Now to store my new shiny keys on my S3 bucket. I found some solutions for this problem however they seemed too convoluted for what essentially was just a file copy. So I decided to use Terraform’s the local-exec provisioner and kick off the copying of the keys on to my S3 bucket with the AWS CLI.</p><pre># — — Copy ssh keys to S3 Bucket<br> provisioner “local-exec” {<br> command = “aws s3 cp ${path.module}/secret s3://PATHTOKEYPAIR/ — recursive”<br> }</pre><pre># — — Deletes keys on destroy<br> provisioner “local-exec” {<br> when = “destroy”<br> command = “aws s3 rm 3://PATHTOKEYPAIR/${module.ssh_key_pair.key_name}.pem”<br> }</pre><pre>provisioner “local-exec” {<br> when = “destroy”<br> command = “aws s3 rm s3://PATHTOKEYPAIR/${module.ssh_key_pair.key_name}.pub”<br> }</pre><p>The first provisioner copies both keys from the path specified in ssh_public_key_path section of the “ssh_key_pair” module to my S3 bucket using AWS CLI commands.</p><p>The last two provisioners remove the keys when Terraform Destroy is performed. This is done using the when = “destroy”. <strong>NOTE: </strong>Don’t forget to add this to your aws_instance resource.</p><p>Now time to get to configure the EC2 instance with the AMI we created with our Packer script in Part 1. To do this we first must find the AMI we created using the Terraform <strong>data “aws_ami”</strong> resource and also filter to find our image.</p><pre>data “aws_ami” “Windows_2016” {<br> filter {<br> name = “is-public”<br> values = [“false”]<br> }</pre><pre>filter {<br> name = “name”<br> values = [“windows2016Server*”]<br> }</pre><pre>most_recent = true<br>}</pre><p>With the AMI image defined we can use it when we create the ec2 instance with <strong>${data.aws_ami.Windows_2016.image_id}</strong>, with image_id as the attribute of the resource. With a few variables thrown in, I had the windows 2016 server ready to deploy.</p><pre>resource “aws_instance” “this” {<br> ami = “${data.aws_ami.Windows_2016.image_id}”<br> instance_type = “${var.instance}”<br> key_name = “${module.ssh_key_pair.key_name}”<br> subnet_id = “${data.aws_subnet_ids.selected.ids[01]}”<br> security_groups = [“${data.aws_security_group.selected.id}”]<br> user_data = “${data.template_file.user_data.rendered}”<br> iam_instance_profile = “${var.iam_role}”<br> get_password_data = “true”</pre><pre>root_block_device {<br> volume_type = “${var.volume_type}”<br> volume_size = “${var.volume_size}”<br> delete_on_termination = “true”<br> }</pre><pre>tags {<br> “Name” = “NEW_windows2016”<br> “Role” = “Dev”<br> }</pre><p><strong>The fifth problem — Automating the decryption of the Admin password.</strong></p><p>Having to log into the AWS console every time to decrypt the admin password when I want to RDP on to the server would be an absolute pain, and not very “DevOps-ey”. So I needed a solution that would be to automate the process of decrypting the password and then presenting to me as an output of that process. In order for me to do this I first needed to get the encrypted password from the server. Terraform is able to do this by using the “get_password_data” argument and setting it to True. However, a base64 encrypted password was useless and a nightmare to type. Terraform to the rescue again! Using <strong>rsadecrypt </strong>as part of the output in my Output.tf will decrypt the password generated and present it in a human-readable format.</p><pre>output “Administrator_Password” {<br> value = “${rsadecrypt(aws_instance.this.password_data, file(“${module.ssh_key_pair.private_key_filename}”))}”<br>}</pre><p>Now when I run Terraform Apply I get:</p><pre>Administor_Password = XXADMIN PASSWORDXXX</pre><p>Success! Now I have my admin password as an output when I run <strong>Terraform apply</strong>, all without having to log in to the AWS console.</p><p><strong>Conclusion</strong></p><p>Automating the build of a Windows 2016 server is not as straightforward as I initially thought. Overcoming build issues in Packer (problem 1, 2 and 3) and my deployment challenges with Terraform (problem 4 and 5) I can now move onto integrating the solution in a CI process. Hopefully, this article assists anyone in the same situation as I was. Thanks for reading!</p><p><strong>Next Steps.. Terratest and BDD</strong></p><p>My next project will be to write a <strong>BDD</strong> framework that will sit on top of my Terratest scripts. <a href="https://github.com/gruntwork-io/terratest">Terratest </a>is a GO library written by the awesome guys at <a href="https://gruntwork.io/">Gruntwork</a> that help write automation tests to test your infrastructure code. The Terraform and Packer scripts are still code and still need to be validated after every update before it is released into Prod, and adding these tests as part of our CI server will provide that feedback early and often.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a7dca0caef2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Windows 2016 server AMI on AWS with Packer and Terraform. Part 1]]></title>
            <link>https://medium.com/@bruce_25864/deploying-a-windows-2016-server-ami-on-aws-with-packer-and-terraform-part-1-8a2f771ecabf?source=rss-d5553ac6322e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a2f771ecabf</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[packers]]></category>
            <dc:creator><![CDATA[Bruce Dominguez]]></dc:creator>
            <pubDate>Thu, 10 Jan 2019 07:45:42 GMT</pubDate>
            <atom:updated>2019-01-10T07:45:42.726Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUk_mC8ojHhStMEURjbI8g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/cOkpTiJMGzA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Oskar Yildiz</a> on <a href="https://unsplash.com/search/photos/developer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Automating a deployment of a Windows 2016 Server on AWS should be easy right, after all deploying an ubuntu server with Packer and Terraform was as easy as typing <strong>Packer Build </strong>and <strong>Terraform apply</strong>! Easy.. well maybe not that easy. I came across a few challenges but I learned a lot along the way and here is what I learned.</p><h4><strong>The First problem — Packer... Not connecting via SSH.. Enter WINRM</strong></h4><p>Having created images before with Packer using SSH to connect, which it uses by default to communicate and build your host server. Here I ran into my first problem in building my Windows server. To build the server I needed to switch to WINRM as the communication method. Now WINRM runs a lot slower, as i painstakingly found out, so be warned. But wait, how does my instance know to use WINRM for my install? To do this I needed to pass a PowerShell script as a User Data file to run the configuration at boot time. If you are used to bootstrapping a server then this shouldn’t be new to you. The script I used can be found <a href="https://github.com/brucedominguez/Windows-2016svr-Packer-Terraform/blob/master/Packer/windows16.json">here</a>. So I updated my packer JSON script with the changes, here is a snapshot below.</p><pre>“builders”: [{<br> “type”: “amazon-ebs”,<br> “source_ami_filter”: {<br> “filters”: {<br> “virtualization-type”: “hvm”,<br> “name”: “Windows_Server-2016-English-Full-Base-*”,<br> “root-device-type”: “ebs”<br> },<br> “owners”: [“amazon”],<br> “most_recent”: true<br> },<br> “instance_type”: “t2.micro”,<br> “ami_name”: “<em>SERVERNAME-WINDOWS</em>”,<br> “user_data_file”:”./scripts/SetUpWinRM.ps1&quot;,<br> “communicator”: “winrm”,<br> “winrm_username”: “Administrator”,<br> “winrm_insecure”: true,<br> “winrm_use_ssl”: true,</pre><h4><strong>The second problem — Getting Ansible working was proving to be a challenge… Enter Chocolatey</strong></h4><p>With my packer woes far behind me, I began to writing my Ansible scripts to configure the server. But again Ansible runs afoul of the same problems, it required WINRM to communicate to the host and several updates and configuration changes.</p><p>To get Ansible working was going to take more time and for a limited return, so I chose to use a more Windows friendly package manager, Chocolatey. It works much like Linux Apt-get or yum to install applications on Windows, with a simple <strong>Choco Install app</strong>. This was simple to implement as part of my packer build by writing a PowerShell script to run the install. Make sure that the global auto confirm is enabled. Here is an example script below or check it out on <a href="https://github.com/brucedominguez/Windows-2016svr-Packer-Terraform/blob/master/scripts/exampleChoc.ps1">GitHub</a></p><pre># Install Chocolatey</pre><pre>iex ((New-Object System.Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1&#39;))</pre><pre># Globally Auto confirm every action</pre><pre>choco feature enable -n allowGlobalConfirmation</pre><pre># Install Python 2.x 2.7.15</pre><pre>choco install python2</pre><p>But before we go ahead and install all our packages I needed to do something about Windows User Access Control (UAC). It can be a pain when you want to install an app on your server and you are prompted for the admin password and will halt my unattended application installs. So running a quick PowerShell script prior to the chocolatey installs will be required. This is also on G<a href="https://github.com/brucedominguez/Windows-2016svr-Packer-Terraform/blob/master/scripts/disable-uac.ps1">itHub</a></p><pre>Write-Host “Disabling UAC…”<br>New-ItemProperty -Path HKLM:Software\Microsoft\Windows\CurrentVersion\Policies\System -Name EnableLUA -PropertyType DWord -Value 0 -Force<br>New-ItemProperty -Path HKLM:Software\Microsoft\Windows\CurrentVersion\Policies\System -Name ConsentPromptBehaviorAdmin -PropertyType DWord -Value 0 -Force</pre><p>So defining my PowerShell scripts in my Packer JSON script looks a little like this.</p><pre>“provisioners”: [<br> {<br> “type”: “powershell”,<br> “scripts”: [<br> “./scripts/disable-uac.ps1”,<br> “./scripts/so-chocolatey.ps1”<br> ]<br> },</pre><h4><strong>Third problem — RDP… why you no work after sysprep!</strong></h4><p>Awesome I have all my packages installed on my server. Now it’s time to Sysprep this server to ensure it has a new computer security Identifier (SID). This can be done with two PowerShell commands that are specifically for Windows 2016 servers (below). These are already available on the server supplied by AWS. So let’s add these new scripts as inline PowerShell scripts to my script.</p><pre>{<br> “type”: “powershell”,<br> “inline”: [<br> “C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule”,<br> “C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown”<br> ]<br> }<br>]</pre><p>Now to make sure that I haven’t missed anything in my script by running</p><pre>Packer validate Windows.json</pre><pre>Template validated successfully</pre><p>Time to test it and create my new Windows 2016 AMI by simply typing <strong>packer build windows.json</strong>. 10 mins later I had a new AMI under images in my EC2 console in AWS, did I mention WINRM is a little slow?</p><p>With my new shiny Windows AMI it was time boot an instance and take it for a spin... Or so I thought, with RDP failing to connect to my instance. After checking NACL’s and Security Group settings it appears the Sysprep had disabled the terminal services. This was easily remedied with a PowerShell script that would run as part of the user data on bootup. You can check out the script I used <a href="https://github.com/brucedominguez/Windows-2016svr-Packer-Terraform/blob/master/scripts/Enable-RDP.ps1">here</a>.</p><p>I now have a brand new Windows AMI with all of my apps installed and I am able to launch multiple instances and connect to them. In part 2 I will we will use Terraform to predictably deploy the new server in AWS, store our key pairs on S3 and automate the decryption of the Windows Admin password to easily connect them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a2f771ecabf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>