<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simon So on Medium]]></title>
        <description><![CDATA[Stories by Simon So on Medium]]></description>
        <link>https://medium.com/@simon.so?source=rss-339658c039dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hW1AOcFUBulehUnzzNO9bg.jpeg</url>
            <title>Stories by Simon So on Medium</title>
            <link>https://medium.com/@simon.so?source=rss-339658c039dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:51:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@simon.so/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How To Setup Terraform Cloud Remote State]]></title>
            <link>https://medium.com/@simon.so/how-to-setup-terraform-cloud-remote-state-e1b3253d960f?source=rss-339658c039dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1b3253d960f</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Simon So]]></dc:creator>
            <pubDate>Fri, 24 May 2019 22:43:16 GMT</pubDate>
            <atom:updated>2019-05-24T22:43:16.890Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vpI-XrIhLgU1M7cEXI20DQ.png" /></figure><h3>Introduction</h3><p>Managing Terraform remote state files has always been an extra bit of hassle that requires some pre-planning and fore-thought for all but maybe the smallest (one-person) teams. Earlier, I wrote about why you should store your Terraform state remotely in a <a href="https://www.simonso.dev/posts/7-tips-to-start-your-terraform-project">previous post</a>, but there are still some key decisions to be made around “how”: which storage backend should you use? Does it support file locking or do you have to implement that yourself? How do you organize the state files across your team? Across your organization?</p><p>This week, Hashicorp helped us answer some of these questions by releasing <a href="https://www.hashicorp.com/blog/introducing-terraform-cloud-remote-state-management">Cloud Remote State Management</a> to all users, which means in a few easy steps, you and you team can can off-load the storage and management of your remote state files to Hashicorp’s cloud. You don’t have to provision and maintain your own infrastructure or create S3 or GCS buckets for your state files any more!</p><h3>Step-by-Step</h3><p>Hashicorp’s own <a href="https://www.youtube.com/watch?v=ZGl8wlxlcIU">demo video</a> does a great job of walking through the steps to set this up. But for those who prefer reading text over watching video, here are the steps and some color commentary:</p><h4>1. Get Terraform (&gt;0.11.13)</h4><p>Get the latest version of Terraform <a href="https://www.terraform.io/downloads.html">here</a>. Cloud Remote State is supported starting from version 0.11.13 and up. Keep in mind the highly-anticipated version 0.12 also was just released at roughly the same time last week and it contains backward-incompatible changes, so if you are working with existing, pre-0.12 Terraform configs, it might be a good idea to just grab the last 0.11 release — 0.11.14 — from <a href="https://releases.hashicorp.com/terraform/">here</a> for now and figure out the upgrade details separately. For new Terraform, pass “Go” and collect version 0.12.</p><h4>2. Create a Terraform Cloud account and an organization</h4><p>Go to <a href="https://app.terraform.io/">app.terraform.io</a> and create an account and an organization to hold your state files. From my experience, organization names need to be globally unique and the creation process will prompt you if an organization already exists.</p><h4>3. Create a user token</h4><p>Create a user token <a href="https://app.terraform.io/app/settings/tokens">here</a>. This step creates a token that is used to authenticate who or whatever is running Terraform with Terraform Cloud. The token should be put in a `.terraformrc` file in the user’s home directory.</p><pre>credentials “app.terraform.io” {<br>  token = “&lt;insert user token here&gt;”<br>}</pre><h4>4. Run `terraform init/plan/apply` to ensure there are no pending changes</h4><p>We will be migrating our state file so it’s a good idea to make sure everything is synced up before moving. One less thing to worry about.</p><h4>5. Add or modify the remote backend definition</h4><p>The definition of your remote backend should look something like this:</p><pre>terraform {<br>  backend “remote” {<br>    hostname = “app.terraform.io”<br>    organization = “&lt;name of your organization&gt;”</pre><pre>    workspaces {<br>      name = “&lt;name of your workspace&gt;”<br>    }<br>  }<br>}</pre><h4>6. Run `terraform init/plan`</h4><p>When you run `terraform init`, you will be asked whether you want to copy your state file to the new remote backend.</p><pre>Initializing the backend…</pre><pre>Acquiring state lock. This may take a few moments…</pre><pre>Do you want to copy existing state to the new backend?</pre><pre>Pre-existing state was found while migrating the previous “local” backend to the newly configured “remote” backend. No existing state was found in the newly configured “remote” backend. Do you want to copy this state to the new “remote” backend? Enter “yes” to copy and “no” to start with an empty state.</pre><pre>Enter a value: yes</pre><pre>Releasing state lock. This may take a few moments…</pre><pre>Successfully configured the backend “remote”! Terraform will automatically use this backend unless the backend configuration changes.</pre><p>Now that setup is done, when you make changes to your infrastructure, you will see the different versions of your state file from the Web UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBaAgj5RJvJd9Vvtl2lQgg.png" /></figure><p>If you click into a version of the state, you can see a diff between the current and the last version — super handy!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ryjzwdtUPinsRHhmWJtfJw.png" /></figure><p>There is also an option to manually lock the state file — effectively preventing any Terraform runs against this state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ustIGG8JyVq7S3lgTnZDjA.png" /></figure><h3>Wrap up</h3><p>Cloud Remote State Management appears to be the first of a few major <a href="https://app.terraform.io/signup">collaboration features</a> that Hashicorp is planning to release to the masses, as promise in the Hashiconf <a href="https://www.hashicorp.com/blog/terraform-collaboration-for-everyone">keynote</a> last year. If you remember, the team behind the open-source project <a href="https://www.runatlantis.io/">Atlantis</a> joined Hashicorp last year, and the key selling point of Atlantis was improving collaboration between Terraform developers via pull request automation. In the coming months, we should expect to see Atlantis’ features being re-introduced as part of the Terraform Cloud offerings, available to all Terraform users for free. And that’s great news for everyone!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1b3253d960f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 Tips to Start Your Terraform Project the Right Way]]></title>
            <link>https://medium.com/@simon.so/7-tips-to-start-your-terraform-project-the-right-way-93d9b890721a?source=rss-339658c039dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/93d9b890721a</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Simon So]]></dc:creator>
            <pubDate>Fri, 28 Dec 2018 21:01:00 GMT</pubDate>
            <atom:updated>2018-12-28T21:01:00.762Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*9FQVJwRJrzPDncseivWmKg.png" /></figure><h3>Introduction</h3><p>Terraform, released 4 years ago in 2014, has risen to the top of a proliferation of DevOps tools, and distinguished itself in a very short amount of time. As far as pure infrastructure-as-code tools go, it competes primarily against proprietary offerings such as AWS CloudFormation, Azure Resource Manager, and GCP Deployment Manager. But Terraform also competes against (and arguably trumps) other popular configuration management tools that have been around for much longer, such as, Puppet, Chef, and Ansible, in terms of ease of use, speed, community support.</p><p>Like Chef/Puppet, Terraform encourages a declarative-style (describing what you want done, instead of how) and immutable infrastructure, a pattern where machine images, servers, other resources are not changed once they have been created. When a change is necessary, new resources are created and the old ones decommissioned.</p><p>In helping our clients move towards IAC and immutable infrastructure, I have been writing Terraform for a little over 2 years. In this time, I have picked up a few recommended patterns and practices to follow when using Terraform. The following is a collection of lessons that I have personally learned, and a few tricks, hacks, and workarounds that I have come across to get Terraform to behave the way I wanted.</p><p>A note of caution: Terraform <a href="https://www.hashicorp.com/blog/terraform-0-1-2-preview">0.12</a> is coming soon. 0.12 brings a host of new features and even some breaking changes in the language (HCL) itself, so some of the workarounds I describe below will no longer be necessary post-0.12</p><h3>1. Remote State</h3><p>Terraform, by default, stores the last-known (the last time you ran it) state of the infrastructure (resource IDs, names, properties, metadata, etc) in a JSON file locally. Keeping your Terraform state file local works fine if you are the only person working on and running the Terraform templates. But as soon as your team grows beyond 2 people, you will run into problems.</p><p>Imagine Alice and Bob are working on the same Terraform file, each on their own laptops. The files are stored and managed with a Git repository. All good so far. Alice writes a simple Terraform template to create an EC2 instance, checks it in to Git, and then runs it locally on her machine to create the EC2 instance. Terraform generates a local state file on her machine that describes all the properties and metadata of the instance. No problem.</p><p>Bob now wants to run the same Terraform, so he checks out the code from Git and then tries to run it. Because Bob has never run this code before, a local state file was never generated. From Bob’s Terraform’s point of view, there are no EC2 instances, so Terraform thinks that it needs to create one (because that’s what the code says). When Terraform tries to create the instance, it will most likely fail with an error saying the instance already exists, because it does, Alice created it, but Bob’s Terraform doesn’t know about it.</p><p>The solution is to use a shared, remote state file. Typically, this would be an S3 or GCS bucket, and the declaration would look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2d1aed73b39028b9e5ecf75c8fa53c4/href">https://medium.com/media/f2d1aed73b39028b9e5ecf75c8fa53c4/href</a></iframe><p>By allowing multiple people to use the same state file, it effectively lets everyone share the same “view” of the infrastructure, and have a consistent experience when Terraform runs.</p><p>One major inconvenience about the remote state backend declaration is that none of the properties can be interpolated, so we cannot do this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2351468a62267f32e1b1fb832fff3421/href">https://medium.com/media/2351468a62267f32e1b1fb832fff3421/href</a></iframe><p>This means if you use a separate bucket per environment (a good practice), you have to hard code it, OR leverage <a href="https://www.terraform.io/docs/backends/config.html#partial-configuration">partial configuration</a> and build your own wrapper script around Terraform to inject the backend values dynamically at runtime. <a href="https://github.com/gruntwork-io/terragrunt">Terragrunt</a> is a great tool in this regard. Instead of having to copy and paste essentially the same Terraform backend block everywhere, you can leave it blank:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1531cc3a8ea34045f4dd03b7aeaf2b11/href">https://medium.com/media/1531cc3a8ea34045f4dd03b7aeaf2b11/href</a></iframe><p>Then, declare the remote state block just once in a .tfvars file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdff56a98df25fc25019de860fed5ae7/href">https://medium.com/media/fdff56a98df25fc25019de860fed5ae7/href</a></iframe><p>See the <a href="https://github.com/gruntwork-io/terragrunt#keep-your-remote-state-configuration-dry">Terragrunt Github page</a> for more details.</p><p>While we are on the topic of remote state: do not <a href="https://www.terraform.io/docs/state/index.html#inspection-and-modification">edit</a> your state files by hand. It usually doesn’t end well.</p><h3>2. Separate Your Environments</h3><p>In addition to separating the remote state backend by environment, it is good practice to separate the Terraform for each environment in its own folder as well, not only for the purpose of code organization but it also allows for better and easier CI and automation integration, which can target a specific environment folder and execute terraform plan and terraform apply separately.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92cba54014c950b4efd68b568f8a036b/href">https://medium.com/media/92cba54014c950b4efd68b568f8a036b/href</a></iframe><p>Implementing this one-folder-per-environment pattern, you end up having to repeat (copy and paste) a bunch of Terraform code from one environment to another. The next two sections present few more ideas of how to re-use Terraform code and not repeat yourself too much.</p><h3>3. Use Modules</h3><p>Using <a href="https://www.terraform.io/docs/modules/usage.html">modules</a> is a must when writing Terraform of any level of complexity, not only does it help you organize your code by separating concerns, it will lead to more code reuse and less repetition.</p><p>One useful feature is the ability to refer to or source a module in source control directly. In this case, we are using Git, but Mercurial sources are supported as well. We can specify a ref to the commit (hash or tags) to lock down the version of the module that we want.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c3cacb2ea7f7c1b9e2a27f89d605618/href">https://medium.com/media/3c3cacb2ea7f7c1b9e2a27f89d605618/href</a></iframe><p>This allows us to publish common modules that are used across multiple teams, which means individual teams have to write less Terraform, and the maintenance burden is lessened as there is one place to look — the common module — if something needs to be fixed.</p><h3>4. Keep it DRY</h3><p>Don’t Repeat Yourself (DRY) is a principle that discourages repetition, and encourages modularization, abstraction, and code reuse. Applying it to Terraform, using modules is a big step in the right direction.</p><p>However, repetitions still happen. You may end up having virtually the same code in N different environments, and when you need to make one change, you have to make the change N times.</p><p>There are a few ways to address this problem. One that I have seen and used at several clients is to create a folder for shared or common files, and then create symlinks to these files from each environment. This way, you can make a change to the common file(s) once and it appears in all your environments.</p><p><a href="https://github.com/gruntwork-io/terragrunt#motivation">Terragrunt</a> solves the same problem in a different way. It is a wrapper around the Terraform CLI commands, which allows you to write your Terraform once, and then in a <a href="https://github.com/gruntwork-io/terragrunt#keep-your-terraform-code-dry">separate repository</a> define only the input variables for each environment — no need to repeat Terraform code for each environment. Terragrunt is also quite handy for orchestrating Terraform in CICD pipelines for multiple separate projects.</p><h3>5. Conditionals</h3><p>Terraform supports conditionals through the syntax of a ternary operator: CONDITION ? VAL_IF_TRUE : VAL_IF_FALSE. The most common use case is a conditional resource based on an input variable and the meta-parameter count. In the following example, a storage bucket is created if create_bucket is true, otherwise, no bucket is created.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f499936f83b31af7eb144e9cecd62d0/href">https://medium.com/media/3f499936f83b31af7eb144e9cecd62d0/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07b6cc130ae8f465f267f697089b609b/href">https://medium.com/media/07b6cc130ae8f465f267f697089b609b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/431dca136000cbdd8f77033c25dbc142/href">https://medium.com/media/431dca136000cbdd8f77033c25dbc142/href</a></iframe><h3>6. The null_resource</h3><p>The <a href="https://www.terraform.io/docs/provisioners/null_resource.html">null_resource</a> may be useful if you need to do something that is not directly associated with the lifecycle of an actual resource. Within a null_resource, you can configure provisioners to run scripts to do essentially whatever you want. For example, you could SSH into an instance and run a command, or connect to a database to execute a query, or simply run a script to register an instance with DNS.</p><p>It should be noted, however, that whatever is being done inside a null_resource is not going to be managed by Terraform. So if in your null_resource you decide to call a gcloud command to create resources, maybe a compute instance, for example, Terraform is not going to know about this and therefore cannot manage its lifecycle and state. Still it is occasionally useful to run a script whenever Terraform runs or whenever a resource changes (using <a href="https://www.terraform.io/docs/provisioners/null_resource.html#triggers">triggers</a>).</p><p>In general, it is a good idea use null_resource sparingly and when you do, vet the scripts being called to make sure it is as idempotent as possible.</p><h3>7. Other Useful Functions</h3><p>As you become more familiar with Terraform, and as the infrastructure and corresponding Terraform code become more and more complex, you start to want more functionality and flexibility. Inevitably, you will need to to use one of the many useful <a href="https://www.terraform.io/docs/configuration/interpolation.html">built-in interpolation functions</a>. The following are a few notable ones that I use a lot or find interesting:</p><ul><li>format() and formatlist() format a string or a list of strings. The following example left-pads the cluster_id with zeros to 4 digits</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d2b3a5dfaa32d12bffec3fff07dd465/href">https://medium.com/media/9d2b3a5dfaa32d12bffec3fff07dd465/href</a></iframe><ul><li>matchkeys(values, keys, searchlist) — filters a list of values with corresponding keys and returns only values that have keys in the searchlist. The following example returns a list of instances that are in the first zone.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ec15376b1bbc92a656ff4cafdfad912/href">https://medium.com/media/5ec15376b1bbc92a656ff4cafdfad912/href</a></iframe><ul><li>element(list, index) — to access elements in a list variable (the [] notation also works). A neat “feature” of this function is that it wraps around the list, for example:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5d39ccab732413ed8710131c6f84744/href">https://medium.com/media/f5d39ccab732413ed8710131c6f84744/href</a></iframe><h3>Conclusion</h3><p>Within the Cloud/DevOps team at <a href="https://www.slalom.com/locations/silicon-valley">Slalom Silicon Valley</a>, Terraform is our team’s current to-go choice for any infrastructure automation projects, due to its feature set and roadmap, ability to work with multiple clouds, ease of use, and community support. If you are starting a Terraform project, here is a handy checklist of best practices to follow:</p><ol><li>Use remote state/backend</li><li>Separate environments</li><li>Use modules</li><li>Keep it DRY (with tools like Terragrunt)</li><li>Use conditionals for flexibility</li><li>Use null_resource for edge cases (use sparingly)</li><li>Use built-in interpolation functions</li></ol><p>If you are working with GCP, Google has a close partnership with Hashicorp — there is a team of Googlers who maintains the GCP provider! There is also a set of Terraform modules authored and maintained by Googlers on <a href="https://github.com/terraform-google-modules">Github</a> that can be used as-is or as references for how to write (and test!) Terraform modules.</p><p><em>Any questions or comments on this article or Terraform in general, feel free to reach out to me at </em><a href="/service/mailto:simon.so@slalom.com"><em>simon.so@slalom.com</em></a><em>! Better yet, join us at the next </em><a href="http://meetu.ps/e/G9mzL/B4jlL/f"><em>Hashicorp User Group meetup</em></a><em> on January 24th, 2019 @ Slalom SV office in Palo Alto! Link to official meetup event page and to previous meetup presentation slides: </em><a href="https://www.meetup.com/Silicon-Valley-HashiCorp-User-Group/"><em>here</em></a><em> and </em><a href="http://bit.ly/oct-svhug-presentation"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93d9b890721a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>