<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nathan Mclean on Medium]]></title>
        <description><![CDATA[Stories by Nathan Mclean on Medium]]></description>
        <link>https://medium.com/@mcleanjnathan?source=rss-7d38c781a431------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6AA_csUivLxyJW0fNJIo3A.png</url>
            <title>Stories by Nathan Mclean on Medium</title>
            <link>https://medium.com/@mcleanjnathan?source=rss-7d38c781a431------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:08:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mcleanjnathan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[We Ditched our On-Call Rota]]></title>
            <link>https://medium.com/spaceapetech/we-ditched-our-on-call-rota-fae4bf948f3?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/fae4bf948f3</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[on-call]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Thu, 28 Sep 2023 13:07:31 GMT</pubDate>
            <atom:updated>2023-09-28T21:00:10.967Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TQMs43X5ZmOKBbrz" /><figcaption>Photo by <a href="https://unsplash.com/@gary_at_unsplash?utm_source=medium&amp;utm_medium=referral">Gary Chan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The DevOps team at Space Ape has always participated in an on-call rota, with each person talking their turn spending 7 straight days and nights on-call, dragging around a laptop. Now the rota is gone and the laptop mostly stays at home, but we’re still on-call.</p><p>In this post I’ll talk through the history of on-call in Space Ape and explain how our new system works.</p><h3>History</h3><p>In the early days of Space Ape (before my time) responding to incidents was a constant in life. Space Ape was figuring out how to develop, run and scale games as scrappy start up. This meant lots of things broke, a lot.</p><p>But lessons were learnt and, by the time I joined in late 2016, on-call was much more manageable. However out-of-hours alerts still went off more often than we’d like; probably a couple of times a month.</p><p>Since then we massively reduced the number of out-of-hours alerts that we had to respond to. How we did this would probably need a couple more blog posts, but in short we:</p><ul><li>Stopped alerts that could wait until morning from escalating — <em>That high CPU alert does not need to wake you up</em></li><li>We focussed on a few key metrics that indicated poor performance for players — Request Success Service Level Objective (<strong>SLO</strong>) and Sharp drops in Concurrently Connected Users (<strong>CCU</strong>)</li><li>Made a huge number of varied performance and stability improvements to both Client and Server code and to the infrastructure</li><li>Invested heavily in load testing prior to launch. We test our servers can handle load beyond even our wildest dreams of success</li><li>Created a company wide escalation procedure. Now anyone can point out a serious problem and escalate it to the right people, any time of the day, and get a suitable response</li><li>Gained a huge amount of experience in running games at scale, which helps us make good decisions in development</li></ul><p>All of this meant that the burden of being on-call 24 hours a day, 7 days a week was starting to out-weight the value to the business. We went months without an out-of-hours incident, but someone still needed to have their laptop with them.</p><p>Team members have to adapt their lifestyles when on-call — can I go on out tonight? Can I go on that hike? Will I have signal, or even a place to work? Can I keep the laptop safe when out in public?</p><p>We also spent a significant amount of time chopping and changing the on-call rota to work around holidays and individual schedules.</p><p>So a change was needed.</p><h3>The New System</h3><p>Ok, so technically the whole team (currently 4 people) are now on-call, all the time… But let me explain.</p><p>Instead of a rota we now send all out-of-hours alerts to one member of the team, if after 15 minutes they haven’t acknowledged the alert then the next member of the team gets the alert and so on until all members of the team have been alerted, then it goes back to the beginning.</p><p>If you respond to an alert you get moved to the bottom of the list, so that in future you’re the last to be called.</p><p><em>At the time of writing I have been at the top of the list for at least 6 months.</em></p><p>Before we implemented this we had a think about what could go wrong and how we could mitigate these issues.</p><h4>Responding without your laptop</h4><p>There’s now a higher chance than before that you might respond to an alert without access to your work laptop.</p><p>So we investigate how we could respond, using just our phones.</p><ul><li>We made sure that we could connect to the VPN to gain access to internal tooling (such as logging and deployment tools)</li><li>We updated internal web tools to ensure they are mobile friendly</li><li>We made sure that we had the apps we needed to access resources — can we get to passwords and MFA codes, Slack and GitHub</li><li>We checked that we could use other web interfaces, like AWS or Kibana, through our phones.</li></ul><h4>What if No One is Available</h4><p>With a small team it seemed possible that everyone could be busy and have gone out without their laptop.</p><p>Firstly we checked that we can provide some support from our phones, as described above.</p><p>We also have a check in each week, at the end of our team guild meeting (a chance to share with the rest of the team) to ensure that at least someone will have access to a laptop, or be within an hour of one, over the weekend.</p><p>Generally we’ve found that at least one person will be around, or that someone is doing something where carrying a laptop with them isn’t a burden.</p><h4>Holidays</h4><p>When you’re on holiday you remove yourself from the list of responders, holidays are your time and you shouldn’t be responding to incidents — go and enjoy yourself!</p><h4>Exceptions</h4><p>Sometimes we revert to our old system of rota based on-call. This is generally around times of significant importance, such as the launch of a new game. As it happens we’re currently doing 24/7 on-call to support the release of <a href="https://www.countrymusicgame.com/">Country Star</a>.</p><p>This is generally for a period of 1–2 weeks post launch and we each get to volunteer for the days that suit us best.</p><h3>Summary</h3><p>It’s possible to ditch your on-call rota, provided that the volume of out-of-hours incidents is at a suitably low level and that your team agrees to it.</p><p>For us at Space Ape it has reduced the stress of maintaining an on-call schedule and re-arranging lifestyles to ensure we can have a laptop, internet access and a place to work 24/7.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fae4bf948f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/we-ditched-our-on-call-rota-fae4bf948f3">We Ditched our On-Call Rota</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A Terraform Provider — Part 2]]></title>
            <link>https://medium.com/spaceapetech/creating-a-terraform-provider-part-2-1346f89f082c?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/1346f89f082c</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Wed, 20 Feb 2019 12:12:30 GMT</pubDate>
            <atom:updated>2019-02-20T12:12:30.379Z</atom:updated>
            <content:encoded><![CDATA[<h3>Creating A Terraform Provider — Part 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdiE4o-pooVfk6Kg6B8PLg.jpeg" /></figure><p>In <a href="https://medium.com/spaceapetech/creating-a-terraform-provider-part-1-ed12884e06d7">part 1</a> of Creating a Terraform Provider, we walked through how to implement the functionality of a provider; we have a Provider that can create, update and delete ‘items’ by interacting with a simple API. In part 2 we’ll look at how to test the provider as well as how to run it.</p><h3>Testing</h3><p>Hashicorp provides helper functions to aid you in testing your provider. One such function is InternalValidate, which runs some checks on your provider. We can add this test to a file called provider_test.go.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f79dc9f4dc564e2eddbf3ba6dfe0762/href">https://medium.com/media/1f79dc9f4dc564e2eddbf3ba6dfe0762/href</a></iframe><p>To this file, we can add an init function which will initialise our provider for use in testing our Item resource.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f539e917e4e1a5456ab2d6ada70e67f2/href">https://medium.com/media/f539e917e4e1a5456ab2d6ada70e67f2/href</a></iframe><p>Lastly, we can add a function that will check that we have all the configuration, in the form of environment variables, that we need to connect to a running instance of a server.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a982d2542468b31d20a50d1a0fede213/href">https://medium.com/media/a982d2542468b31d20a50d1a0fede213/href</a></iframe><p>As we have the code for the server we could run it in memory and use setup and teardown functions in the tests, but you won’t always have this option, so we’ll test against a server that is started outside of the tests.</p><p>Now we can start testing our Item resource. Hashicorp has provided more helper functions on the resource Type that make testing resources much easier. We can set up a resource.TestCase and provide some setup functions and test cases and let Terraform take care of the rest:</p><ul><li>Creating the plan</li><li>Applying the plan</li><li>Another plan again — to check that there are no changes left to apply (which will fail the test)</li><li>Checking the resource exists</li><li>Checking that the Terraform state is correct</li><li>Tidy up the resource(s) we created.</li></ul><p>Let’s look at a test for our Item resource. These tests will live in resource_item_test.go.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bd6d253268202b2a3e43c703357e66d/href">https://medium.com/media/3bd6d253268202b2a3e43c703357e66d/href</a></iframe><p>You can see that we set up a resource.Test and provide it with the following:</p><ul><li>PreCheck — This is the testAccPreCheck function that we created in provider_test.go which will check our environment variables are set.</li><li>Providers — This is a map of providers, containing our example provider that we set up in the init function of provider_test.go</li><li>CheckDestroy — This is a function that will check that the item is successfully tidied up at the end of the test and also acts as a test for the resourceDeleteItem function in resource_item.go</li><li>Steps — A number of test steps to run, in this case, we only have one, we’ll see examples with multiple steps later</li></ul><p>In each Step of a test we there are a couple of things to provide:</p><ul><li>Config — This is a string containing Terraform (HCL) code. I’m using a function that returns this</li><li>Check — A function (of type resource.ComposeTestCheckFunc) which takes a number of TestCheckFunc — some like TestCheckResourceAttr are provided as helpers but we can also write our own such as testAccCheckExampleItemExists.</li></ul><p>Let’s have a look at the various components of the test in more detail.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be93fe0fc7f9c2ce4bd14bcd947e32c7/href">https://medium.com/media/be93fe0fc7f9c2ce4bd14bcd947e32c7/href</a></iframe><p>The testAccCheckItemDestroy is run at the end of our test after all of the Steps have been run and the resources have been destroyed. The function retrieves our client from the provider (testAccProvider) we set up using the init function in provider_test.go. Then for each Resource in the state s it checks if the resource exists. When we make an apiClient.GetItem call we expect a not found error to be returned, otherwise, we return an error which fails the test.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05b111a89c39caebfdf42fd80be7971c/href">https://medium.com/media/05b111a89c39caebfdf42fd80be7971c/href</a></iframe><p>The testAccCheckItemBasic function returns a string containing some Terraform code. This is what the Test will run. Note that by default the tests will be run in parallel, so it’s important to give these test resource unique names so that they don’t collide on the server and give you unexpected errors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f38cb2b56d1f3398a61dc837e46a8d5e/href">https://medium.com/media/f38cb2b56d1f3398a61dc837e46a8d5e/href</a></iframe><p>testAccCheckExampleItemExists checks that the Item we asked the test to create was actually created, this is run before the testAccCheckItemDestroy function gets run. It makes a GetItem request to the server and expects not to receive an error.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41a30fb85ed0f60c69e4f269187e1d66/href">https://medium.com/media/41a30fb85ed0f60c69e4f269187e1d66/href</a></iframe><p>We also use a built-in check, TestCheckResourceAttr, which takes the resource we want to test, the attribute name and the value we expect that attribute to be and verifies that the value is correct in state.</p><p>Where you have a List or Set you can check the length by using the # value. For a List, you can check the value of each element by providing the index, eg tags.0.</p><p>For Sets there is a hash of the value that’s stored used as the id to look up, eg tags.1931743815. To find out the hash I’ve tried using the schema.HashString function, which seems to be what is used to generate the hash, but we get a different result back. I’ve ended up just printing out the resource (fmt.Println(rs.Primary)) in the testAccCheckExampleItemExists function and then using that.</p><h4>Testing Resource Updates</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e5d152644b54572ac96b50ff05d7821/href">https://medium.com/media/3e5d152644b54572ac96b50ff05d7821/href</a></iframe><p>We can also test updating a resource. This can be done using multiple test steps. The first step is exactly as we have done before with the TestAccItem_Basic.</p><p>The second step has a different Config value which updates the resource created during the first step. We use testAccCheckItemUpdatePost which updated the description. We can then pass in a number of checks to verify that the resource exists and all of the values are as we expect, including the updated description.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5761137180476f5f1402a648c4b78d14/href">https://medium.com/media/5761137180476f5f1402a648c4b78d14/href</a></iframe><p>Another test is one that validates that the validateName function is correctly applied to the name attribute. In this test, we apply some Config, which is a resource with spaces in its name and tells the TestStep to expect to fail by providing a regex that should match the error that Terraform returns to the ExpectError field. We can provide the regex whiteSpaceRegex which, is defined in a variable.</p><p>The final test I’ve created for this provider is one that tests the import function. This test sits is a separate file; import_resource_test.go</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90fc6cc9c9c7be98858df4300c1003cb/href">https://medium.com/media/90fc6cc9c9c7be98858df4300c1003cb/href</a></iframe><p>This test starts the same way as the others; we apply some configuration to create a resource. In a second Step, we then tell Terraform to import that resource and verify the state.</p><h4>Running the Tests</h4><p><strong>Note</strong>: I’m using <a href="https://github.com/golang/go/wiki/Modules">Go modules</a> for this project so you’ll need to be using a Go version ≥ 1.11 and have the environment variable export GO111MODULE=on set to be able to run the API server, tests and build the provider. Alternatively, you can use a tool like <a href="https://github.com/golang/dep">dep</a> to vendor the dependencies.</p><p>To run the tests you first need to start the API server — go run api/main.go</p><p>Then, in a separate terminal session, we can run our tests:</p><p>TF_ACC=true SERVICE_ADDRESS=<a href="http://localhost/">http://localhost</a> SERVICE_PORT=3001 SERVICE_TOKEN=superSecret go test -v provider/*</p><p>TF_ACC tells Terraform that we’re running acceptance tests. We also set the environment variables that our provider requires to set up a client. Then we run all of the tests in the provider package. Terraform requires that the -v flag is applied when TF_ACC is true.</p><p>TF_ACC is used to differentiate Acceptance tests from Unit tests as you won’t always want to test against your external dependencies.</p><h3>Running The Provider</h3><p>Usually when you run a provider Terraform will go off and download it for you, but this is only possible with officially supported providers. To run your own provider it must be stored in the ~/.terraform.d/plugins/[distro] directory on Linux/OSX or in terraform.d\plugins in your user&#39;s &quot;Application Data&quot; directory on Windows. The provider needs to be named in a specific way; terraform-provider-[providerName]_[version] where version is a semantic version in the form vX.X.X.</p><p>Running make in the root of the sample code will build the provider, which you can then move the correct location. The version comes from the version file. This plugin file won’t work for people on different platforms though, you’ll need to compile it separately for each platform people may use it from.</p><p>Once you have your plugin built and in the correct location you can create a terraform file similar to this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f093e1a4f665d01a2675dfe932beeb3/href">https://medium.com/media/3f093e1a4f665d01a2675dfe932beeb3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*Yxd8LbEOklMnGpIKP2Dnjg.png" /></figure><p>Next, we can start the server, to do this run make startapi or go run api/main.go which will start a server on localhost:3001.</p><p>Now we can run terraform init, terraform plan and terraform apply to see our provider in action.</p><h3>Some Final Advice</h3><p>Two final pieces of advice to leave you with:</p><ol><li>Reading through the code for the of the official providers can be a really useful way to learn how to write your own. All of the official providers live in the <a href="https://github.com/terraform-providers">terraform-providers</a> organisation on GitHub</li><li>The Terraform Go Docs are well written, so I recommend spending some time reading through the docs for the helpers that Hashicorp provides, for example, the <a href="https://godoc.org/github.com/hashicorp/terraform/helper/resource">resource helper</a>. For instance, the <a href="https://godoc.org/github.com/hashicorp/terraform/helper/resource#TestStep">TestStep</a> explains how to use each of the fields, including some I haven’t mentioned, such as Taint, which will let you test that your provider works as expected with a tainted resource.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1346f89f082c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/creating-a-terraform-provider-part-2-1346f89f082c">Creating A Terraform Provider — Part 2</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A Terraform Provider - Part 1]]></title>
            <link>https://medium.com/spaceapetech/creating-a-terraform-provider-part-1-ed12884e06d7?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed12884e06d7</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Tue, 20 Nov 2018 10:45:06 GMT</pubDate>
            <atom:updated>2018-11-20T10:45:06.047Z</atom:updated>
            <content:encoded><![CDATA[<p>A fairly common task in Ops is to figure out how to manage a service programmatically; it’s often easy to get started with a service by using the UI but this won’t scale and eventually, we’ll want to have more control over the configuration of a service; a Terraform provider can be the tool that gives you that control.</p><p>To create a Terraform provider you just need to write the logic for managing the Creation, Reading, Updating and Deletion (CRUD) of a resource and Terraform will take care of the rest; state, locking, templating language and managing the lifecycle of the resources.</p><p>Just taking a look at the list of <a href="https://www.terraform.io/docs/providers/index.html">existing providers</a> shows you how versatile Terraform can be. The list includes cloud providers, Kubernetes, DNS services, GitHub, monitoring tools and TLS certificate providers among others.</p><p>At <a href="https://spaceapegames.com/">Space Ape</a> we decided that we wanted to start managing the configuration of our metrics service, <a href="http://wavefront.com/">Wavefront</a>, programmatically. We were all set to write a new tool that would allow us to template Alerts and Dashboards in YAML and then create them in Wavefront via their API. We had already started writing a <a href="https://github.com/spaceapegames/go-wavefront">Go client</a> for the Wavefront API when Hashicorp announced the <a href="https://github.com/hashicorp/terraform/blob/v0.10.0/CHANGELOG.md">release of Terraform v0.10.0</a>, which split Providers out of Terraform core allowing each provider to be developed and released independently of Terraform and for custom providers to be created.</p><p>So we set about building the <a href="https://github.com/spaceapegames/terraform-provider-wavefront">Wavefront Terraform Provider</a>.</p><h3>How to Build Your Own Provider</h3><p>To avoid having to work against a real API for this blog post, so that you can follow along if you wish, I’ve created a small API that stores (imaginatively) “items”. The API allows you to create, read, update and delete items. An item has a name, description and a list of tags. An items’ name must be unique. I won’t go into much more detail about the working of the API, you can find a more detailed description <a href="https://github.com/spaceapegames/terraform-provider-example/blob/master/Readme.md">here</a> and the code in <a href="https://github.com/spaceapegames/terraform-provider-example/tree/master/api">terraform-provider-items/api</a>.</p><p>The client that the Terraform provider will use to interact with the API shouldn’t be implemented within the provider itself, as that would overcomplicate the provider and mean that the client can’t be used for other purposes. So I’ve also written a client in the <a href="https://github.com/spaceapegames/terraform-provider-example/tree/master/api/client">API package</a>.</p><p>The full source code of the example provider and API is available on <a href="https://github.com/spaceapegames/terraform-provider-example">GitHub</a>.</p><h4>Getting Started</h4><p>The provider we’re going to build during this blog post will allow us to create an Item on our server using the following Terraform code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yFSDdnXt1mBwq1qC1D5RdA.png" /></figure><p>Terraform Plugins are binaries that Terraform communicates with via RPC. It’s possible to write a provider in any language, but in reality, you’ll want to write it in Go; Terraform provide helper libraries in Go to aid in writing and testing providers.</p><p>The name of the repository (and therefore directory) that your provider lives in is important; all providers start with terraform-provider- anything after that represents the name of the provider. In this case, I’m going for the very imaginative terraform-provider-example.</p><p>Next, we’ll create ./main.go which will serve as the entry point to our provider. main.go is just used to invoke our provider, which we will implement in a separate package, in this case, called provider.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9524cfd63974d8c7b613d6909fd637e7/href">https://medium.com/media/9524cfd63974d8c7b613d6909fd637e7/href</a></iframe><p>Now, we’ll create the provider package, within which will sit the implementation of your provider. Within the provider package, we’ll create provider.go and define the <em>Provider</em> function that our main.go called.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef8ced16693df1bc13c5b60dc735a462/href">https://medium.com/media/ef8ced16693df1bc13c5b60dc735a462/href</a></iframe><p>The Provider requires:</p><ul><li>A Schema which represents the various attributes we can provide to our provider via the provider block of a Terraform file. Note that if no value is provided we will check if environment variables are set. This is useful for making sure we don’t need to store secrets in the provider block of terraform files</li><li>A ResourceMap defines the names of the resources the provider has and where to find the definition of those resources. In this case, you can see we have example_item resource, the definition of which is a *schema.Resource returned by the resourceItem() function, which we’ll define later</li><li>A ConfigureFunc which can do any setup for us. In this case, we have providerConfigure which takes the address, port and token and returns a client that we’ll use to communicate with the API. Note the providerConfigure returns an interface{} so we can store anything we like here</li></ul><h4>Defining Our Item Resource</h4><p>Before we start defining our resource there are some more naming conventions to cover:</p><ul><li>Resource files are named resource_[resourceName].go<em>,</em> eg resource_item.go</li><li>Test files follow the usual Go naming standard of resource_[resourceName]_test<em>,</em> eg resource_item_test.go</li><li>We can also write import tests which, by convention, are in files named import_[resourceName]_test.go, eg import_item_test.go. Although these tests could sit within the resource_[resourceName]_test.go file</li><li>Lastly, there are also data source definitions, which we won’t be implementing here, but you may see in other providers. These are named data_source_[resourceName].go. Data Sources allow you to pull in information from resources that already exist, but that you don’t want to manage.</li></ul><p>We start by creating resource_item.go and defining the<em> </em><em>resourceItem() </em>function that we call from provider.go. This returns a *schema.Resource— the definition of our Item resource.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0a2923659e49eea9536c2bac7079cfa/href">https://medium.com/media/e0a2923659e49eea9536c2bac7079cfa/href</a></iframe><p>A schema.Resource needs us to set up a few things:</p><ul><li>A Schema— the attributes the Resources has</li><li>A number of functions. Earlier I said you just need to set up the Create, Read, Update and Delete functions for the provider, there are also two more, Exists and Imports, I’ll cover these later on.</li></ul><p>Let’s look at the Schema in more detail. The Schema element is a <em>map[string]*schema.Schema</em>, where the string is the attribute name and the *schema.Schema defines what the attribute is. In this case, the attribute name is of a Type TypeString, it is required and has a short description. It also has ForceNew set to true, this is because the API doesn’t allow you to change the name of an item after it is created, therefore Terraform would have to destroy the resource and create it again for this to happen.</p><p>Lastly, there is a <em>ValidateFunc</em> which is a function with the signature (v interface{}, k string) (ws []string, es []error)</p><p>Where v is the value of the attribute, which you need to use type assertion on to retrieve the actual type, k is the name of the attribute (“name” in this case) and it returns a slice of warnings (strings) and a slice of errors.</p><p>The ValidationFunc used for the name attribute is the validateName function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/929abf2a1fecb6a17fd8016f67a70fc0/href">https://medium.com/media/929abf2a1fecb6a17fd8016f67a70fc0/href</a></iframe><p>It returns an error if the name has any whitespace (The API doesn’t allow whitespace in names). This prevents us from making API calls we know will fail.</p><p>The description attribute is similar to name but doesn’t force a new resource and doesn’t have a validation func.</p><p>The tags attribute has a few differences. It is of type TypeSet, which is similar to a list, but where order doesn’t matter. You can use TypeList when you can be sure that the API will always return a list in the same order, otherwise, you’ll always have changes to apply. You’ll also notice that tags has a Elem field, which lets us define the type that is stored in the Set (or List), in this case, a string.</p><p>In our case, the API purposely returns tags in a random order. You can try changing the TypeSet to TypeList to see that there will (nearly) always be changes to apply due to the reordering of the tags.</p><p>It’s also worth noting that when you have a TypeSet or TypeList the Elem field can be of type&amp;schema.Resource, which allows for a deeper resource structure. For instance, this is how the listener blocks of an <a href="https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_elb.go#L159">aws_elb resource are defined</a>.</p><p>Also notice that tags are not a required attribute, but rather than just setting Required to false (it’s default value), we set Optional to true. Either Required or Optional must be true</p><h4>Functions</h4><p>The Create, Read, Update and Delete functions are functions with the signature (d *schema.ResourceData, m interface{}) error. Where d is essentially the Schema we defined above, but with values added, e.g. the “name” attribute has a value we can send to our API.</p><p><em>m</em> is the interface{} returned from the ConfigureFunc in provider.go, which in our case is our Client for talking to the server.</p><p>Each of the CRUD functions essentially just call their corresponding methods in the client, but there are a few things to note:</p><ul><li>When you create a resource you need to set the ID of the Terraform resource, this is done using the d.SetId<em> </em>method. The ID is generally the what the API uses to uniquely identify an item, in our case name is the ID, so we use that — d.SetId(item.Name)</li><li>Setting the ID to an empty string indicates to Terraform the item no longer exists. So in the resourceDeleteItem()<em> </em>function we call d.SetId(&quot;&quot;) after deleting the Item. We also do this in the <em>resourceReadItem</em> function if we get an error and that error contains “not found”</li><li>For each of these methods we obtain the Client from m, as it is an interface we have to assert the type as the client type —<em> </em><em>apiClient := m.(*client.Client)</em></li></ul><p>As the underlying data structure for the d *schema.ResourceData ends up being an interface, rather than a solid type we end up doing a lot of type assertion. On a small resource like our Item it is easy enough to do it within the function but, for larger resources, it may be worth splitting these out into separate functions.</p><p>In the resourceCreateItem function, we can see that we use d.Get to retrieve the values from the resource that we wish to pass to the API via the client and that we must perform type assertion on the result d.Get(&quot;name&quot;).(string).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea34ea94769bbaa93c90ae52efa7f252/href">https://medium.com/media/ea34ea94769bbaa93c90ae52efa7f252/href</a></iframe><p>The Exists function (resourceExistsItem()) is slightly different in that it doesn&#39;t modify Terraform state, nor does it update any resource on the server; it’s just used to check if a resource exists. It has a slightly different function signature than the CRUD functions in that it also returns a bool to indicate if the resource exists — resourceExistsItem(d *schema.ResourceData, m interface{}) (bool, error)</p><p>Finally, there is an import function which is used for importing a resource into Terraform, an import function is an implementation of &amp;schema.ResourceImporter. Terraform provides an implementation of this called schema.ImportStatePassthrough, which seems to work for the majority of use cases and you can always write your own implementation if you need to.</p><h4>End of Part 1</h4><p>At this point we have a functional Terraform provider, you could compile it and start creating <em>Items</em> with Terraform. However, there is a important piece missing — <strong>tests!</strong> In part 2 we’ll add tests to our Provider and run through how we get Terraform to use the provider.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed12884e06d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/creating-a-terraform-provider-part-1-ed12884e06d7">Creating A Terraform Provider - Part 1</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Journey to Better Deployments]]></title>
            <link>https://medium.com/spaceapetech/a-journey-to-better-deployments-a255eb69bbf2?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/a255eb69bbf2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[spinnaker]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Mon, 08 Oct 2018 09:58:52 GMT</pubDate>
            <atom:updated>2018-10-08T09:58:52.335Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OuFo5e3J3oy_PATxAnWTHw.jpeg" /></figure><p>For the past year, we’ve been working to improve our deployment process. Over that time we’ve built custom tooling and started using new (and said goodbye to old) tools and practices. This post will try and give an overview of why we wanted a new deployment process, how we went about implementing it and what we have now.</p><h3>The Old System</h3><h4>Infrastructure</h4><p>Our infrastructure was built using CloudFormation. CloudFormation templates were generated from <a href="https://github.com/cfndsl/cfndsl">CFNDSL</a>, which in turn were wrapped in an in-house Ruby Gem which made it a little easier to work with.</p><p>A CloudFormation stack generally produced an Auto Scaling Group, fronted by a Classic Load Balancer, with associated Security Groups and IAM roles and profiles.</p><p>CloudFormation was also used to set up other parts of our AWS infrastructure, such as VPC networking, CloudFront distributions and S3 Buckets.</p><p>All of our instances were managed by Chef. Chef ran every 15 minutes to configure our systems as we specified, it controlled every aspect of the machine. That means managing the configuration of the OS, installing system packaging and ensuring our instances were kept up to date and patched. It is also responsible for installing and configuring our applications.</p><h3>Deployments</h3><p>All deployments were orchestrated by Chef. When a deployment happened it was controlled by us and happened in one of three ways:</p><ol><li>Environments could be set up to take the latest available package on each Chef run. We used this to deploy the latest versions of our code when a Jenkins builds completed.</li><li>Deployments to other development environments were made using either a Web UI or via a CLI — Knife plugin — (both effectively using the same Ruby code underneath) which told Chef to deploy a specific version of code and configuration.</li><li>Production deploys were driven via command line on developers machines using a Knife plugin. This allowed a higher degree of control over the deployment. For instance, a deploy typically consisted of a canary node being deployed, a developer checking the logs and metrics before deciding whether or not to continue the deploy. If the deployment was continued then a rolling deploy to the remaining instances was started. Otherwise, the canary was rolled back.</li></ol><h3>Why Develop a New Deployment Process?</h3><h4>Faster Deployments</h4><p>Each deploy required a full Chef run, meaning we were checking all other parts of the system, just to install a Jar and a YAML configuration file and restart a service.</p><p>Deploys from CI took too long, Chef ran every 15 minutes, which means we waited, on average 7.5 minutes for a deployment to start, after a new version had been published. Developers want to see if their code works ASAP.</p><p>Production deployments took a long time:</p><ol><li>We took an instance out of the load balancer</li><li>Ran chef</li><li>Added the instance back in the load balancer</li><li>Waited for a developer to check the logs</li><li>Repeated steps 1–3 for the remaining instances (or rollback)</li></ol><h4>Safer Deployments</h4><p>Chef managed servers are mutable — they change over time. We were never 100% certain if we could build a new instance from scratch to the current state, this can lead to problems when machines (inevitably) die, or when we needed to scale up new instances.</p><p>Production deployments were run from developers machines, which meant there were any number of number of factors that caused problems for deployments, such as:</p><ul><li>Did the developer have all the pre-requisites installed?</li><li>Did they have the correct AWS credentials?</li><li>Was their network connection stable? What happened if it dropped out part way through a deployment?</li></ul><p>We relied heavily on developers checking logs and metrics to confirm that the new version we were deploying was working as expected. Much of this could be automated.</p><h3>The New System</h3><h4>Targets</h4><p>We set our requirements for our deployment system as:</p><ol><li>Deployments should happen in seconds or minutes (not 10’s of minutes or longer)</li><li>Deployments should not be run from a developers machine</li><li>We should deploy Immutable instances</li><li>Automate as many post-deployment checks as possible</li></ol><h4>Phase 1</h4><p>As we wanted immutable instances it was clear that we needed to bake application code into AMIs (Amazon Machine Images). We’d then need to orchestrate the deployment of these AMIs. We decided to use <a href="https://www.spinnaker.io/">Spinnaker</a> to do this. Spinnaker is “an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence”.</p><p>Essentially it allows you to define a pipeline defining how you want a deployment to happen.</p><p>A minimal Pipeline consists of a Bake stage and a deployment stage. A deployment stage can be made of a number of different <a href="https://www.spinnaker.io/concepts/#deployment-strategies">deployment strategies</a> that are built in (for example Red/Black, Canary or Highlander) or you can define your own. You’re also free to add other steps, such as adding optional steps to rollback or steps that can perform checks.</p><p>By using Spinnaker it is clear we are going to be able to meet our goals:</p><ul><li>Deployments of Baked AMIs should be fast using Spinnaker</li><li>Safe as we can promote an AMI through environments</li><li>Our instances are immutable</li><li>We’re also not running these deployments from our own machine</li></ul><p>But we haven’t done much in terms of automating deployment checks.</p><p>We quickly found that whilst the Spinnaker UI was perfect for our DevOps team to use, there is too much unnecessary information in there for many users (members of development teams that are just interested in deploying code, not the details of how the deployments work).</p><p>We decided to build a system to wrap Spinnaker (and any future deployment types) in order to abstract away some of the complexities of tools like Spinnaker and also to overlay some of the terminology used by development teams.</p><p>We chose to build an API, based on Ruby on Rails, which which will store information about our environments and services as well as allowing us to use background workers to track the state of deployments.</p><p>In front of the API we’ve built a React application, which acts as the primary user interface to the API.</p><h4>Pause and Reflect</h4><p>We now had a working deployment system, but before we continued we took time to pause and review our progress with our users. People had been used to our old deployment system and like any change it can be difficult to adjust. Many things that seem intuitive when developing a system turn out not to be for users.</p><p>From this process, we came up with a few things we wanted to change:</p><ul><li>The time to the first deployment (after a CI build of our application) is still too high. Baking an AMI can take several minutes, meaning we are only slightly faster than Chef deployed instance.</li><li>Our Web interface required some work to become more intuitive and better display information to users.</li><li>Integration between tools could be better. For instance, Jenkins jobs were starting Spinnaker deployments directly, skipping our own tooling.</li></ul><p>This feedback has led to a number of improvements.</p><h3>Phase 2</h3><h4>‘Trigger’ Deploy</h4><p>To speed up the first deployment times we’ve had to make some compromises. We realised that we were never going to make AMI baking fast enough, so instead we’ve broken one of our initial requirements — immutable instances — in the interest of speed.</p><p>We’ve developed a deployment strategy which we have dubbed Trigger Deploy. The full workings of Trigger Deploy could make an entire blog post on their own, so I won’t go into too much detail here.</p><p>Essentially we pass a message to long-lived instances via SNS and SQS to deploy a new version of a package. Each instance listens to the queue for a message that it’s interested in and then acts upon it.</p><p>This means that we now deploy in seconds following a successful CI build of code. But we’re a little more prone to errors, in this case, we feel it’s an acceptable compromise.</p><h4>Web Interface</h4><p>We actively solicited feedback in both face-to-face sessions and via a feedback button embedded in the tool to collect a lot of feedback from our users about how we can improve the UI. Such as where we navigate the user to after certain actions and how we lay out information so that it’s easy to read, balancing providing all the information a user needs with not overloading them.</p><p>We also pulled in additional information, such as the current state of an environment.</p><p>This is an ongoing exercise. Gathering and acting on user feedback through meetings, a feedback sheet and ad-hoc conversions means that we’re always improving the system.</p><h4>Tooling Integration</h4><p>We wanted our deployment API to be the central source of information for all deployments — deploys should be started, controlled by and monitored via the API. One way of doing this was through the React UI. But sometimes this doesn’t cut it — we need to programmatically interact with the API, for instance, if we want Jenkins to start a deployment after a CI build.</p><p>Of course, this could be achieved via the API, but dealing with authentication and setting up each Jenkins job was more difficult than just calling Spinnaker directly (Spinnaker jobs can be set up to trigger from Jenkins builds)</p><p>So we built a CLI, this lets us abstract most of the pain away from Jenkins calling the API directly and also gives users the option of using the CLI rather than having to use the UI.</p><h4>Terraform</h4><p>We’ve also started using Terraform, rather than CloudFormation for building out infrastructure. One of the primary benefits this gives us is being able to share information between stacks using remote state, rather than have a global configuration file. Other benefits include the use of modules, which give us a better interface than our old CloudFormation templates.</p><p>A side effect of using Spinnaker is that out stacks are a little simpler, in that we don’t have to create the Auto Scaling Group, as Spinnaker manages that for us.</p><h3>API</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*sGXFPkLnJnQ7eWCO4zRDAA.png" /></figure><p>The API has now become the glue that holds our deployments together. It is the source of truth for all deployment related activities.</p><p>Not only does it orchestrate Spinnaker and ’Trigger’ deployments, it has been built in such a way we can add new deployment types as we go. We currently have prototypes for deploying AWS Lambda applications using <a href="https://github.com/awslabs/serverless-application-model">SAM</a> and for running <a href="https://www.terraform.io/">Terraform</a>.</p><p>We’re also investigating how we can manage the configuration of our services via the API.</p><p>With the API at the centre of our deployment workflow we have been able to leverage it to create a number of tools, such as the React frontend, a Golang CLI and a Lambda based Slack bot.</p><h3>Conclusion</h3><p>The result of our work so far is that deployments are faster, it takes just a few seconds to deploy following a new version of the code being made available.</p><p>It takes a few minutes for the next environment, but we have a lot more confidence that the deployment will be successful.</p><p>Prod deployments are no longer run from a user’s machine and are faster than they used to be. We haven’t made as much progress as we’d have liked when it comes to the automated checking of the deployments health, this is something we’ll continue to work on going forward.</p><p>Going through this process has highlighted how important it is to check in with your users at regular intervals to see if you are meeting their requirements.</p><p>This is just the beginning of the journey to better deployments, we’ve laid a solid foundation, which we can build on to enable us to manage different deployment types as well as to continue to make deployments faster and safer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a255eb69bbf2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/a-journey-to-better-deployments-a255eb69bbf2">A Journey to Better Deployments</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing with Dynamo Local and Go]]></title>
            <link>https://medium.com/@mcleanjnathan/testing-with-dynamo-local-and-go-7b7000ef9602?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b7000ef9602</guid>
            <category><![CDATA[dynamo]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Fri, 10 Aug 2018 19:49:02 GMT</pubDate>
            <atom:updated>2018-08-10T19:49:02.311Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve recently done some work with Go and DynamoDB and needed to test my work. Luckily Amazon provides <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html">DynamoDB local</a>, which means that I don’t need to provision any real infrastructure in AWS and can run my tests offline. This post will walk through a simple example of interacting with DynamoDB with Go and how to test this code with Dynamo Local. All of the code for this example can be found in <a href="https://github.com/nathmclean/dynamodb-local-testing">GitHub</a>.</p><p>I use the <a href="http://github.com/guregu/dynamo">guregu/dynamo</a> library to interact with Dynamo as I find it provides a nice abstraction to the DynamoDB API.</p><p>To start with I create a struct representing the Items I wish to store in Dynamo:</p><pre><strong>// item/item.go<br>type </strong>Item <strong>struct </strong>{<br>   Id          string    `dynamo:&quot;item_id,hash&quot;`<br>   Name        string    `dynamo:&quot;name&quot;`<br>   Description string    `dynamo:&quot;description&quot;`<br>   CreatedAt   time.Time `dynamo:&quot;created_at&quot;`<br>   UpdatedAt   time.Time `dynamo:&quot;updated_at&quot;`<br>}</pre><p>Notice the tags on the struct fields, these provide some addition information to the <a href="http://github.com/guregu/dynamo">guregu/dynamo</a> library on how to marshal and unmarshal data to and from Dynamo. For instance, the ‘Id’ field will be shown in DynamoDb as ‘item_id’ and it will be a hash (primary key). The hash is only required here when we want to create Dynamo Tables using this type, as we will later.</p><p>Next, I’ll create an ItemService which will hold the DynamoDB client and for which we can write methods to interact with Dynamo, such as adding items to the database.</p><pre><strong>// item/item.go<br>type </strong>ItemService <strong>struct </strong>{<br>   itemTable dynamo.Table<br>}</pre><p>Next, come a couple of methods, one that creates a new dynamo.Table, which is a client for communicating with Dynamo and one that creates a new ItemService</p><pre><strong>// item/item.go<br>func </strong>newDynamoTable(tableName, endpoint string) (dynamo.Table, error) {<br>   <strong>if </strong>tableName == &quot;&quot; {<br>      <strong>return </strong>dynamo.Table{}, fmt.Errorf(&quot;you must supply a table name&quot;)<br>   }<br>   cfg := aws.Config{}<br>   cfg.Region = aws.String(&quot;eu-west-2&quot;)<br>   <strong>if </strong>endpoint != &quot;&quot; {<br>      cfg.Endpoint = aws.String(endpoint)<br>   }<br>   sess := session.Must(session.NewSession())<br>   db := dynamo.New(sess, &amp;cfg)<br>   table := db.Table(tableName)<br>   <strong>return </strong>table, nil<br>}</pre><p>There are a few things going on here. First we’re checking that we have been provided with a tableName (the name of the table we’re connecting to). We make this function private, so that we’re in control of when it’s used — we only need to create a dynamo client when we’re setting up our ItemService, or testing.</p><p>Next we’re setting up an AWS session with some configuration. In a real-world use case we’d also pass in the AWS region, rather than hard-coding it.</p><p>If we’ve been provided with an endpoint we’ll point the client to that, rather than allowing the client to use it’s default endpoints. In normal usage, we won’t supply an endpoint, but for testing it allows us to point to Dynamo Local.</p><pre><strong>// item/item.go<br>func </strong>NewItemService(itemTableName string) (*ItemService, error) {<br>   dynamoTable, err := newDynamoTable(itemTableName, &quot;&quot;)<br>   <strong>if </strong>err != nil {<br>      <strong>return </strong>nil, err<br>   }<br>   <strong>return </strong>&amp;ItemService{<br>      itemTable: dynamoTable,<br>   }, nil<br>}</pre><p>This function takes the name of our item table, sets up a client using the ‘newDynamoTable’ function we discussed above and finally returns a new ItemService which holds the dynamo client. Notice that we always send an empty string as the endpoint, this means we can’t accidentally send an invalid endpoint when we’re in production, but it does mean we can’t use this function in our tests.</p><p>Now we need some methods to interact with DynamoDB to read and write Items. These methods will be associated with the ItemService so that they have access to the Dynamo client.</p><pre><strong>// item/item.go<br>func </strong>(i *ItemService) CreateItem(item *Item) error {<br>   now := time.Now()<br>   item.CreatedAt = now<br>   item.UpdatedAt = now<br>   item.Id = xid.New().String()<br>   <strong>return </strong>i.itemTable.Put(item).Run()<br>}</pre><p>Here we add a new Item to the database. We set the created and updated times to the current time, generate an Id for the item and the use the Put method to write to the Dynamo Table.</p><pre><strong>// item/item.go<br>func </strong>(i *ItemService) GetItem(item *Item) error {<br>   <strong>return </strong>i.itemTable.Get(&quot;item_id&quot;, item.Id).One(item)<br>}</pre><p>Here’s an example of reading from DynamoDB.</p><p>Note that these are basic examples and don’t have any verification on the Items we’re operating on or any detailed error handling. For instance we would probably want to check that the Item we’re are creating has a Name and Description before we try and actually create it. There may also be errors we can handle. For instance, if we were throttled by Dynamo, we could retry with an incremental backoff (wait for a longer period of time between each retry).</p><p>Next, it’s time to test the code. In each test I create a new, randomly named, Dynamo table that each test can run against, meaning that two tests won’t clash if they are run in parallel.</p><p>I created a <a href="https://github.com/nathmclean/dynamodb-local-testing/blob/master/test_utils/dynamo-local.go">test_utils package</a> that creates a new table, using an interface as a schema.</p><pre><strong>// </strong>test_utils/dynamo-local.go<br><strong>func </strong>CreateTable(table <strong>interface</strong>{}) (string, error) {<br>   cfg := aws.Config{<br>      Endpoint: aws.String(&quot;http://localhost:9000&quot;),<br>      Region:   aws.String(&quot;eu-west-2&quot;),<br>   }<br>   sess := session.Must(session.NewSession())<br>   db := dynamo.New(sess, &amp;cfg)<br>   tableName := xid.New().String()<br>   err := db.CreateTable(tableName, table).Run()<br>   <strong>if </strong>err != nil {<br>      <strong>return </strong>&quot;&quot;, err<br>   }<br>   <strong>return </strong>tableName, nil<br>}</pre><p>The ‘table interface{}’ that the function accepts in our case will be the ‘Item’ type we created at the start of this post. By taking an interface we can reuse this function to create any table. Of course, this function will fail if you pass an interface that cannot be made to represent a Dynamo Table . For instance, a struct that does not have a tag defining the hash key — (`dynamo:”hash”`).</p><blockquote><strong>type </strong>OtherItem <strong>struct </strong>{<br> Id string `dynamo:&quot;item_id,hash&quot;`<br>}</blockquote><blockquote>This would be sufficient to create a table that we can use to test the Item type. The other fields will be added to the table when we create Item’s in the database. As long as the keys match there isn’t a problem.</blockquote><p>Now in each test of ItemService methods, we can use this function to set up a table for us. To save some repetition I’ve created a ‘newItemService’ function in item_test.go that will call ‘CreateTable’ and set up an ItemService configured to use that table and to use the Dynamo Local endpoint (we’ll set up Dynamo Local later).</p><pre><strong>// item/item_test.go<br>func </strong>newItemService() (*ItemService, error) {<br>   tableName, err := test_utils.CreateTable(Item{})<br>   <strong>if </strong>err != nil {<br>      <strong>return </strong>nil, fmt.Errorf(&quot;failed to set up table. %s&quot;, err)<br>   }<br><br>   db, err := newDynamoTable(tableName, &quot;http://localhost:9000&quot;)<br>   <strong>if </strong>err != nil {<br>      <strong>return </strong>nil, err<br>   }<br>   service := ItemService{<br>      itemTable: db,<br>   }<br>   <strong>return </strong>&amp;service, nil<br>}</pre><p>Tests for CreateItem and GetItem follow the same pattern:</p><ol><li>Setup a slice of test conditions (table tests)</li><li>Setup the ItemService using ‘newItemService’</li><li>Run through each test condition and evaluate if it passes or fails.</li></ol><pre><strong>// item/item_test.go<br>func </strong>TestItemService_CreateItem(t *testing.T) {<br>   cases := []<strong>struct </strong>{<br>      name string<br>      item *Item<br>      err  bool // Whether we expect an error back or not<br>   }{<br>      {<br>         name: &quot;created successfully&quot;,<br>         item: &amp;Item{<br>            Name:        &quot;spoon&quot;,<br>            Description: &quot;shiny&quot;,<br>         },<br>      },<br>   }<br><br>   service, err := newItemService()<br>   <strong>if </strong>err != nil {<br>      t.Fatal(err)<br>   }<br><br>   <strong>for </strong>_, c := <strong>range </strong>cases {<br>      t.Run(c.name, <strong>func</strong>(t *testing.T) {<br>         err := service.CreateItem(c.item)<br>         <strong>if </strong>c.err {<br>            assert.Error(t, err)<br>         } <strong>else </strong>{<br>            assert.NoError(t, err)<br>            assert.NotEqual(t, time.Time{}, c.item.CreatedAt)<br>            assert.NotEqual(t, time.Time{}, c.item.UpdatedAt)<br>         }<br>      })<br>   }<br>}</pre><p>We create a slice of an anonymous struct, to which we add a name (or short description of the test), the item we want to test and whether we expect it to succeed. You could also add the error you expect to receive, if you want to test specific error conditions.</p><p>We then create out ItemService and fail the test (t.fatal) if any of this process fails.</p><p>Next, we iterate through each test case, with each case being a subtest (t.Run). We use the name we defined for each test case as the name parameter to t.Run, which helps us identify which test failed and what we were trying to test with that test case.</p><p>For each case, we use <a href="https://github.com/stretchr/testify">assert</a> to check for errors. If we expected to get an error we assert that we did receive one and visa versa if we don’t expect an error. If we didn’t expect to receive an error we also check that the item has the time set correctly (remember we set this time in the CreateItem method).</p><p>Now we have tests that will set up tables in Dynamo Local and test our code using those tables, but we don’t have an instance of Dynamo Local to run against… We’ll use Docker to set this up.</p><p>First we’ll set up a ‘Dockerfile’</p><pre>FROM openjdk:7<br>RUN mkdir -p opt/dynamodb<br>WORKDIR /opt/dynamodb<br>RUN wget https://s3.eu-central-1.amazonaws.com/dynamodb-local-frankfurt/dynamodb_local_latest.tar.gz -q -O - | tar -xz<br>EXPOSE 8000<br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;DynamoDBLocal.jar&quot;]</pre><p>This builds on top of the openjdk container, setups up a directory for dynamo local, downloads and extracts the jar, open a port for it to listen on and then sets the entrypoint to ensure that Dynamo Local starts when the container does.</p><p>Next, we’ll set up a docker-compose.yml which will allow us to use docker-compose to build, start and stop our container for us.</p><pre><strong>dynamo</strong>:<br>  <strong>build</strong>: .<br>  <strong>ports</strong>:<br>  - 9000:8000<br>  <strong>command</strong>:<br>    -sharedDb</pre><p>This builds from The Dockerfile we created above in the same directory, maps our local port 9000 to the containers port 8000 (You may have noticed that we used localhost:9000 as the endpoint for DynamoDB in our code).</p><p>Finally, it sends the -sharedDb flag to Dynamo Local when we start the container. If we don’t use this flag then each request will use a different table, which means we can’t do things like reading the Item we just created to check if the create worked.</p><p>Next up is to create a Makefile which will run docker-compose to setup dynamo local, run the tests and the use docker-compose to stop dynamo local.</p><pre>TEST?=$$(go list ./... |grep -v &#39;vendor&#39;)<br>GOFMT_FILES?=$$(find . -name &#39;*.go&#39; |grep -v vendor)<br><br>default: test<br><br>fmt:<br>   gofmt -w $(GOFMT_FILES)<br><br>test: fmt<br>   docker-compose down<br>   docker-compose up -d --build --force-recreate<br>   go test -i $(TEST) || exit 1<br>   echo $(TEST) | \<br>      xargs -t -n4 go test -v<br>   docker-compose down</pre><p>Now i can just run make and my tests will run, hopefully successfully.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b7000ef9602" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alexa, air pollution in London]]></title>
            <link>https://medium.com/@mcleanjnathan/alexa-air-pollution-in-london-53afdd187cbd?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/53afdd187cbd</guid>
            <category><![CDATA[health]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[air-pollution]]></category>
            <category><![CDATA[amazon-echo]]></category>
            <category><![CDATA[alexa]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Thu, 05 Oct 2017 20:39:58 GMT</pubDate>
            <atom:updated>2017-10-05T20:39:58.598Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltpiLTbEVe3tVM72_YcsBw.png" /></figure><p>I’ve just released my first Alexa skill, <a href="https://www.amazon.co.uk/gp/aw/d/B01N5U78SC/ref=mp_s_a_1_2?ie=UTF8&amp;qid=1507212757&amp;sr=1-2&amp;pi=AC_SX236_SY340_QL65&amp;keywords=air+pollution&amp;dpPl=1&amp;dpID=71t11obB2UL&amp;ref=plSrch">Air Pollution</a>, which looks up air pollution levels from the <a href="http://www.londonair.org.uk/">London Air Org</a> API. Data is available for most London Boroughs and a few other places, located in SE England.</p><p>Future releases might include:</p><ul><li>Daily forecast lookup.</li><li>Specific pollutant levels.</li><li>Accepting more locations, perhaps mapping postcodes to the nearest measurement site.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53afdd187cbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Custom Terraform Provider for Wavefront]]></title>
            <link>https://medium.com/@mcleanjnathan/building-a-custom-terraform-provider-for-wavefront-6804ebcca8fb?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/6804ebcca8fb</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[wavefront]]></category>
            <category><![CDATA[hashicorp]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Thu, 28 Sep 2017 09:34:08 GMT</pubDate>
            <atom:updated>2017-09-30T17:36:22.037Z</atom:updated>
            <content:encoded><![CDATA[<h4>Originally posted on <a href="https://tech.spaceapegames.com/2017/09/28/building-a-custom-terraform-provider-for-wavefront/">The Space Ape Game Tech Blog</a></h4><h4>At Space Ape we’re increasingly turning to Golang for creating tools and utilities, for example — <a href="https://tech.spaceapegames.com/2017/06/05/de-comming-ec2-instances-with-serverless-and-go/">De-comming EC2 Instances With Serverless and Go</a>. Inevitably we’ll need to interact with our metric provider — Wavefront. To this end, our colleague <a href="https://tech.spaceapegames.com/author/louismccormack/">Louis</a> has been working on a <a href="https://github.com/spaceapegames/go-wavefront">Go client</a> for interacting with the Wavefront API, which allows us to query Wavefront and create resources such as Alerts and Dashboards. Up until now, we’ve been configuring these components by hand, which worries us — what happens if they disappear or are changed? How do we revert to a known good version or restore a lost Dashboard?</h4><p>We were set to start creating our own tool for managing Wavefront resources, but as luck would have it Hashicorp released version <a href="https://github.com/hashicorp/terraform/blob/v0.10.0/CHANGELOG.md">0.10.0 of Terraform</a> which splits providers out from the main Terraform code base and allows you to load custom (not managed by Hashicorp) providers without recompiling Terraform.</p><p>So we set about creating a <a href="https://github.com/spaceapegames/terraform-provider-wavefront">custom provider</a> and have so far implemented Alerts, Alert Targets and Dashboards and fully intend to continue adding functionality to both the <a href="https://github.com/spaceapegames/go-wavefront">SDK</a> and the provider in the future.</p><p>Now creating an Alert is as simple as:</p><pre>resource &quot;wavefront_alert&quot; &quot;a_terraform_managed_alert&quot; {<br>  name                   = &quot;Terraform Managed Alert&quot;<br>  target                 = &quot;test@example.com&quot;<br>  condition              = &quot;ts()&quot;<br>  display_expression     = &quot;ts()&quot;<br>  minutes                = 4<br>  resolve_after_minutes  = 4<br>  additional_information = &quot;This alert is triggered because...&quot;<br>  severity = &quot;WARN&quot; tags = [ &quot;terraform&quot;, ] <br>}</pre><p>You can find the latest released version, complete with binary <a href="https://github.com/spaceapegames/terraform-provider-wavefront/releases">here</a>.</p><p>Creating our own provider for Wavefront means that we get all the benefits of Terraform; resource graphs, plans, state, versioning and locking with just a little bit of effort required by us. Hashicorp has made a number of helper methods which means that writing and testing the provider is relatively simple.</p><p>Another benefit to writing a provider is that we can use the import functionality of Terraform to import our existing resource into state. Hopefully, Hashicorp will improve this to generate Terraform code soon, in the meantime, it shouldn’t be too difficult to script turning a state file (JSON) into Terraforms HCL.</p><h3>Using the Provider</h3><p>Terraform is clever enough to go and fetch the officially supported providers for you when you run terraform init. Unfortunately, with custom providers, it’s a little bit more complicated. You need to build the binary (We upload the compiled binary with each of our releases) and place it in the ~/.terraform.d/plugins/darwin_amd64/ (or equivalent for your system). Now when we run terraform init it will be able to find the plugin. After this the setup is pretty simple:</p><pre>provider &quot;wavefront&quot; { <br>  address = &quot;foo.wavefront.com&quot; <br>  token = &quot;wavefront_token&quot; <br>}</pre><p>You can export the address and token as an environment variable (WAVEFRONT_ADDRESS and WAVEFRONT_TOKEN respectively) to avoid committing them to Source Control (We highly recommend you do this for the token!).</p><h3>Writing your own Provider</h3><p>If you fancy having a go at writing your own provider then <a href="https://www.terraform.io/guides/writing-custom-terraform-providers.html?">this blog post</a> by Hashicorp is a good way to get started. I’d also recommend taking a look at the <a href="https://github.com/terraform-providers">Hashicorp supported</a> providers and using them as a reference when writing your own.</p><p><em>Originally published at </em><a href="https://tech.spaceapegames.com/2017/09/28/building-a-custom-terraform-provider-for-wavefront/"><em>tech.spaceapegames.com</em></a><em> on September 28, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6804ebcca8fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vault Configuration as Code]]></title>
            <link>https://medium.com/@mcleanjnathan/vault-configuration-as-code-95f97fd188da?source=rss-7d38c781a431------2</link>
            <guid isPermaLink="false">https://medium.com/p/95f97fd188da</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[vault]]></category>
            <dc:creator><![CDATA[Nathan Mclean]]></dc:creator>
            <pubDate>Sat, 02 Sep 2017 22:31:00 GMT</pubDate>
            <atom:updated>2017-09-02T22:31:00.282Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I wrote a <a href="https://tech.spaceapegames.com/2017/07/26/vault-configuration-as-code/">post</a> for the <a href="https://tech.spaceapegames.com/">Space Ape Games tech blog</a> describing how we’ve written a ruby gem to apply Vault configuration. We can now store out Vault configuration in config files, rather than just applying changes through the CLI. It also allows us to test our configuration changes against a dev instance of Vault – we spin up the server as part of a Jenkins job and then apply the configuration. This not only means that we test new configuration, but we also that we know that all of the configuration can be applied from scratch.</p><p><a href="https://tech.spaceapegames.com/2017/07/26/vault-configuration-as-code/">Read the full post.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95f97fd188da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>