<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis McCormack on Medium]]></title>
        <description><![CDATA[Stories by Louis McCormack on Medium]]></description>
        <link>https://medium.com/@louismccormack?source=rss-eb00737b90b5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VnTtX4Xc2brCYaSj1jKn1w.png</url>
            <title>Stories by Louis McCormack on Medium</title>
            <link>https://medium.com/@louismccormack?source=rss-eb00737b90b5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:30:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@louismccormack/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Performance Testing at Space Ape Games]]></title>
            <link>https://medium.com/spaceapetech/performance-testing-at-space-ape-games-eb6045c2b1b2?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb6045c2b1b2</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[aws-step-functions]]></category>
            <category><![CDATA[aws-device-farm]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Tue, 05 Mar 2024 17:47:24 GMT</pubDate>
            <atom:updated>2024-03-07T10:55:27.321Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L0qS-bMykqMwuj8UYN3H7Q.png" /></figure><p>Performance is paramount in the world of free-to-play mobile gaming. There is the obvious corollary: if we offer a sub-par experience to our players, they are unlikely to stick about, unlikely to come back and unlikely to spend money.</p><p>But there are also more opaque reasons: the Google discovery algorithms, for instance, treat performance issues (along with app crashes) as key signals for discoverability. At <a href="https://spaceapegames.com/">Space Ape Games</a>, we’ve seen a huge variance in the number of organic installs just from dropping out of certain performance tiers.</p><p>But <em>measuring</em> device performance is difficult. We need to support a wide range of devices; how can we ensure our games run smoothly on all of them? We release regular updates and new features; how do we ensure these changes haven’t degraded performance?</p><p>It certainly is challenging. One approach is to simply eyeball it: play through the game and hope to notice any performance issues before they make it into the wild. However this doesn’t scale. We have a world-class QA function here at Space Ape, but there is no way they could regularly play through our games on all of the devices we support. Plus their judgement would be subjective, and they may miss iterative degradations that eventually add up to a larger problem.</p><p>No, we needed a more scientific approach. As a tech organisation, we have a lot of experience measuring the performance of our backend systems. Could we apply some of that knowledge to this domain?</p><p><strong>This article is the story of how we did just that, with the help of two of our partners — </strong><a href="https://www.gamebench.net/"><strong>GameBench</strong></a><strong> and </strong><a href="https://aws.amazon.com/"><strong>AWS</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*TjQe8FLFgO9OOMwOdefxPg.png" /></figure><h3>What to Measure?</h3><p>How can we <em>objectively</em> measure the on-device performance of our mobile games?</p><p>The way we approached this was to consider the factors that may cause our players to have a <em>bad</em> experience, and then find representative metrics. For instance:</p><ul><li><strong>Are our players waiting too long to play our game? </strong>App Startup time, scene load time</li><li><strong>Is the game stuttering or juddering? </strong>Frames Per Second (FPS)</li><li><strong>Is our game having an adverse impact on the player’s device?</strong> CPU / Memory Usage</li></ul><p>Of course, identifying the metrics is only half the battle. We also needed to pick values that represent <em>good</em> or <em>bad</em> measurements. This is where things get a little complicated, due to the diverse range of mobile devices that we need to support.</p><p>For instance — ideally, our games run at <strong>60 FPS</strong>. On some low-end devices we accept that they will run at <strong>30 FPS</strong>. But on some <em>mid</em>-range devices we allow <em>parts</em> of the game to run at 30 and parts at 60 FPS. Its a similar story with load times, high-end devices will load the game faster than low-end.</p><p>CPU usage we found to be somewhat volatile depending on device, and mostly useful as an investigative aid. Memory usage thankfully is more clear-cut: iOS will terminate apps that use greater than <strong>70% of RAM</strong> on a device, so we want to know if we are even approaching that threshold.</p><p>By this point we were building up a clearer picture of <em>what</em> we wanted to measure, we just needed a way to actually measure it. Enter GameBench…</p><h3>GameBench</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/0*jyfEkIEjSu8_ihkm.png" /></figure><p>We fairly quickly settled on a tool to help us in this endeavour, from a company called <a href="https://www.gamebench.net/">GameBench</a>.</p><p>GameBench (from the company blurb) are a <strong>world leader in performance management </strong>and their product (the Studio SDK) promised to help us <strong>in the scientific measurement and optimisation </strong>of our games. They are also trusted by some pretty big game studios. Sounds good!</p><p>We make our games using <strong>Unity</strong>, and GameBench provided us a <strong>Unity SDK</strong> which we dropped into our projects. Now, whenever anyone in the studio plays one of our games, a rich set of metrics are sent off to GameBench and presented to us in a Web UI. All without any instrumentation of our own code!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqQZRHV_1x-YGr02nfAzgw.png" /><figcaption>The GameBench web UI</figcaption></figure><p>Another requirement which GameBench ticked off neatly was the ability to define different <em>sections</em> of our games, through their <a href="https://docs.gamebench.net/docs/web-dashboard/the-marker-pane/">Marker</a> system. This means we are able to have different desired performance characteristics in different parts of our game. It also reduces the danger of metrics being smoothed out over the course of a test — e.g. if we had maintained 60FPS for 90% of the test but for 10% it dipped to 20FPS, the overall test may still pass but that wouldn’t be a great user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5alGFMuFIbW6HfuEQjvCQ.png" /><figcaption>CVC Autoplay Build sends metrics to GameBench cloud service</figcaption></figure><p>In order to provide a repeatable process, we concocted an <strong>autoplay build</strong> of our games. The first game we tried this with was our match-three hit <a href="https://www.chromevalleycustoms.com/">Chrome Valley Customs</a> (henceforth referred to as <strong>CVC</strong>). The CVC autoplay build steps through the vehicles, completing various parts of the car restoration process, and plays through match-three levels just like a player would!</p><p>Now we had a way to reliably measure our performance indicators on-device, we needed a way to run performance tests at scale, on a wide range of devices. Enter AWS Device Farm…</p><h3>AWS Device Farm</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*wjbIjIYJ-u8U9g-w" /></figure><p>Imagine thousands of mobile phones in a data-centre, twinkling and beeping away whilst being cooled by giant fans. That’s <a href="https://aws.amazon.com/device-farm/">AWS Device Farm</a>, probably. Regardless, what it does is provide us compute time on a huge range of devices running in a <strong>controlled environment</strong>.</p><p>That last point is an important one: performance tests <em>must</em> be run in a controlled environment. If external factors are able to influence the outcomes, we cannot trust the results. Before we went much further we needed to be assured that AWS Device Farm did indeed fit the bill.</p><p>We performed a simple validation of this fact: we ran the same performance test on the same devices, <strong>twice a day</strong>, for a <strong>whole month</strong>. At the end we collated all of the results, analysed the deviation from the mean and found that the variance was within acceptable bounds.</p><p>By this point, things are shaping up nicely. We had a nightly autoplay build of CVC with the GameBench SDK integrated. We had a way to run this build in parallel across a diverse set of devices. We had our desired metrics being collated and presented back to us in the GameBench web portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6TEQjEaqiJmGADpCNGLApw.png" /></figure><p>But there was still something missing. To make this <em>truly</em> useful we needed to expose this information to our developers. It wasn’t enough to simply collect this information, we needed to be <em>told</em> if we had breached our performance thresholds. We needed <strong>alerts</strong>.</p><p>GameBench does provide <a href="https://docs.gamebench.net/docs/onboarding-guide/performance-alerts/">Alerts</a>, which would work just fine for smaller projects. But we had dozens of tests running each night, on several devices each with different performance criteria. A failure would trigger dozens of Slack messages which would, we know from experience, lead to <strong>alert fatigue</strong>. Furthermore, we now had Device Farm logs and screenshots to aid in any investigation. We felt we needed something to orchestrate all of this and present all of the information in a single place…Enter AWS Step Functions</p><h3>AWS Step Functions</h3><p>In order to explain what <a href="https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html">AWS Step Functions</a> <em>is</em>, it is useful to show what we wanted to <em>do</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVxCaNelGRz9z9hdAKQ1xg.png" /></figure><p>The diagram shows the <strong>workflow</strong> that we wanted our performance tests to take. Each node in the diagram represents a state in the workflow, and the adjoining lines are transitions between states. What we have here is an example of a <strong>State Machine</strong>.</p><p>That, essentially, is AWS Step Functions: a way to define a State Machine (or a workflow) in AWS. Each step in the workflow can either consist of a logical function (e.g. a Choice or a Wait) or enact some sort of <strong>Task</strong>. Tasks are where things get interesting. AWS Step Functions are tightly integrated with many other AWS services, so a Task can easily trigger some other action.</p><p>In our case, we configured a Task which would kick off a <strong>Device Farm</strong> run. Within each run, we play the <em>performance build </em>of <strong>CVC</strong> (the autoplay build with the GameBench SDK integrated) on a pre-configured collection of different devices (called a <strong>Device Pool</strong>). We then configured another that would loop until all of the runs had completed.</p><p>We now have a way to orchestrate our nightly performance tests, to trigger them on a range of devices and watch until they all complete. But what then? How do we extract the vital statistics and broadcast the results?</p><p>Well, another nicety of AWS Step Functions is that each of the steps can also invoke a Lambda function (or indeed an ECS task). Once all of our Device Farm runs are complete, we trigger a Lambda function which:</p><ul><li>Identifies the GameBench session ID of each run.</li><li>Queries the GameBench API to get the performance metrics for that session</li><li>Compares the results with our pre-defined thresholds</li><li>Collates all of the information into a nice Slack message</li></ul><p>Ultimately, it is the Slack message which is the output of this entire fandango. Here is an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2jlUD2JBRSmQz-QHrig1Bw.png" /></figure><p>As you might expect, it shows clearly which devices have succeeded and which have failed. But we can also see in which <strong>section</strong> of the game the failure occurred. We also provide links to a wealth of information that can be used in investigating <em>why</em> they failed: the GameBench web UI for the specific session, Device Farm logs, application logs, even a link to a <strong>screen recording</strong> of the failed run!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*9p4tCq9y7dQRn6-HQryiYQ.png" /></figure><h3>Conclusion</h3><p>We now have a way of objectively knowing if our games are suffering from poor performance. We run these tests nightly against our development builds, so we can be told if any changes we’ve made during the day have impacted performance.</p><p>This is incredibly useful — we catch issues on a regular basis: from unsuspected memory creeps to unwanted dips in FPS caused by new features.</p><p>This means we can be assured that the products we put out will continue to please our players, and we won’t fall afoul of any performance penalties from our platform partners.</p><p><em>But</em> we’ve also seen a change in our developers. They are now able to take a much more analytical approach to performance; we have performance champions within our games teams, we now talk of performance budgets, and in general device performance is shifted more to the forefronts of our collective minds.</p><p>Thanks for reading!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Est9Y1cU7VcmcuAKohjUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*giRQ6kibl-JdmWhGavS6sw.png" /><figcaption>Donna and Uncle Hank approve</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb6045c2b1b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/performance-testing-at-space-ape-games-eb6045c2b1b2">Performance Testing at Space Ape Games</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running AWS Fargate Services on Graviton2]]></title>
            <link>https://medium.com/@louismccormack/running-aws-fargate-services-on-graviton2-3de260b270f6?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3de260b270f6</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[fargate]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Wed, 23 Nov 2022 20:43:19 GMT</pubDate>
            <atom:updated>2022-11-23T20:43:19.727Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS’ Graviton2 chips are part of the cloud giant’s custom processor range. They are based on the ARM architecture of CPUs. Entire articles, books even, could be written about ARM: its meteoric rise and its value in an energy-conscious, recession-facing world. But all we <em>really</em> need to know is that workloads running on AWS Graviton2 exhibit <strong><em>up to 40% better price performance than Intel-based instances.</em></strong></p><p>We can already choose to run many of AWS’ services on Graviton2. Indeed it is almost a no-brainer to switch your OpenSearch or ElastiCache clusters to do just that, and save <a href="https://aws.amazon.com/opensearch-service/pricing/">about 10%</a> in the process.</p><p>AWS Fargate, the serverless(-ish) container platform, has supported running Graviton2 containers since 2021.</p><p>However the mere thought of compiling and building applications to run on Graviton2 is enough to make one grimace. <em>Imagine</em> the hoops that must be jumped through to update our container images, build chains and compiler flags to run on ARM…</p><p>The truth though, as I recently found out, is somewhat different. The tooling and support for building ARM-based images has matured such that there are vanishingly few excuses not to try it out.</p><p>This article aims to explain the steps required to get <em>your</em> application running on Fargate Graviton2…</p><h3>The Task Definition</h3><p>The unit of deployment on AWS Fargate is the <em>Task</em>.</p><p>A <em>Task</em> can comprise several containers, all of which share a network stack.</p><p>The containers, along with many other runtime characteristics, are defined in a <em>Task Definition</em>.</p><p>In order to tell Fargate we wish to run on Graviton2, all we need to do is add this snippet to the <em>Task Definition</em>:</p><pre>&quot;runtimePlatform&quot;: { <br>  &quot;cpuArchitecture&quot;: &quot;ARM64&quot; <br>}</pre><p>So that’s it!? Well, no…for what a boring article that would make. In fact, before we even think about flipping that switch, we need to make sure that all of the container images within the <em>Task Definition</em> are ARM-compatible.</p><p>This means they must be built atop OSes or with binaries that were themselves compiled with the ARM instruction set in mind.</p><p>Now admittedly, this is starting to sound like a pain in the proverbial rear. This might be the part where we back away whistling and find something else to do. But read on! People cleverer than us (well, certainly me) have trodden this path before, and left a wonderful toolchain in their trail…</p><h3>The Container Images</h3><p>The first thing to understand on this voyage of ARM discovery is that:</p><blockquote>One does not just build ARM container images, one builds multi-architecture container images.</blockquote><p>That is, we build 2 images: one for ARM and one for x86. Then we compile them both into an <em>image manifest</em>. The manifest acts like a pointer to the correct image, which is de-referenced at runtime, depending on the underlying compute architecture.</p><p>And the best part is that the good folks at Docker Inc. have provided a wonderful tool that does all of this for us: docker buildx .</p><p>To the uninitiated (a group which included myself only last week), docker buildxis indistinguishable from magic. If that evokes a big shrug in you, then you can probably skip the next section. For the rest of us, lets take a look at what docker buildx actually does…</p><h3>Docker Buildx</h3><p>Traditional Docker builds happen via the ol’ tried-and-tested docker build command. The <em>build context</em> (all of the required files) is dutifully shipped off to a Docker daemon running somewhere on your machine, some things happen, and eventually you end up with a Docker image.</p><p>Newer versions of the Docker Engine ship with Buildkit, which provides a number of tantalising improvements over the standard Docker build. <em>One</em> of these improvements is the ability to build images for multiple architectures, and compile them into a single image manifest.</p><p>docker buildxis the front-end for Buildkit. It is invoked in much the same way as the traditional build commands. But the builds themselves actually happen <em>inside a container </em>running (by default) a moby/buildkit image on your local machine.</p><p>The commands required to build a multi-arch image are as follows:</p><pre>$ docker buildx create --use<br>$ docker buildx build --platform linux/amd64,linux/arm64 --push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-multi-arch-image .</pre><p>Anybody accustomed to ordinary Docker builds may be furrowing their brow at this point. We have hit a small wrinkle in the intuitiveness of docker buildx - it seems it is not an <em>entirely</em> like-for-like replacement for the traditional build tool.</p><p>But thats ok, we just need to learn a few new keystrokes.</p><p>The first line above creates a <em>builder. </em>Without this, buildx will default to using the standard Docker daemon to build the image, which does not understand how to build for multiple architectures at once<em>.</em></p><p>(Incidentally, the <em>builder</em> is just a docker container, which you can see with docker ps -a )</p><p>The second line above instructs that <em>builder</em> to actually build something, in this case an image called my-multi-arch-image. It will use the Docker context in the current directory, and it will --push the image to a remote ECR repository.</p><p>Here we stumble upon a second wrinkle in the intuitiveness of this command: if you run docker images you <em>won&#39;t</em> see my-multi-arch-image. Remember, we didn&#39;t use the local Docker daemon to build this image, we used the <em>builder </em>inside a container. The local daemon has no knowledge of it.</p><p>Regardless, the multi-arch image has now been built, and pushed to ECR. We can inspect it using docker buildx imagetools:</p><pre>Name:      123456789.dkr.ecr.us-east-1.amazonaws.com/my-multi-arch-image<br>MediaType: application/vnd.docker.distribution.manifest.list.v2+json<br>Digest:    sha256:3f7117eef97a8fbeeeec<br><br>Manifests:<br>  Name:      123456789.dkr.ecr.us-east-1.amazonaws.com/my-multi-arch-image@sha256:21b37e49123bc12f5eee<br>  MediaType: application/vnd.docker.distribution.manifest.v2+json<br>  Platform:  linux/amd64<br><br>  Name:      123456789.dkr.ecr.us-east-1.amazonaws.com/my-multi-arch-image@sha256:1e8e0923cf679eeeee<br>  MediaType: application/vnd.docker.distribution.manifest.v2+json<br>  Platform:  linux/arm64</pre><p>Wonderful, we have indeed crafted a multi-arch image manifest. Now we need just reference that image in our Task Definition, and Fargate will handle the rest!</p><h3>A Note on Compiling</h3><p>Of course, life is never <em>really</em> that simple.</p><p>You probably <em>will</em> have to compile your application to run on ARM64. Thankfully AWS offer <a href="https://github.com/aws/aws-graviton-getting-started">some official guidance</a>. It provides examples for compiling a wide range of languages to run on Graviton2.</p><p>Enjoy being <em>up to</em> 40% more efficient!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3de260b270f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bursting To Lambda]]></title>
            <link>https://medium.com/spaceapetech/when-autoscaling-just-isnt-enough-60666554b322?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/60666554b322</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[autoscaling]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Fri, 07 May 2021 14:23:15 GMT</pubDate>
            <atom:updated>2021-05-07T15:28:05.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsXniNVLDL6UdJKUbrzB7w.jpeg" /></figure><p>The murky art of capacity planning has undergone something of a quiet revolution over the last decade or so. With AutoScaling, lead time — the time it takes for new capacity to be in-place, serving traffic — has gone from weeks and months to minutes. Latterly, with Serverless, it has been almost eradicated.</p><p>In fact that forms a large part of the Serverless promise: assign capacity planning to the history books by drawing on a near-infinite pool of compute.</p><p>So why isn’t everybody scrambling to migrate to Serverless?</p><p>Firstly, a lot of people <em>are</em>.</p><p>Also there are good reasons why others are not. Certain high-performance, latency-sensitive workloads are still not a good fit (like online multiplayer games).</p><p>Primarily though, at a certain scale, there is a cost argument. For high traffic systems the raw economics may rule out Serverless as an option over more traditional compute.</p><p>For instance, this excellent <a href="https://www.bbva.com/en/economics-of-serverless/">recent white paper</a> shows that there is a clear break-even point where Lambda usage becomes more expensive than EC2. The point varies for different applications and workloads but, once reached, the extra cost of using Lambda quickly starts to rise.</p><p><strong><em>But. What if we could have the best of both worlds?</em></strong></p><p>A (relatively) slow-scaling tier of (relatively) cheap compute lumbering away in the background, supplemented by an ever-ready tier of Lambda functions to soak up unexpected spikes in traffic. What if we could… <em>burst </em>to Lambda?</p><p>You’re probably reeling off in your head all the reasons why this is a bad idea. But let us offer one on the side of viability: Last year, container image support was announced for Lambdas. This raises an interesting possibility — is it feasible to deploy the <em>exact</em> <em>same</em> artefact to ECS and Lambda, and have them running alongside one another?</p><p>The answer is yes, in certain circumstances, you can. Furthermore you can, in certain circumstances, share traffic between the two.</p><p>Intrigued? Please read on…</p><p>What is described here is a proof of concept, and the concept being proofed has the following high-level properties:</p><ul><li>The application under scale is a simple Golang HTTP server fronted by an Application Load Balancer.</li><li>The slow-scaling tier takes the form of containers running on AWS Fargate, configured to AutoScale.</li><li>The bursting tier takes the form of a Lambda ALB backend, configured as a separate target group.</li></ul><p>Clearly there’s a lot of detail to be bedevilled.</p><p>All of the code referenced here can be found in the <a href="https://github.com/spaceapegames/lambda-burst-poc">accompanying Github repo.</a></p><h3>The Application</h3><p>The application exposes a single endpoint: /doThing. DoThing does nothing more than calculate all the prime numbers between 0 and 10000, and return the result. A sisyphean task for sure, but that is in fact the least interesting thing about the application.</p><p>The <em>most</em> interesting thing about the application is that it is a <strong>single container image that can be deployed to either Fargate or Lambda</strong>.</p><p>In order for this to work it must respond in a similar fashion, to both ordinary HTTP requests and the HTTP-requests-as-JSON proxy Events that Lambda deals in.</p><p>This is achieved through use of the excellent <a href="https://github.com/apex/gateway">apex-gateway</a> library. This library converts Lambda proxy Events into ordinary Go HTTP requests; making it possible to use standard Golang HTTP constructs (e.g. HttpHandlers, Routers etc.) to serve Lambda proxy requests.</p><p>What it means for our application is that we can respond to HTTP requests <em>or</em> Lambda proxy events at the flick of a feature-flag:</p><pre>if s.lambdaMode {<br>  log.Println(“running in Lambda mode”)<br>  gateway.ListenAndServe(“:8000”, s.Router)<br>} else {<br>  log.Printf(“running in http mode on port %d”, s.port)<br>  http.ListenAndServe(fmt.Sprintf(“:%d”, s.port), s.Router)<br>}</pre><p>There are some further quirks to ensuring the same container image can be run on both tiers. Full details can be found in the <a href="https://github.com/spaceapegames/lambda-burst-poc">README</a>, but suffice it to say that the image contains a single binary (the Go application) that is invoked slightly differently by Fargate and by Lambda, by way of an altered entrypoint.</p><h3>The Infrastructure</h3><p>The AWS infrastructure we’ll use to run this PoC is refreshingly simple. It is simply an ALB with two listeners configured:</p><ul><li>Port 8080, forwarding to a target group managed by the Fargate tier.</li><li>Port 8081, forwarding to a Lambda backend.</li></ul><h3>AutoScaling the Slow Tier</h3><p>For reasons that will become clear, the metric we’ll use to AutoScale the Fargate tier is the <strong><em>ALB request rate per target</em></strong>. We’ll set a Target Tracking Policy such that Fargate tasks will be added or removed in order to maintain a consistent rate of requests against each of them.</p><p>But how do we know what that rate is?</p><p>To help us find out, we fired up the Space Ape load-testing systems.</p><p>Then donned lab-coats and specs before undertaking the scientific endeavour of running increasingly onerous load-tests, until one broke.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JY3w1lmVipLowX4NKzmNOw.png" /></figure><p>This graph shows a successful load-test at 100 requests-per-second (RPS). The blue line represents HTTP 200 Responses from the ALB.</p><p>If we dial the RPS up to 120, this happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DLSIR5rprmDuftEgAFg0fg.png" /></figure><p>In this graph, the green line shows HTTP 500 Responses. Bad news.</p><p>So, someone proffered, each Fargate task can comfortably handle 100rps? We nodded, seems about right.</p><p>Experiment concluded: a<strong> <em>Target Tracking Policy of 100 requests per ALB target</em></strong> would be appropriate.</p><p>With the AutoScaling Target Tracking policy in place, we re-ran the same test but with twice the load (200 RPS). This graph shows the response codes, alongside the number of Fargate tasks (on a logarithmic scale):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EtrBfNTSDePs9uTcbAFi4A.png" /></figure><p>We can see that AutoScaling did indeed kick in — if a little over-zealously — and decide to scale out 5(!) new tasks. This was more than enough to handle the 200 RPS.</p><p>So that’s the end of it — AutoScaling works, lets move on?</p><p>Well, unfortunately, no.</p><p>AutoScaling works only if the rate of increase is sufficiently low. Or, put another way: if the duration over which the load is added is long enough for AutoScaling to a.) realise it needs to add capacity and b.) actually add the capacity.</p><p>To illustrate this, we’ll run the same test, but halve the ramp up time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WDVN3_sImLXqzOyu8gPxRg.png" /></figure><p>Here we can see that, in fact, AutoScaling does not work.</p><p>The green line shows HTTP 500 responses, of which there were a fair number, until AutoScaling finally got its act together.</p><p>One way around this problem is to pre-scale: add capacity in advance of a known traffic increase. At Space Ape we do precisely that when an event is about to begin in one of our mobile games.</p><p>This of course presupposes that you know in advance. Some workloads do not have this advantage; think of a link which suddenly goes viral, or a TV advert driving a call to action. Systems with workloads such as this would likely have to either run consistently over-scaled, accept some errors, or have client retry mechanisms. None of which sound particularly attractive.</p><h3>Lambda to the Rescue!</h3><p>And now we come to the <em>coup de grace</em>. Can we use Lambda to soak up spikes in load, to give AutoScaling a chance to catch up?</p><p>The answer is probably, yes. But there are many ways to skin this cat.</p><p>Here is what we did:</p><p>Recall that we have 2 backends, one Fargate and one Lambda, both using the same container image.</p><p>Recall also that we believe each Fargate task can handle 100 RPS.</p><p>The aim then is take the portion of load which is above 100 RPS <em>per target</em>, and somehow send it to the Lambda backend.</p><p>To this end we added a rate-limiter to the application. It is only enabled on the Fargate backend (through an environment variable). Most requests will pass the rate limit and be served by the Fargate application.</p><p>However, if the rate is seen to be &gt;100rps, it will issue an HTTP 301 redirect instead. The <em>target</em> of the redirect will be the Lambda backend. The client merrily follows the redirect, and the request is served.</p><p>There are a number of things to be said about this approach:</p><ul><li>This is a form of <em>backpressure</em>. Ordinarily backpressure would result in the client request being rejected, or told to try again in a moment. But here we have still serviced the request, by sending it to our Lambda pool.</li><li>In this approach <em>all</em> requests are still sent to the Fargate tier, even those which are above &gt;100rps. It works in this contrived example because a 301 redirect uses significantly fewer CPU cycles than calculating prime numbers. A better approach may be to have some out-of-band process which tinkers with the Target Group weighting when it sees a spike in traffic.</li></ul><h3>The Results</h3><p>Ok. But does it actually work?</p><p>Well, here is the same test as above, with the Lambda redirection turned on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1raxbfDI914Od4hmS5HRkw.png" /></figure><ul><li>The blue line shows HTTP 200 responses from the <em>Fargate</em> backend</li><li>The orange line shows HTTP 200 responses from the <em>Lambda</em> backend.</li><li>The green line is the number of Fargate tasks.</li></ul><p>We can clearly see the Lambda tier taking on a high number of requests as load ramps up. Once AutoScaling arrives to the party it adds 5 tasks(see the bump in the green line), after which 100% of requests are once more served by the Fargate tier, and the Lambdas retreat.</p><p>So, yes, we successfully bursted to the Lambda tier! We provided a buffer until AutoScaling caught up, and no errors were returned to the clients.</p><h3>Summary</h3><p>We have proven that is indeed possible to a.) build a single artefact for use with both Lambda and ECS and b.) serve most requests from a cheaper ECS tier, only switching to Lambda when capacity on that tier is insufficient.</p><p>Obviously what we’ve proven is a specific solution, to a very contrived example, borne out of a hypothetical question. But we like to think that there may be some teams out there, in circumstances as specific as these, with whom it may resonate. If so, we’d love to hear from you!</p><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60666554b322" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/when-autoscaling-just-isnt-enough-60666554b322">Bursting To Lambda</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom AWS Config Rules]]></title>
            <link>https://medium.com/spaceapetech/custom-aws-config-rules-20d7995561a8?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/20d7995561a8</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[infrastucture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-config]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Thu, 13 Jun 2019 13:00:59 GMT</pubDate>
            <atom:updated>2019-06-13T13:00:59.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpr2Z5MajaV866BFopE5uA.png" /><figcaption>Photo by <a href="https://unsplash.com/@jessicatootoo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jessica To’oto’o</a> on <a href="https://unsplash.com/search/photos/rules?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><em>This article will attempt to explain how to create a custom </em><a href="https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config.html"><em>AWS Config Rule</em></a><em>. A working example is provided, using SAM and a Go-flavoured Lambda function.</em></p><p>All organisations, regardless of size, will have some sort of Policy. In large enterprises it will be formalised, codified and audited. In small startups it’s more likely to be yelled across the office. But it’s the same thing — a set of rules or idioms that must be followed. Often, Policy is associated with security, but that needn’t always be the case: a naming standard is an example of Policy.</p><p>Enforcement of Policy in AWS is achieved through correct configuration of resources (e.g. security groups or bucket policies). Infrastructure as Code can help here (think Terraform or CloudFormation) along with good practices such as code reviews or deploy-time checks.</p><p>However ongoing adherence, or compliance, to Policy is another thing entirely.</p><p>Let’s take an example: you have made the sensible decision not to expose SSH or RDP ports to the internet at large and your well-drilled CloudFormation templates reflect this. Consider the hapless engineer who temporarily opens up port 22 so they can SSH in from home and who, being hapless, forgets to revert the configuration afterwards. Or worse, some malignant manages to get their hands on a set of access keys and immediately cracks open an RDP port.</p><p>This is where AWS Config Rules come in. They provide an AWS-native way to perform <em>dynamic</em> compliance checking. In the above example they could immediately inform us that a security group is in breach of Policy.</p><p>There are many pre-rolled <a href="https://docs.aws.amazon.com/config/latest/developerguide/managed-rules-by-aws-config.html">Managed Rules</a> to cover such no-brainers as ‘vpc-flow-logs-enabled’ or ‘access-keys-rotated’. These are extremely useful. But sometimes just a little <em>too</em> rigid. You might, for instance, wish to exclude certain resources from being reported (again taking the example above, you might allow port 22 to be open on certain bastion hosts) or just require a policy more specific to your business.</p><p>Happily, Config Rule checks, even managed ones, are just Lambda functions. It’s entirely possible to write a custom Lambda function and wire it up as a Custom Config Rule. The rest of this article is dedicated to explaining how to do just that.</p><h3>Custom Config Rules</h3><p>Custom AWS Config Rules come in two flavours:</p><ul><li>Periodic: a check is run on a user-defined schedule to confirm compliance of existing resources. It probably involves enumerating some list of resources and checking each in turn.</li><li>Triggered: a check is run whenever a resource of the specified <em>type </em>is created, or undergoes a configuration change.</li></ul><p>We’ll focus on the Triggered checks here as they are more powerful and, frankly, more interesting.</p><p>The basic premise of a custom check is that it must make an evaluation as to whether a resource conforms to some policy. The result must be one of:</p><ul><li>COMPLIANT</li><li>NON_COMPLIANT</li><li>NON_APPLICABLE</li></ul><p>The function must then make a call to the AWS API <a href="https://docs.aws.amazon.com/config/latest/APIReference/API_PutEvaluations.html">PutEvaluation</a> endpoint with that result.</p><h3>A Contrived Example</h3><p>Let’s say that we have a Policy:</p><p><em>All Production Database EC2 Instances must have EBS Optimization enabled.</em></p><p>Yes it’s a little contrived, maybe ill-advised (why run your own database?), but still altogether plausible.</p><p>We can translate this into cloud-speak thusly:</p><p><em>If an EC2 instance has the tags ‘ENVIRONMENT=Production’ and ‘SERVICE=database’ then it must have EBS optimization enabled.</em></p><p>Naturally this assumes that the EC2 tags have been set correctly in the first place but, hey, we have to take <em>some</em> responsibility. (Or not: we could conceivably have a different Config rule that checks for the existence of these tags.</p><p><strong><em>Anatomy of a Triggered event</em></strong></p><p>Before we can passably attempt to explain any code, we need to delve into the data structure that AWS Config uses to communicate a triggered Event. This data structure, or ‘Event’, is passed into a specified Lambda handler whenever a change happens to<em> any resource of the specified type</em>. In our case that means <em>any</em> change to <em>any</em> EC2 instance.</p><p>Full documentation can be found <a href="https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_develop-rules_example-events.html">here</a>, but essentially an event looks like this:</p><pre>{ <br>    &quot;invokingEvent&quot;: &quot;...&quot;,<br>    &quot;ruleParameters&quot;: &quot;{\&quot;myParameterKey\&quot;:\&quot;myParameterValue\&quot;}&quot;,<br>    &quot;resultToken&quot;: &quot;myResultToken&quot;,<br>    &quot;eventLeftScope&quot;: false,<br>    &quot;executionRoleArn&quot;: &quot;arn:aws:iam::123456789012:role/config-role&quot;,<br>    &quot;configRuleArn&quot;: &quot;arn:aws:config:us-east-2:123456789012:config-rule/config-rule-0123456&quot;,<br>    &quot;configRuleName&quot;: &quot;change-triggered-config-rule&quot;,<br>    &quot;configRuleId&quot;: &quot;config-rule-0123456&quot;,<br>    &quot;accountId&quot;: &quot;123456789012&quot;,<br>    &quot;version&quot;: &quot;1.0&quot;<br>}</pre><p>Some points:</p><ul><li>We’ll need resultToken when we eventually make the call back to PutEvaluation.</li><li>Of greater importance here is eventLeftScope. Put simply, if this is true then we should immediately call PutEvaluation with a value of NOT_APPLICABLE. The most common scenario for eventLeftScope is when a resource is deleted.</li><li>However the real meat on the bones is to be found in the ellipsized invokingEvent. This contains details of the configuration change that triggered this Event. It takes the form of a JSON string representing a ‘Configuration Item’. The docs <a href="https://docs.aws.amazon.com/config/latest/developerguide/config-item-table.html">here</a> explain what that entails: a whole load of information about the underlying resource such as time-of-capture of event, tags, relationships with other resources, and a snapshot of the resource’s current configuration.</li></ul><p>With that information suitably digested, lets move on to actually writing a custom check.</p><h3>The Code</h3><p>We will write a Lambda handler to process a triggered Config Event, and then deploy it using the SAM framework.</p><blockquote>All of the code described below, along with the SAM template and some deployment helpers, is to be found <a href="https://github.com/spaceapegames/custom-config-rules-tutorial">here</a>.</blockquote><p>First off, wouldn’t it be good to get our hands on some sample data to gain some degree of confidence in our function before we release it into the wild? Look no further than the AWS CLI. The <a href="https://docs.aws.amazon.com/cli/latest/reference/configservice/get-resource-config-history.html">get-resource-config-history</a> command will, with a little tinkering, furnish us with a <a href="https://docs.aws.amazon.com/config/latest/developerguide/config-item-table.html">Configuration Item</a> that matches what AWS Config will send to our Lambda function.</p><p>The following command will, with a sprinkling of jq , give us something to work with (obviously replace the instance-id with one that is real):</p><pre>$ aws configservice get-resource-config-history \<br> --resource-type &quot;AWS::EC2::Instance&quot; \<br> --resource-id i-111111111111 \<br> --max-items 1 | \<br>   jq &#39;.configurationItems | .[] | {&quot;configurationItem&quot;: .}&#39;</pre><p>The resulting JSON still requires some doctoring in order to match what we will see in the wild:</p><ul><li>The timestamps configurationItemCaptureTime and resourceCreationTime must be converted to ISO 8601 human-readable time formats.</li><li>The configurationStateId must be converted to a number (just remove the quotes)</li><li><strong>Most importantly</strong>, the configuration field must be converted from a JSON string to an actual JSON object.</li></ul><p>Failure to carry out any of these tweaks will result in your function imploding the moment it <em>sets foot</em> in the wild.</p><p>In the code example, we have saved this patched-up JSON in <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/fixtures/sampleConfigItem.json">fixtures/sampleConfigItem.json</a>. We will use this shortly to test our function.</p><p><strong><em>Data Models</em></strong></p><p>The Event that is generated by a triggered Config change is passed into our Lambda handler as JSON. We can take advantage of a few truisms to avoid creating elaborate data structures in which to house it:</p><ul><li>The Event itself is an instantiation of a <a href="https://godoc.org/github.com/aws/aws-lambda-go/events#ConfigEvent">ConfigEvent</a>. As long as our Lambda function accepts a ConfigEvent as a parameter, it will be automatically un-marshalled.</li><li>Unfortunately we don’t have that luxury with the InvokingEvent, so we are in fact forced to create <em>one</em> elaborate data structure. See <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/model/invoking_event.go">here</a>.</li><li>We are told <a href="https://docs.aws.amazon.com/config/latest/developerguide/config-item-table.html">here</a> that the configuration field of the Configuration Item is ‘<em>Information returned through a call to the Describe or List API of the resource</em>’ which means it should nicely un-marshall into an object of the underlying type* (in our case an <a href="https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/#Instance">ec2.Instance</a>). This can be seen in the <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/evaluators/database_ebs.go#L31">code example</a>:</li></ul><pre>// unmarshal the ConfigurationItem<br>  var instance ec2.Instance<br>  err = json.Unmarshal(<br>          invokingEvent.ConfigurationItem.Configuration, <br>          &amp;instance)</pre><p>After this we have an instance object neatly populated with the attributes of the resource that has changed.</p><p><em>(*this should be taken with a pinch of salt. We have found, for example, that the </em><em>configuration field of EC2 Security Groups does </em>not<em> correlate with what is returned by an API call to </em><a href="https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html"><em>DescribeSecurityGroups</em></a><em>)</em></p><p><strong><em>Evaluation Logic</em></strong></p><p>Full code listing for the evaluation of the check can be found <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/evaluators/database_ebs.go">here</a> . It’s recommended to have a peruse, but the most interesting part is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mq0p_usLDNVYuzEalV_yA.png" /></figure><p>Here we take advantage of the fact that the ConfigurationItem object of the invokingEvent contains a list of the instance’s EC2 Tags, in order to filter for ‘Production’ and ‘database’. If we find a match then we use our instance object to check that EBS optimization has been turned on. The return value is a struct that contains all the information required to complete a PutEvaluation call.</p><p>In the same file we provide a CompleteEvaluation function which our Lambda handler will use to actually send the evaluation through to AWS Config.</p><p>Remember the sample data we generated above? <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/evaluators/database_ebs_test.go">Here</a> we can see it in action in a unit test that caters for three different outcomes: compliant, non-compliant and non-applicable.</p><p><strong><em>The Handler</em></strong></p><p>Our <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/handlers/databaseTerminationProtection/main.go">Lambda handler function</a> is now very simple. It takes the ConfigEvent, sends to our evaluator, and then completes the evaluation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e67092e141dd93974e5a831838c22be/href">https://medium.com/media/9e67092e141dd93974e5a831838c22be/href</a></iframe><p><strong><em>Deploying</em></strong></p><p>We can easily deploy our custom config rule using SAM. The <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/template.yml">template looks like this</a>:</p><pre><strong>AWSTemplateFormatVersion </strong>: &#39;2010-09-09&#39;<br><strong>Transform</strong>: AWS::Serverless-2016-10-31<br><br><strong>Description</strong>: Custom Config Rule<br><br><strong>Resources</strong>:<br>  <br>  <strong>DatabaseCheckFunction</strong>:<br>    <strong>Type</strong>: &#39;AWS::Serverless::Function&#39;<br>    <strong>Properties</strong>:<br>      <strong>Runtime</strong>: go1.x<br>      <strong>CodeUri</strong>: ./main.zip<br>      <strong>Handler</strong>: bin/databaseTerminationProtection<br>      <strong>Policies</strong>:<br>      - <strong>Statement</strong>:<br>        - <strong>Sid</strong>: ConfigPutEvaluation<br>          <strong>Effect</strong>: Allow<br>          <strong>Action</strong>:<br>          - config:PutEvaluations<br>          <strong>Resource</strong>: &quot;*&quot; <em># No resource level permissions :(<br><br>  </em><strong>DatabaseCheckRule</strong>:<br>    <strong>Type</strong>: AWS::Config::ConfigRule<br>    <strong>DependsOn</strong>: DatabaseCheckConfigRole<br>    <strong>Properties</strong>:<br>      <strong>ConfigRuleName</strong>: &quot;database-ebs-optimization&quot;<br>      <strong>Description</strong>: &quot;Checks for production dbs without EBS optimization&quot;<br>      <strong>MaximumExecutionFrequency</strong>: One_Hour<br>      <strong>Scope</strong>:<br>        <strong>ComplianceResourceTypes</strong>:<br>        - &quot;AWS::EC2::Instance&quot;<br>      <strong>Source</strong>:<br>        <strong>Owner</strong>: CUSTOM_LAMBDA<br>        <strong>SourceDetails</strong>:<br>        - <strong>EventSource</strong>: aws.config<br>          <strong>MessageType</strong>: ConfigurationItemChangeNotification<br>        <strong>SourceIdentifier</strong>:<br>          !GetAtt<br>        - DatabaseCheckFunction<br>        - Arn<br><br>  <strong>DatabaseCheckConfigRole</strong>:<br>    <strong>Type</strong>: AWS::Lambda::Permission<br>    <strong>Properties</strong>:<br>      <strong>FunctionName</strong>:<br>        !GetAtt<br>      - DatabaseCheckFunction<br>      - Arn<br>      <strong>Action</strong>: lambda:InvokeFunction<br>      <strong>Principal</strong>: config.amazonaws.com</pre><p>The template creates:</p><ul><li>A Lambda function to run our handler, with permissions to make a call to PutEvaluations. We also grant permission for the config.amazonaws.com service to invoke the function.</li><li>A ConfigRule that acts as a trigger to the function. Note that we list “AWS::EC2::Instance” as a ComplianceResourceType. This means that any configuration changes to any EC2 instances will trigger an invocation of the Lambda function.</li></ul><p>Exactly how to build and deploy this application is a little out of scope for this article (if you’re interested, see <a href="https://medium.com/spaceapetech/developing-applications-with-aws-sam-295c320d563c">here</a>). However a <a href="https://github.com/spaceapegames/custom-config-rules-tutorial/blob/master/Makefile">Makefile</a> is provided, so if you are keen you can just run:</p><p>S3_BUCKET=<em>&lt;an-s3-bucket-you-own&gt;</em> <strong>make deploy</strong></p><p>Once deployed, AWS Config will trigger a run-through of the check for all current EC2 instances. Subsequent checks will only be triggered on configuration changes.</p><p>To see the results, navigate to the Config page in the AWS Console; then choose Rules -&gt; database-ebs-optimization. Of course, it’s unlikely that you’ll have any non-compliant resources.</p><p>If you <em>were</em> so inclined, you could add an ENVIRONMENT=Production and SERVICE=database tag to an EC2 instance <em>without</em> EBS optimization, then watch the magic happen…(disclaimer: the magic may take a few minutes to happen)</p><p><strong><em>Summary</em></strong></p><p>Hopefully this has given a glimpse of the power of AWS Config Rules in showing that they can immediately notify us of any changes to our estate that have resulted in a breach of Policy. Properly configured they provide a valuable ally in the fight against malicious actors and witless engineers.</p><p>Also, we’ve barely scratched the surface here. Happy Rule-Making.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20d7995561a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/custom-aws-config-rules-20d7995561a8">Custom AWS Config Rules</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless For Devops Teams]]></title>
            <link>https://medium.com/spaceapetech/serverless-for-devops-teams-39202dba2ce?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/39202dba2ce</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Tue, 26 Feb 2019 18:12:06 GMT</pubDate>
            <atom:updated>2019-02-26T18:12:06.627Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOYUFv04eI6pdz0xEvggQQ.png" /><figcaption>Photo by <a href="https://unsplash.com/photos/1hE01VmbIOg?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Hanny Naibaho</a> on <a href="https://unsplash.com/search/photos/welding?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Serverless as a technology is no longer in its infancy. It has matured to a point where we can at least say it is in its lanky teenage years, yet to enter the full flushes of adulthood, but with all its vitals shaping up nicely.</p><p>We in the DevOps team here at Space Ape are avid Serverless fans. This post aims to explore the factors that have made it so appealing to us, from an operations standpoint.</p><p>Serverless has enjoyed a rapid rise to fame. AWS Lambda was launched in late 2014. There was a collective intake of breath from the industry before we realised that, if nothing else, here was something new to argue about.</p><p>First we argued about the name, “but there are <em>still</em> <em>servers</em>!” some wailed. Then some declared Serverless would usher in the demise of Ops teams (remember<em> #noops?</em>). Later, battle lines were drawn between budding futurists predicting an all-in on Serverless, and scornful adversaries convinced that a Kubernetes cluster was all you’d ever need.</p><p>Perversely, those very people whose jobs were purportedly threatened — Ops teams — have been some of the first to embrace Serverless, warts ’n’ all. For warts there most definitely still are: the cold-start times remain a barrier to adoption for many; and monitoring, along with the build and deploy tooling, is somewhat clunky and ill-defined.</p><h3>Why Ops Teams?</h3><p>Every Ops person will know an example of <em>that</em> server, with an uptime of several years, crontab-of-several-pages, documentation-of-none, that nobody dare reboot. In the old days it may have been tucked under somebody’s desk, in the cloud era it may be a woefully undersized instance that has fortuitously avoided termination all these years.</p><p>With Serverless, we can replace those hack-job cron-jobs with scheduled, documented and tested Lambda functions! Many of the tasks important to us are not latency-sensitive, so we can handle a <em>little</em> cold start-up time.</p><p>Better than this, we can even begin to extend the functionality of our cloud providers! Instead of judiciously choosing who to appease, the providers (AWS in particular) have cleverly handed us some tools and pointed at the shed. Do it yerself, they are practically saying. We, in turn, are becoming adept at welding together the various conduits on offer.</p><p>The factor that makes this possible is the ever-growing cast of AWS services that can be used to trigger a Lambda function. We’ve long had S3, SNS and API Gateway triggers; with Kinesis, DynamoDB, SQS and more added along the way. These enable Dev teams to plumb together applications to build impressive, cost-effective, high-performing systems.</p><p>What makes this technology more attractive to Ops teams is the ability to hook into <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html">Cloudwatch Events</a>. Several AWS services emit Cloudwatch Events as they go about their daily lives. For instance, a Cloudwatch Event is sent as an EC2 instance enters different stages in its lifecycle. In the old world, we’d need an SNS topic to listen for those transitions. But now, a Cloudwatch Event is broadcast like the proverbial falling tree in the forest, and it’s down to us if we want to listen.</p><p>Two of the services that emit Cloudwatch Events are particularly useful to us.</p><p>The first is <a href="https://docs.aws.amazon.com/config/latest/developerguide/monitor-config-with-cloudwatchevents.html">AWS Config</a>. Config can ‘watch’ certain components (for instance security groups) and run a customisable Rule against them when they change (for instance, to check that a security group does not have port 22 open to the world). If anything is amiss, a Cloudwatch Event is pinged into the ether. This is a boon for security monitoring.</p><p>The second is CloudTrail. CloudTrail monitors all actions made against the AWS API, and <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-CloudTrail-Rule.html">can be configured</a> to send a Cloudwatch Event when a specific action is made, for example when a new Cloudwatch Logs Log Group is created. What this means is that you can trigger a Lambda function off of the back of <em>any action performed within AWS</em>. Powerful stuff indeed.</p><h3>Serverless at Spaceape</h3><p>With that in mind, here is a selection of just some of the weird and wonderful use cases that the DevOps team here at Space Ape have found for Lambda functions within AWS:</p><p><strong><em>Authenticated Proxies</em></strong></p><p>Need to punch a hole in your firewall to access resources in your VPC? No problem, just use a Cognito-protected API Gateway endpoint with a Lambda function to proxy requests.</p><p><strong><em>Autoscaling Aid</em></strong></p><p><em>Problem</em>: you need to perform some procedure before giving Autoscaling the nod that it can indeed terminate your EC2 instance (an example might be ensuring that a job queue has been emptied).</p><p><em>Solution</em>: Subscribe a Lambda function to an SNS topic receiving notifications from your ASG. Have it trigger your shut-down process, perhaps via SSM Run Command. Go to pub.</p><p><strong><em>S3 Object Replication</em></strong></p><p>Yes, S3 has built-in support for replication. But it only allows for a single replication destination, what if you need to replicate the same data to multiple regions? Why, use a Lambda of course. With the S3 bucket as an event source.</p><p>Incidentally, why might you even <em>want</em> to do this? Well, if you are using Cloudformation to deploy your Serverless applications (as both SAM and the Serverless framework do), you need a copy of your code sitting on S3 in each region you wish to deploy to…</p><p><strong><em>Load Testing</em></strong></p><p>Wouldn’t it be nice if you could corral a global flash-mob of thousands of minions and focus them, tractor-beam like, on an endpoint you’d like to test? You can!</p><p>Granted, this one is a little more involved. We have built an elaborate system to this end (blog post incoming). The fast and transient nature of Lambda functions means we are able to easily incorporate load-tests into our CI pipeline.</p><p><strong><em>Tidy Up Terminated EC2 Instances</em></strong></p><p>We have a handful of tasks we like to do when an EC2 instance is terminated (like removing a Route53 entry, for example). By hooking a Lambda function up to the Cloudwatch Event Bus we can ensure that this gets done.</p><p><strong><em>Generate Metrics</em></strong></p><p>We’ve all been there: you need to pull some data from an ill-conceived 3rd-party API, convert to metrics as understood by <em>your</em> metrics system, then push to said system. Do you write a cron-job? Nope. A daemon? Definitely not. Just have a scheduled Lambda function do it for you!</p><p><strong><em>Security Monitoring</em></strong></p><p>Who knew that Lambda functions would be such an ally in the fight against cyber-crime? Increasingly we are employing them to assist us in the securing of our growing estate: as a snitch that raises the alarm every time someone logs into an AWS account as Root. Or as a fence for custom AWS Config rules gathered over many accounts, or indeed for the custom AWS Config rules themselves. Their uses are legion.</p><p><strong><em>Logging</em></strong></p><p>A corollary of running more and more Lambda functions is that more and more you find yourself embroiled in the murky world of Cloudwatch Logs.</p><p>To use the Cloudwatch Logs console is to wonder immediately how you can avoid using it ever again. The solution is — clearly — more Lambdas! These notice when a new Log Group is created and subscribe it to a separate function that in turn forwards the logs to an AWS Elasticsearch cluster.</p><h3>Summary</h3><p>Hopefully this article has highlighted some of the many ways that Serverless can help Ops teams build stable, secure infrastructure. If you fancy working in such an Ops team, <a href="https://spaceapegames.com/careers/it-devops/devops-engineer">we’re hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39202dba2ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/serverless-for-devops-teams-39202dba2ce">Serverless For Devops Teams</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing Applications With AWS SAM]]></title>
            <link>https://medium.com/spaceapetech/developing-applications-with-aws-sam-295c320d563c?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/295c320d563c</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Thu, 25 Oct 2018 14:59:48 GMT</pubDate>
            <atom:updated>2018-10-25T14:59:48.405Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDZOKrPJ7JP7ycuRqDGjAA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/cokIcMulPJA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Miles Storey</a> on <a href="https://unsplash.com/search/photos/sam?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>So you’ve decided there may be something to this Serverless malarkey, after all. You’ve breezed through the paste-in-the-console examples and feel ready to move onto something <em>non-trivial. </em>One of the first decisions you’ll have to make is which deployment framework you should use<em>.</em></p><p>In the absence of any better ideas, frameworks are what we’re left with to define and manage serverless applications. A perfunctory Google search will probably lead you to conclude that the <a href="https://serverless.com/">Serverless framework</a> is the best option. Indeed, that may well be the case: it is the incumbent in this space, and boasts support for multiple providers, has good documentation and a bustling community.</p><p>However, if you are deploying solely to AWS, and you prefer your documentation <em>challenging</em>, there are a growing number of reasons to look at Amazon’s offering, the <a href="https://github.com/awslabs/serverless-application-model">Serverless Application Model </a>(SAM).</p><p>First, there is the fact the it sits within the AWS ecosystem. This means that (in theory) it should be privy to upcoming Lambda and API Gateway developments, and (in theory) will be supported by AWS.</p><p>Second, relatedly, is the fact that it is essentially <em>just CloudFormation</em>. If you can stomach its infuriating verbosity there is very little that can’t be achieved with CloudFormation. Also, unlike with the Serverless framework, you don’t have to choreograph updates to a Node.js package across your team (and your build system).</p><p>Another reason is the Serverless Application Repository. This is a platform for sharing (either publicly or internally) applications built with SAM. At present the repository is somewhat hit-and-miss, little more than an oft-handy menagerie of examples. But it doesn’t take <em>too</em> much imagination to envisage a day when applications in the SAR are monetised and useful, a la the AWS Marketplace.</p><p>If these arguments sound a bit flimsy, it’s because they are. But there <em>are</em> some less wooly, more technical reasons why SAM is becoming a good choice, and that’s what this article hopes to illustrate. It will aim to do this by way of what is really a beginner’s guide to developing SAM applications, covering testing and deploying, then finishing up with a look at some of its limitations.</p><h3>The Application</h3><p><em>Note, if you want to play along at home, the code can be found at </em><a href="https://github.com/spaceapegames/developing-sam-applications"><em>https://github.com/spaceapegames/developing-sam-applications</em></a></p><p>The application which we’ll be deploying simply replicates objects from one S3 bucket to another. You’ll no doubt be aware that S3 can do this already so, yes, this is a completely pointless endeavour, but bear with us.</p><p>It is a Go application, with a single function (<em>Replicate</em>) that looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f870b7e77f17d1a4b46b5863f2d02ff7/href">https://medium.com/media/f870b7e77f17d1a4b46b5863f2d02ff7/href</a></iframe><p>When an object is created (in a yet-to-be-specified S3 bucket), the Replicate function will receive a JSON event which is de-serialised into an S3Event, from which we can pluck information about the bucket and the object itself. The information is used to invoke a CopyObject API call (which neatly copies an S3 object from one location to another, without spooling it into memory).</p><p>The destination bucket is specified through an environment variable, DESTINATION_BUCKET.</p><p>Simple.</p><h3>Testing Locally</h3><p>The first superpower SAM bestows upon us is the ability to fabricate sample Lambda events.</p><p>Lambda applications are by design event driven: some event happens in the cloud (an upload to an S3 bucket, a request against an API Gateway endpoint, a write to a DynamoDB table) and some JSON-encoded context about that event is used to invoke a Lambda function. The list of possible events is ever-growing, and to stand any hope of handling one, you need to know what to expect.</p><p>If you’ve had any substantive dealings with Lambda then you’ve probably found yourself unhappily logging one of these events, just to discern its structure. You are not alone.</p><p>Thankfully SAM, or more specifically the <a href="https://github.com/awslabs/aws-sam-cli">SAM CLI</a>, makes this practice a thing of the past, via sam local generate-event ; a command that will spit out some sample JSON that can be used to develop and test a Lambda function, without going <em>near</em> any clouds.</p><p>In our case, the sample event we are interested in is an S3 PUT:</p><pre>$ sam local generate-event s3 put</pre><p>But, in order to put this to use, we need to back up a bit and create a template.yml</p><h3>The Template</h3><p>All SAM applications are driven by a CloudFormation template called, by convention, template.yml . SAM extends the CloudFormation spec with a few new resource types:</p><ul><li><a href="https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction">AWS::Serverless::Function</a></li><li><a href="https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessapi">AWS::Serverless::Api</a></li><li><a href="https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlesssimpletable">AWS::Serverless::SimpleTable</a>.</li></ul><p>Only the first will be covered here, as it is by far the most useful. (The other two can be used to provision an entry-level API Gateway and an entry-level DynamoDB table respectively, anything slightly more complex will require escaping into CloudFormation-proper).</p><p>So, we’ll start off with as simple a template as can be (just so we can test locally) and then build upon it in the next section:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/881969429180a1fd49a2340c57c63f78/href">https://medium.com/media/881969429180a1fd49a2340c57c63f78/href</a></iframe><p>The first two lines are SAM/CloudFormation boilerplate, with the third being a description of the application.</p><p>Then we have a <a href="https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst">Globals section</a>. Any settings here are applied to all of the relevant resources defined within the template. Above we have set the runtime to be Go, and specified the DESTINATION_BUCKET environment variable.</p><p>Next is the Resources section, where CloudFormation resources are defined. The single resource defined above is an AWS::Serverless::Function which we have called Replicate. This function, when invoked, will look for a binary called bin/replicate within a zip archive ./main.zip.</p><p>It is probably a good time, then, to create bin/replicate and stuff it into main.zip:</p><pre>$ GOOS=linux go build -o bin/replicate handlers/replicate/main.go<br>$ zip -r main.zip bin/*</pre><p>This will compile (or cross-compile) the replicate function into a bin/replicate binary, and then zip the contents of bin into main.zip . The Lambda execution environment is inside an Amazon Linux OS, hence the GOOS specificity.</p><p>OK, now we are ready to test the function:</p><pre>$ sam local generate-event s3 put | sam local invoke Replicate</pre><p>This invokes the Replicate function (<em>Replicate</em> being the resource name as specified in template.yml ) with the output of the generate-event s3 put command.</p><p>What actually happens is that SAM goes off and downloads a Docker image which mimics the Lambda runtime environment. Using this image, it creates a container, with main.zip mounted inside. Finally the sample S3Event is used to invoke bin/replicate as specified in the template.</p><p>What you should see is a log entry informing us that the function is:</p><pre>replicating test/key from bucket example-bucket, to my-destination-bucket</pre><p>Followed by an error, blood red, telling us that the bucket does not exist. (Unless by some sick twist of fate you actually have a bucket called my-destination-bucket , in which case, um…sorry). This error is in fact the return value of the function.</p><p>As well as providing easy access to a bevy of sample events, the SAM CLI has the ability to fire up a local API to mimic API Gateway requests/responses. This can be useful as a development aid but is fairly limited in that it only supports a fraction of the features that a full-blown API Gateway offers.</p><p>Now that we’ve tested our S3 Replicator locally, lets fire it at AWS, see what sticks.</p><h3>Deploy</h3><p>It should be clear that we need to augment our template if we want it to do anything more than shout at us about non-existent buckets. With that in mind:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f2ac639ba18e6a3f2ac57a3fbefa103/href">https://medium.com/media/0f2ac639ba18e6a3f2ac57a3fbefa103/href</a></iframe><p>Our template has blossomed, and there are some things to point out:</p><ul><li>We have added a Parameters section. There is nothing special about the parameters therein, they are just common-or-garden CloudFormation parameters.</li><li>We have referenced these Parameters throughout the template, by using <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html">CloudFormation Intrinsic Functions</a> (e.g.!Sub and !Ref ). Again, this is just run-of-the-mill CloudFormation, but it highlights how we can inject a dose of pliability into the template. (Not shown here, but in a similar vein, <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html">Pseudo Parameters</a> provide easy access to things like the AWS Account ID and region)</li><li>We have created resources for two S3 buckets, a source and a destination. (If you <em>are</em> playing along at home, be sure to change the names of those buckets).</li><li>We have added an Event to the Replicate function, without which it would never be invoked. This particular Event is triggered whenever an object is created in the source bucket and — having tested locally above — we know that our application expects and will handle the event correctly.</li><li>Finally we have given our Replicate function some IAM permissions, using <a href="https://github.com/awslabs/serverless-application-model/blob/develop/docs/policy_templates.rst">SAM Policy Templates</a>.</li></ul><p>This last point is worth highlighting, as another way that SAM attempts to make life a little easier. <a href="https://github.com/awslabs/serverless-application-model/blob/develop/docs/policy_templates.rst">SAM Policy Templates</a> provide groupings of commonly used IAM permissions that aim to save the beleaguered developer from inscribing reams of IAM policy docs. In the above example, we have used the S3CrudPolicy to permit CRUD access on both the source and destination buckets.</p><p>OK, we should now be ready to Deploy!</p><p>Deploying a SAM application is a two stage process. The first stage involves packaging the application code and uploading it to S3; the second involves sending the yaml template off to CloudFormation so that it can turn that code into something useful.</p><p><strong><em>Package:</em></strong></p><p>Lets package our S3 Replicator:</p><pre>$ sam package --template-file template.yml \<br>              --output-template-file packaged.yml \<br>              --s3-bucket developing-sam-applications</pre><p>This command will find the location of our compiled-and-zipped code (./main.zip) from the <em>CodeUri</em> property of the <em>Replicate</em> resource (in template.yml).</p><p>It will then upload that zip archive to the specified s3 bucket (<em>note: this bucket must already exist</em>), with a hashed value as the key.</p><p>Finally it takes that hashed value, injects it into the template.yml and outputs the result as packaged.yml, expanding bits of the CloudFormation along the way.</p><p>We are left with a template which can be used to summon the hounds of CloudFormation.</p><p><strong><em>Deploy:</em></strong></p><p>The deploy command looks like this:</p><pre>$ sam deploy --template-file packaged.yml \<br>             --stack-name developing-sam-applications \<br>             --capabilities CAPABILITY_IAM</pre><p>This command takes the packaged.yml generated above and uses it to create or update a CloudFormation stack named after the--stack-name argument. We have to grant CloudFormation some additional capabilities as it will be creating IAM permissions.</p><p>A point to note about both of the above commands is that sam is simply an alias for aws cloudformation , the two are interchangeable. (Apart from being a uninteresting curiosity, what this means is that there is no requirement on the SAM CLI to <em>deploy </em>SAM applications, i.e. you can possibly forego installing it on your CI system).</p><p>Indeed, now your SAM application <em>is</em> just a plain-old CloudFormation stack; you can check the progress of your deploy with plain-old CloudFormation tools:</p><pre>$ aws cloudformation describe-stack-events \<br>      --stack-name developing-sam-applications</pre><p><strong><em>Test:</em></strong></p><p>With a bit of luck, and a following wind, the deploy will complete without incident. The final thing we need to do is to test out our dazzling new tool.</p><p>Watch, as we put something in the source bucket and see it magically appear in the destination!</p><pre>$ mkfile 10k /tmp/testfile<br>$ aws s3 cp /tmp/testfile s3://my-source-bucket/<br>.....<br>$ aws s3 ls s3://my-destination-bucket/testfile &amp;&amp; echo &quot;woah!&quot;</pre><p>But what happened? To dispel the magic we can unveil another little widget in the SAM swiss-army-knife, the logs command:</p><pre>sam logs -t -n Replicate --stack-name developing-sam-applications</pre><p>It doesn’t seem much, to be able to easily see your application logs, but this is Lambda, the new Wild West. The sam logscommand is clever enough to take the CloudFormation resource name of our Lambda function (<em>Replicate</em>), look up the CloudWatch log group it is communing with, tail that log group, and then relay that information to your screen, eyes and ultimately, brain.</p><p>If all went to plan, you should see the logs of the function invocation we induced above.</p><h3>Safe Lambda Deploys</h3><p>A final benefit to mention, perhaps the most compelling, are <a href="https://github.com/awslabs/serverless-application-model/blob/develop/docs/safe_lambda_deployments.rst">Safe Lambda Deploys</a>.</p><p>These allow for defining a strategy for directing traffic (e.g. a certain percentage each minute) at freshly deployed Lambda functions. They provide hooks for canary analysis and monitoring of CloudWatch alarms, to automatically decide whether to rollback or continue a deployment. They are tremendously useful and, best of all, fit snugly into the same SAM template as everything else.</p><p>Regrettably Safe Lambda Deploys are not covered here, as they deserve a whole blog post to themselves.</p><h3>Summary</h3><p>Finally a word of caution; not all in the SAM world is rosy, there are of course some trade-offs.</p><p>For example, if you wish to deploy the same SAM application to multiple regions, you must make sure it exists in an S3 bucket in <em>all</em> of those regions. No problem, you say, just turn on S3 Replication. Unfortunately this only allows for a single replication destination, any more than that and you’re stuck uploading to multiple buckets. (Or, <em>or!</em>, you could write your own S3 Replicator…just a thought).</p><p>Another limitation is that it is often not possible to hook up your Lambda function to resources which pre-exist. If, say, we wanted to trigger our S3 Replicator from a bucket which already exists, we would soon be disappointed. This is down to CloudFormation not being able to modify resources outside of its immediate stack. It is apparently feasible to write a <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html">CloudFormation Custom Resource</a> to handle this for you, and the SAM maintainers are <a href="https://github.com/awslabs/serverless-application-model/issues/249">actively considering</a> bundling this functionality.</p><p>However, SAM is still a new technology, and these shortcomings are being addressed in turn. Even with them taken into account, hopefully this article has highlighted that there are reasons why SAM might still be a good choice for firing your Lambdas at your clouds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=295c320d563c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spaceapetech/developing-applications-with-aws-sam-295c320d563c">Developing Applications With AWS SAM</a> was originally published in <a href="https://medium.com/spaceapetech">spaceapetech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring in a Post Nagios World]]></title>
            <link>https://medium.com/devops-college/monitoring-in-a-post-nagios-world-2f6bb738ee96?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f6bb738ee96</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[nagios]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Fri, 23 Feb 2018 14:32:58 GMT</pubDate>
            <atom:updated>2018-02-23T14:32:58.783Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*z0iTh1ET60VJJbUGozck1w.png" /></figure><p>Monitoring is changing. It has been a glacial, unheralded change, driven by attitudinal shifts and technological advances, but traditional monitoring systems such as Nagios are being usurped by metrics platforms and even machine learning. We are living in a post Nagios world.</p><p>There will be those to whom that statement is achingly obvious, sun-tanned Bay Area pioneer-types. Then there’ll be those who’ve spent years building protectionist Nagios empires glaring angrily at their screens. Most of us will recognise some truth in it, but will conclude that our monitoring is fine, thank-you-very-much. However there is most definitely a vanguard taking a different approach.</p><p>Allow me to explain.</p><p>Traditional monitoring systems — Nagios, Xymon, Zabbix, Sensu et al — all share a common approach: they run checks on hosts and then report whether the result of those checks are OK or Not OK, good or bad, green or red (ignoring the murky yellow for now). By setting arbitrary thresholds on disk usage, CPU utilisation, memory pressure etc. we can be woken up in the depths of night to ward off any disasters that <em>might </em>have been. At the cost of a few additional alerts, and maybe some sleep, we are generally able to maintain excellent uptime statistics.</p><p>And so, throughout the short life-span of our fledgling profession, it has always been. The Nagios family had firmly entrenched itself as the establishment of the old world.</p><p>But ours is an evolutionary profession, ideas and abstractions engender change and often the drivers behind change are the very things that enable it. In fact monitoring has changed little compared with the seismic shifts we’ve seen elsewhere: the tools may be different but the techniques are decades-old. Monitoring is the overgrown teenager about to blossom.</p><p>We can actually chart the demise of traditional monitoring using the signal markers that have characterised our industry over the last decade, beginning with the first, probably the single most impactful change of the millennium — The Cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CaHb_EK4j9Ob0ioV-sjJlw.png" /><figcaption>The Cloud presented a huge philosophical change.</figcaption></figure><p>Along with the tangible benefits cloud computing bought, a less obvious corollary was the inexorable attitude shift it engendered. Gradually it evoked (and continues to evoke) a mental shift away from the data centre. Previously hard-won knowledges of Spanning Trees, LUN alignments and ILOMs began to fade, like ageing war heroes, supplanted by millennial ideas such as immutable infrastructure and infinite scalability. Where once we cared for our servers, lovingly tucking their cables behind rack clips and meticulously replacing their creaking disks; now we shoot them in the head to save on the EC2 bill.</p><p>We had unwittingly embarked on the relentless pursuit of abstraction. Abstraction away from the spinning and buzzing, the earplugs, the cold-and-hotness and the draconian security measures of the data-centre. Like a careening juggernaut this trend continues, unsentimentally dismantling anything in its way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*I3xkJONTD4cEjZt0WN94Pw.png" /></figure><p>The first inklings of an existential crisis was on the cards for Nagios. Its centralised file-based configuration model started to feel abstruse in this new world of transient hosts. Younger relations like Sensu, with an API for registering and de-registering fleeting servers, stepped in to bolster the old world order and the teetering status-quo managed to right itself.</p><p>Then along came containers.</p><p>The new prince of the realm is Kubernetes. Another abstract layer — the nebulous Cluster — has been slapped atop the existing virtualisation layer. Furthermore, in almost all cases it makes very little sense to actually <em>manage</em> the Cluster oneself, no, just throw some containers at GKE or EKS and hope they stick. Naturally this means that we start to care even less about our servers. Our ‘hosts’, such as they are, swim around like fish in the proverbial barrel, to be shot at and re-animated by the omnipotent orchestration layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*UbqCOYqs08zGpVzkEjtAbw.png" /></figure><p>And so the very bedrock of the old monitoring establishment begins to creak and groan. All along they had been treating the host as the basic unit of monitoring, but the interminable quest for abstraction had driven a wedge between the host and what we actually care about — our services.</p><p>The abstraction gods are not yet appeased. Kubernetes may reign supreme, but it is a fragile rule, with one eye on the impending Serverless hordes, who care little even for containers. They’ll just take some code and run it, somewhere. With this the downfall of traditional monitoring tools will be complete. The only way they can be used in a Serverless world is in some sort of black-box capacity, poking and prodding from outside the circle.</p><p>Of course none of this means that Monitoring is going away. On the contrary, it has just evolved with the times. The demise of traditional monitoring tools can be charted diametrically against the rise of a different sort of tool — the time-series database. Again we can reach into the annals of recent history to trace this rise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*hdtNqvE-YCp9N7XBPlteMg.png" /><figcaption>Etsy’s pithy edict</figcaption></figure><p>Aeons ago, in 2010, Etsy gifted the world statsd and Graphite. Amazingly statsd (or at least the statsd protocol) is still widely in use today, as a means for aggregating metrics and sending them to a visualisation tool like Graphite. Before Graphite, the only graphs we had were RRDs (Round Robin Databases), graphs that looked as though they’d been hand-pixellated in the ’80s. Aside from any aesthetic complaints, they lacked something else — a query language. A way to aggregate and apply mathematic formulae to our metric data and, most importantly, key alerts off of it.</p><p>Again it was The Cloud that provided the catalyst for expansion, in its role as the great enabler: erstwhile hobbyists, open source heroes and budding enterprises now had access to freely available compute with (more recently at least) fast disks. Advances in time-series storage spawned OSS tools like InfluxDb and Prometheus, and 3rd party SaaS vendors like Datadog and Wavefront were able to build systems that could process a rate of metrics that would leave Graphite dribbling into its carbon relays. The introduction of tagged metrics and the enrichment of query languages made the data more useful.</p><p>Containerisation and Serverless bought a common bedfellow in the shape of Microservices, with all their promise and all their baggage. If we accept their benefits — smaller units of deployment, enablement of polyglot infrastructure etc. then we must concede their shortcomings — difficulty in discerning just what the bleedin’ hell is going on in there. We were effectively <em>forced</em> to re-evaluate our monitoring techniques. Metrics were our only real option, and we would need a <em>lot</em> of them.</p><p>Recently the term Observability has been coined, to differentiate the larger bulk of the information we collect from the small subset that we want to actively watch (i.e. Monitor). The triumvirate of Observability are logging, metrics and tracing. You may wish to alert on some of the measures that fall into the first 2 categories, but probably only a very small portion. The rest is passively gathered and exists as an investigatory aid in the event of one of those alerts being triggered (or for capacity planning, performance engineering etc.)</p><p>This fork in the road is only achievable because of the evolving technologies described above, and it is only required because of those self same technologies. In her <a href="https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e">excellent and seminal article</a>, Cindy Sridharan argues eloquently that ‘<em>the ideal number of signals to be “monitored” is anywhere between 3–5, and definitely no more than 7–10</em>’ . To suggest that 10 years ago would’ve been laughable, and is testament to how far we’ve come.</p><p>But before you go ahead and rip out your Nagios or Sensu clusters and replace them with Prometheus or Wavefront, I want to concede a disclaimer: most of the above is written with tongue firmly wedged in cheek. The arguments hold true for buzz-word architectures; serverless-containerised-microservices. Naturally there is an increasingly sizeable cohort with such models, but equally there is a much larger long-tail. Some of us aspire to such things, and probably have a mixture of ‘new-wave’ and traditional systems.</p><p>Trying to monitor traditional systems (let’s say those where you do still care somewhat about your hosts) using solely metrics is not without its difficulties. How do you know, for example, if an EC2 instance has stopped sending metrics or if it has been legitimately scaled out? There is also a certain amount of soul-searching in unwinding years of conditioning for you and your team.</p><p>But the winds of change are blowing, new paradigms require new approaches, and the future looks ever more distinguishable from the past. It may be time to take a different approach.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f6bb738ee96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-college/monitoring-in-a-post-nagios-world-2f6bb738ee96">Monitoring in a Post Nagios World</a> was originally published in <a href="https://medium.com/devops-college">DevOps College</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linuxkit on AWS, with Terraform]]></title>
            <link>https://medium.com/devops-college/linuxkit-on-aws-with-terraform-86786c51d894?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/86786c51d894</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[linuxkit]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Thu, 07 Sep 2017 08:40:49 GMT</pubDate>
            <atom:updated>2017-09-07T08:40:49.232Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*9JFZu0PRTOkTEodz4L_8KA.jpeg" /></figure><p>The container revolution is well underway. The Dockeristas swept from their coastal hideouts to infiltrate and disrupt even the most entrenched orders of the old establishment. The democratisation of containers is almost complete and whilst battles rage over cloud orchestration, a new bearded general enters stage-left in the form of Linuxkit. Unassuming and relatively unheralded, <a href="https://github.com/linuxkit/linuxkit">Linuxkit</a> could end up providing the bridge between containers and microkernels.</p><p>A containerised application is bundled with everything it needs in the way of system libraries and software packages. So why bother replicating those libraries in the host OS? Not only does this mean shifting more bits around, it also leads to an increased surface area for attacks.</p><p><a href="https://github.com/linuxkit/linuxkit">Linuxkit</a> provides the ability to build an extremely small distribution (~50m) stripped of everything but that which is required to run containers. The init system and all other userland system components (e.g. dhcp, sshd) are containers themselves and as such can be swapped out, or others be plugged in. The whole OS is immutable (unless data volumes are configured), and will run anywhere one finds silicon or a hypervisor: Windows, MacOS, IoT devices, the Cloud etc.</p><p>This article hopes to uncover some of the intimacies of Linuxkit, whilst explaining how to build and run one of these minimal distros on AWS using Terraform. The outcome will be an AMI from which we can create an EC2 instance that will run an SSH daemon, and little else.</p><p>A Linuxkit machine is represented as a yaml file, and built with the moby tool. Here is the yaml configuration for the machine we’ll build (to play along, paste this into a file called aws.yml):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a1b4127aa5375db6b58aa4a405f96dd/href">https://medium.com/media/0a1b4127aa5375db6b58aa4a405f96dd/href</a></iframe><p>The mobytool will snaffle up this yaml and transmute it into a bootable image (such as an ISO).</p><p>The yaml describes a set of specialised containers plugged together in a Lego-esque fashion to build a very small OS with just enough nous to run other (normal) containers. What’s interesting here is the opportunity for innovation: a savvy developer could elect to write their own init system, add additional system components or even reduce the OS further (maybe they decide they don’t need containerd for instance).</p><p>It is divided into discrete sections, starting with the ‘kernel’ section that defines which kernel the OS should run. Each kernel is a Docker image containing the kernel along with a tarball of compiled modules. The kernels themselves are based on latest stable releases, with some patches back-ported from newer kernels. That same savvy developer is of course free to compile his or her own customised kernel should they see fit.</p><p>The ‘init’ section defines the Docker images that go together to comprise everything required to get us to a stage where we can run containers. It is followed by the ‘onboot’ section; a list of images describing short-lived, one-shot services to be run on-boot (such as dhcpd, to get us an IP address) and finally the ‘system’ section, which will generally be long-running processes that give the machine a purpose (for example one could run nginx as a system service).</p><p>So, lets take this yaml definition and turn it into something we can actually run on AWS. To do this we invoke the moby build command.</p><p>moby build -name aws -output raw -size 100M aws.yml</p><p>The tool will go off and pull any containers which are not immediately present. It <em>unpacks</em> the filesystem of each of the containers, does some shifting to make the whole thing palatable to the init process, then bundles the lot into an <em>initramfs</em> (a compressed cpio archive). The <em>initramfs</em>, along with the kernel and kernel command line, is the build output. It is an entirely immutable Linux machine, with all system services baked in. The default size of 1G is overridden here, and even 100M is probably generous (in all honesty, I couldn’t see the relevance of this option once it becomes an AMI).</p><p>Let’s stop and think about this for a moment: an entirely immutable system, coming in at around 50MB with nothing extraneous to that which is needed to run containers. The root filesystem is read-only, making it stateless and tamper-proof. The build itself takes a matter of seconds, and is eminently reproducible, making it an ideal candidate to pass through a CI system. Although we have added an SSH daemon, by default there is no login (not even a terminal unless you add a getty container). This is starting to feel like the fabled promised-land of <em>Proper Devops</em>…</p><p>Furthermore it is entirely possible, and probably expected, that you run something like Docker or Kubernetes to enable dynamic scheduling of applications. So we are by no means limited only to the applications which are baked into the image.</p><p>In order to run this image on AWS we need to turn it into an AMI. Going back to the moby build command above; apart from the name of the resulting image and the yaml definition, we have also specified a size parameter and an output format. There are several different output formats to choose from, depending on which kind of host system the image is to be run. AWS AMIs require the raw output.</p><p>We need to take that raw output and convert it to an AMI. Linuxkit makes this easily achievable through the linuxkit push command. Under the hood, the linuxkit push command does a few things: it uploads the raw disk to an S3 bucket, then initiates an <em>import-snapshot</em> job through <a href="http://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-import-snapshot.html">VM Import/Export service</a> to create an EBS snapshot from that raw disk image. Finally it creates an AMI using that snapshot.</p><p>Clearly we’ll need an S3 bucket to upload to, but we also need to configure an IAM role called <em>vmimport</em> and allow the Virtual Machine Import Export service to assume it.</p><p>We’ll use <a href="https://www.terraform.io/">Terraform</a> to manage these entities as not only is it in the title of this post, but it makes tidying everything up that much easier. So, given a directory structure that now looks like this:</p><pre>.<br>├── aws.raw<br>├── aws.yml<br>└── terraform<br> ├── main.tf<br> └── files<br>  ├── assume-role-policy.json<br>  └── policy.tpl</pre><p>Paste the following into the assume-role-policy.json file (this allows the vmie service to assume the <em>vmimport</em> role):</p><pre>{<br>   &quot;Version&quot;: &quot;2012-10-17&quot;,<br>   &quot;Statement&quot;: [<br>      {<br>         &quot;Effect&quot;: &quot;Allow&quot;,<br>         &quot;Principal&quot;: { &quot;Service&quot;: &quot;vmie.amazonaws.com&quot; },<br>         &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>         &quot;Condition&quot;: {<br>            &quot;StringEquals&quot;:{<br>               &quot;sts:Externalid&quot;: &quot;vmimport&quot;<br>            }<br>         }<br>      }<br>   ]<br>}</pre><p>And this into policy.tpl</p><pre>{<br>   &quot;Version&quot;: &quot;2012-10-17&quot;,<br>   &quot;Statement&quot;: [<br>      {<br>         &quot;Effect&quot;: &quot;Allow&quot;,<br>         &quot;Action&quot;: [<br>            &quot;s3:ListBucket&quot;,<br>            &quot;s3:GetBucketLocation&quot;<br>         ],<br>         &quot;Resource&quot;: [<br>            &quot;arn:aws:s3:::${bucket}&quot;<br>         ]<br>      },<br>      {<br>         &quot;Effect&quot;: &quot;Allow&quot;,<br>         &quot;Action&quot;: [<br>            &quot;s3:GetObject&quot;<br>         ],<br>         &quot;Resource&quot;: [<br>            &quot;arn:aws:s3:::${bucket}/*&quot;<br>         ]<br>      },<br>      {<br>         &quot;Effect&quot;: &quot;Allow&quot;,<br>         &quot;Action&quot;:[<br>            &quot;ec2:ModifySnapshotAttribute&quot;,<br>            &quot;ec2:CopySnapshot&quot;,<br>            &quot;ec2:RegisterImage&quot;,<br>            &quot;ec2:Describe*&quot;<br>         ],<br>         &quot;Resource&quot;: &quot;*&quot;<br>      }<br>   ]<br>}</pre><p>This is the policy that will be attached to the <em>vmimport</em> role. It allows access to the S3 bucket that Terraform will create, as well as snapshotting and image permissions.</p><p>Finally sling this code, which will create the IAM role and S3 bucket, into main.tf:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd0a034e1d63fb23561c8ff4fa792878/href">https://medium.com/media/fd0a034e1d63fb23561c8ff4fa792878/href</a></iframe><p>Running terraform apply will result in the creation of an S3 bucket along with the requisite IAM permissions. Note the name of the bucket created, and substitute it in running this command:</p><p>linuxkit -v push aws -bucket &lt;YOUR_BUCKET_NAME&gt; aws.raw</p><p>This may take some time to complete (believe me, it takes even longer if you don’t override the default 1G size in the moby build command above!). With a bit of luck and a following wind you will eventually be presented with an AMI ID. Use that in the following Terraform code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42785bcde5cd2d6f142935e0c654fb3b/href">https://medium.com/media/42785bcde5cd2d6f142935e0c654fb3b/href</a></iframe><p>(You can put this in main.tf if you like, although it is now independent of the infrastructure set up before)</p><p>This will boot up an AMI in EC2 Classic, which you should be able to log into using your local SSH key (assuming it can be found at ~/.ssh/id_rsa.pub), as root, on its external IP address:</p><p>ssh 54.227.66.666 -l root</p><p>Unfortunately you can’t do a great deal, but perhaps that is the point. It is however illustrative to have a poke around, to highlight the properties of a machine built with Linuxkit…</p><p>First thing to note is that you are connected to the SSH container, <em>not</em> the machine itself. Files and binaries available to the machine at large are not available within individual containers, unless they are explicitly mounted.</p><p>This can be seen with the aid of a slight detour into an explanation of how Linuxkit handles AWS metadata:</p><p>Linuxkit’s metadata package will handle the metadata of VMs booted under different cloud providers. On AWS it extracts a certain amount of information from the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html">metadata service</a> upon boot, and makes it available under /var/config as files. These files can then be bind-mounted into individual containers. An example of this can be seen in our very own aws.yml, from above, where/var/config/ssh/authorized_keys is mounted into the root home directory of the ssh container, allowing us to SSH as the root user.</p><p>It’s natural (at least was to me) to try and look in this directory from the command line, however:</p><pre>ls: /var/config: No such file or directory</pre><p>This is of course because we are in the SSH container. To enter the host OS we need to enter the mount namespace of pid 1 (the init process):</p><pre>nsenter — target 1 — mount</pre><p><em>Now</em> we can look in /var/config:</p><pre># ls -1 /var/config<br>availability_zone<br>hostname<br>instance_id<br>instance_type<br>local_hostname<br>local_ipv4<br>provider<br>public_ipv4<br>ssh<br>userdata</pre><p>It should be noted that the metadata service <em>is</em> still available from within containers (so you can use IAM Instance Profiles, for example). But you’ll have trouble curl-ing it, as there’s no curl remember? Minimalist OS. (You’d need to add a curl container).</p><p>Whilst in the parent namespace we can draw notice to another interesting property of a Linuxkit OS: if you run a df -h you’ll see that there is <em>no root filesystem</em>. However, running the mount command yields, amongst other things:</p><p>rootfs on / type tmpfs (ro,relatime)</p><p>Rootfs is a special type of filesystem. On boot, the contents of <em>initramfs</em> (which if you recall from above is a cpio archive containing the filesystems of all the containers that make up the OS) is copied into rootfs and mounted on /. The kernel then looks for a binary called init on this rootfs, which eventually becomes pid 1. On normal Linux systems the <em>actual</em> root filesystem is simply mounted over the top of rootfs and it is forgotten about. But on Linuxkit machines this never happens, the system runs with its root ‘disk’ in read-only memory, lending the system its immutability. There are no system upgrades (there isn’t even a package manager), if you need to make a change you build a new machine.</p><p>Hopefully this has provided a <em>somewhat</em> interesting look at Linuxkit, a project very much in its infancy, which hasn’t even yet reached a production milestone. Apart from a few showcase projects (such as a <a href="https://github.com/linuxkit/linuxkit/tree/master/projects/kubernetes">port of Kubernetes</a>) it hasn’t attracted an enormous amount of press (or indeed marketing). Also, a minimal container-focussed OS is not a new idea — CoreOS have been doing it for years with Container Linux <em>and</em> can actually boast production workloads. However what I think sets Linuxkit apart is its dogmatic approach to security — truly immutable, read-only operating systems without so much as a terminal — and its potential flexibility; you could see specialised kernels being plugged in for embedded devices, or a customised init system. The system components (e.g. init, containerd) are written in languages which are slightly more accessible than C (e.g. Go or Rust) which again provides scope for innovation. Furthermore it feels very much like the next logical step in the container revolution: essentially treating the OS like a large container itself. Who knows, if the buzzword magpies decide that microkernels (or unikernels!) are the new shiny thing, Linuxkit stands ready to pounce.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86786c51d894" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-college/linuxkit-on-aws-with-terraform-86786c51d894">Linuxkit on AWS, with Terraform</a> was originally published in <a href="https://medium.com/devops-college">DevOps College</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stubbing the AWS SDK, or Why Testing Rocks]]></title>
            <link>https://medium.com/devops-college/stubbing-the-aws-sdk-or-why-testing-rocks-b4a5feb49e9f?source=rss-eb00737b90b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4a5feb49e9f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Louis McCormack]]></dc:creator>
            <pubDate>Thu, 05 Jan 2017 21:31:31 GMT</pubDate>
            <atom:updated>2017-01-06T08:23:22.576Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rwaPwilMXiWL93SHoTEe3A.jpeg" /></figure><p>If you’re reading this, dear Devop, you can probably remember a time before you were a Devop, or a Site Reliability Engineer, or whatever. Back to a time when you were a Systems Administrator, or a Unix Technician, or an Infrastructure Engineer; when your job involved switch blades and RAID cards, and you had to worry about LUN alignment and BIOS configurations.</p><p>Of course nobody is suggesting those tasks have disappeared but, with the advent of cloud computing, much of the yuckiness has been abstracted away from us. Instead of ‘remote hands’ we are presented with a nice clean API, gifting us the ability to build complex infrastructure…provided we have the skills to utilise it.</p><p>A corollary to this has been rise of OSS. Instead of black-box COTS products with expensive support contracts we are encouraged to use free software, with the source code hung out like fresh linen, inviting us to fix our own bugs and craft our own features.</p><p>Inevitably then, our roles have transmuted. Increasingly we have to live up to the Dev part of our job title. Where once we wrote scripts we now write code; we can find ourselves unexpectedly in coding interviews, and writing tooling has become an essential skill.</p><p>But this is good news! It should be embraced. What it means in practicality is that we’ve had to imbibe knowledge from our developer cousins, to catch up with them in a relatively short period of time. We have quickly adopted their coding practices: pull requests, code reviews, coding standards…and testing.</p><p>Testing. Until recently, I’ll admit, tests were things that were bolted on once everything else was done. TDD was something that happened in text books. I’d embark on a test suite with a heavy heart and a dramatic sigh, and the afternoon would drag.</p><p>Of course I was wrong. I’m now a convert, a born again testafarian. The moment of epiphany came when I realised that, far from being a hindrance, a good test suite is absolutely indispensable. Its a small investment up-front, a downpayment for months or years of smooth maintenance and easy extendability.</p><p>This is particularly true when writing code that interacts with vendor APIs, such as AWS. As long as the API is well designed and well documented, we are easily able to mock the expected responses, and write whole applications without a single credit card detail being passed.</p><p>To illustrate this point let’s write a very small Ruby class to perform the simplest of tasks — listing DynamoDB tables. We’ll use RSpec and the frankly awesome response-stubbing powers of the AWS Ruby SDK to test two common scenarios: listing only tables with a given prefix, and handling paginated responses.</p><p>In true TDD fashion, lets write some tests first:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a399b179a76612062a1ca56c3a036031/href">https://medium.com/media/a399b179a76612062a1ca56c3a036031/href</a></iframe><p>Nothing too radical here. We are describing tests for the eminently useful DynamoLister class. We wish to test listing all tables and then listing only those with a given prefix. (In the absence of any sort of namespacing, it is recommended practice to prefix DynamoDB tables with the environment in which they run)</p><p>One thing that does beg explanation is the setting of Aws.config[:dynamodb]. Here we are telling the AWS SDK to stub responses for — and only for — calls made to the :list_tables method of any instance of the Aws::DynamoDb::Client. Furthermore, we are telling it exactly what should be returned when this call is made, a list_tables_response(prefix: test).</p><p>Of what provenance is this mythical list_tables_response? Well, let me draw your attention, dear Devop, to the inclusion of the DynamoFactory module and its forebear, the require_relative &quot;factories/dynamo_factory.rb&quot;</p><p>If we open up this file we’ll find a single function, the list_tables_response:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85d8fa304131921cdac54f360cce4fc0/href">https://medium.com/media/85d8fa304131921cdac54f360cce4fc0/href</a></iframe><p>This function returns a handcrafted Aws::DynamoDB::Types::ListTablesOutput, populated with a list of names of DynamoDB tables. This is exactly the response that the AWS SDK returns when a list_tables call is made. How do we know this? Because the <a href="http://docs.aws.amazon.com/sdkforruby/api/index.html">AWS SDK docs</a> are blessedly complete. We can easily identify the types and nested types that are expected to be returned from any API call. This happens to be a very simple example, we can see <a href="http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#list_tables-instance_method">here</a> that the ListTablesOutput consists simply of a table_names array of strings, and potentially a last_evaluated_table_name, which we’ll come to in a moment.</p><p>The factory function takes 2 values: :count, which is the number of tables to be included in the response, and :prefix, a string which all of the tables will be prefixed with (i.e. the environment).</p><p>So, we now know that any call to list_tables made against any instance of Aws::DynamoDB::Client will result in a list of 10 tables (the default :count value), all prefixed with “test” (the value we pass for :prefix in the initial stub_responses config). With this knowledge, the 2 test cases we have written should start to make sense. The first expects that our DynamoLister will return 10 tables, the second that it will return 0 tables when we ask only for those prefixed with “dev”.</p><p>With the tests in place, we can go ahead and write a simple DynamoLister class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b8f3411646bc3fcdf01b06b441ea235/href">https://medium.com/media/3b8f3411646bc3fcdf01b06b441ea235/href</a></iframe><p>This very simply invokes the list_tables call, and processes the returned list of table_names, weeding out those which do not match the given prefix. The prefix defaults to all (*) so by default all tables are returned.</p><p>A quick run of rspec will confirm that our tests pass.</p><p>However we can’t in all good conscience leave our test suite here.</p><p>The AWS docs tell us explicitly that:</p><blockquote>The output from <em>ListTables</em> is paginated, with each page returning a maximum of 100 table names.</blockquote><p>This means that, if we ever create more than 100 DynamoDB tables (which, across all environments, is wholly feasible) then our DynamoLister will cease to work. We need to ensure it can handle paginated responses. Fortunately, we can easily figure out what a paginated response might look like, and cause our test suite to return one.</p><p>Recall from above that the list_tables response <em>may</em> also include a last_evaluated_table_name. Also note that the call to the list_tables <em>may</em> include an exclusive_start_table_name and you can probably start to figure out how to deal with paginated responses: if a response includes a last_evaluated_table_name then we must make the call again, passing that value as an exclusive_start_table_name.</p><p>Again, let’s write the test first:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80bd17bf8736fd65504971c9e36ef443/href">https://medium.com/media/80bd17bf8736fd65504971c9e36ef443/href</a></iframe><p>The assignment of a stubbed Aws::DynamoDB::Client instance to @client shows that you don’t <em>have</em> to stub responses globally, you can do so on a per-instance basis. The next thing we do here is to cause our doctored @client object to be returned when we instantiate an Aws::DynamoDB::Client object (i.e. the call to :new).</p><p>Next we tell rspec we expect that @client to receive a call to :list_tables with no arguments, and when it does we are to ‘<em>call_original</em>’. In this case calling original will result in our stubbed list_tables_paginated_response(count: 100)being returned. Lets take a look at this response (which has been added to dynamo_factory.rb):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/563b743a1077cc25aaa89a3bf5b70771/href">https://medium.com/media/563b743a1077cc25aaa89a3bf5b70771/href</a></iframe><p>It is almost identical to the non-paginated response function apart from the addition of a last_evaluated_table_name, which in our case will be set to test_table_100.</p><p>Back in the test above, the next expectation we state is that the @client object will receive a second call to :list_tables, with an exclusive_start_table_name parameter. This time it is to return a standard, non-paginated response of 10 tables. Finally we state that we expect the total number of tables to be returned by the DynamoLister.list_all call will be 110.</p><p>The last thing we need to do is amend the DynamoLister itself to be aware of paginated responses by telling it to loop until last_evaluated_table_name is nil:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81119767e4d5481c3c9dcc13ef89da86/href">https://medium.com/media/81119767e4d5481c3c9dcc13ef89da86/href</a></iframe><p>The last <em>last</em> thing we need to do is to run rspec and check that all of our tests pass. Success!</p><p>The clear benefit to having written this test suite is that we were able to develop the DynamoLister and ensure it worked with over 100 tables <em>without</em> using any AWS resources. But a less obvious benefit is that we were able to completely refactor the code, and still ensure backwards functionality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4a5feb49e9f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-college/stubbing-the-aws-sdk-or-why-testing-rocks-b4a5feb49e9f">Stubbing the AWS SDK, or Why Testing Rocks</a> was originally published in <a href="https://medium.com/devops-college">DevOps College</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>