<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Gremlin - Medium]]></title>
        <description><![CDATA[Find and fix your reliability risks - Medium]]></description>
        <link>https://medium.com/gremlininc?source=rss----94cbb73a7983---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Gremlin - Medium</title>
            <link>https://medium.com/gremlininc?source=rss----94cbb73a7983---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:47:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/gremlininc" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaos Engineering: Necessary, but Not Sufficient]]></title>
            <link>https://medium.com/gremlininc/chaos-engineering-necessary-but-not-sufficient-80fe5c947785?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/80fe5c947785</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[reliability]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[reliability-engineering]]></category>
            <dc:creator><![CDATA[Kolton Andrus]]></dc:creator>
            <pubDate>Mon, 20 Nov 2023 17:50:32 GMT</pubDate>
            <atom:updated>2023-11-20T17:15:37.771Z</atom:updated>
            <content:encoded><![CDATA[<p>I love the ideas behind Chaos Engineering. I’ve been championing them for the last fourteen years. So it pains me to write this article, but I feel like the truth needs to be out there: Chaos Engineering by itself isn’t enough.</p><p>Don’t get me wrong. Fault Injection is a valuable technique. It’s one of the many tools we need as engineers to build and operate complex distributed systems. Knowing how a system fails, knowing where the sharp edges are helps us ‌practice and adapt to real-world situations. By performing this testing before the failure has occurred, we can anticipate and address many potential issues before they become problems.</p><p>Wait, Fault Injection? Don’t you mean Chaos Engineering? Well, that leads us to one of the limitations of Chaos Engineering.</p><h3>Chaos Engineering was created for the cloud</h3><p>That name “Chaos Engineering” was originally chosen because it was exciting — and it fit Netflix’s approach back in the early 2010s.</p><p>Back then, Netflix made a big push to move everything into the cloud. One aspect of the cloud (especially in the early days) is that your servers were ephemeral, i.e. they could be replaced at any time. Prior, a lot of engineers relied on having the same server by storing state locally, writing to disk, etc. This caused a lot of surprise outages when a host disappeared and the replacement wasn’t fungible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*tyCNaZfzBGksVXZ15pNF1w.jpeg" /></figure><p>In a brilliant move by Netflix leadership, they decided to make this the engineer’s problem by introducing Chaos Monkey. If a host might disappear at any time, let’s ensure that it happens in staging and in production! This forced engineers to adapt to the new environment, stubbing their toes while ultimately building more resilient code.</p><p>We can see why Chaos Monkey and, hence, Chaos Engineering are a good fit for this use case. Netflix’s management decided to introduce chaos to mimic the real world and force their engineers to build <strong>for</strong> the chaos in their systems.</p><p>But in doing so, they also introduced another concept. And that idea is still essential to reliability efforts.</p><h3>It’s not about chaos — it’s about reliability</h3><p>A lot of teams don’t want to introduce chaos into their environments. It scares them. They know things will fail — after all, they’re already failing.</p><p>To many, Chaos Engineering is like a Western movie — a lot of shooting from the hip and seeing where things lie when the dust settles. And so those of us that advocate for this practice start from a position of weakness. We’re viewed as agents of mayhem, out to cause pain and hoping to luck upon something valuable along the way, instead of smart engineers making calculated decisions and using precision tools to verify the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOeDNhwt7J35gBjPLkpPKQ.jpeg" /></figure><p>But what Netflix leadership also provided when they introduced Chaos Monkey was permission to work on this problem. By showing engineers it was an important investment of time, they allowed engineers to think about and work on reliability problems, not just shipping features.</p><p>And Netflix wasn’t alone. When I was on the reliability team at Amazon, our status emails went straight to Jeff Bezos. The Amazon team had figured out the dollar cost of reliability outages, and made it a priority with the CEO’s attention. Was it intimidating? Hell, yes. But it also meant that when we found something wrong, we could get it fixed.</p><p>Most companies haven’t provided the signal that reliability is important to the business and to their customers. Therefore using Chaos Engineering to cause failures without the means to correct them is just reckless, inflicting harm without any benefit.</p><p>For ‌teams just beginning this journey, they don’t want chaos. They want testing. They want to systematically verify their understanding of their systems. They want things to behave as they expect, and they want to know if those expectations are invalid.</p><p>We don’t need a lot of chaos to facilitate this. Instead, we should provide these teams with a set of tools to inject failure. We should teach them the potential side effects of different experiments, how to scope it to the right impact, and help them interpret the results. All of this is about mitigating risk, not introducing chaos.</p><h3>Chaos Engineering is one tool in your kit</h3><p>Chaos Engineering is an incredibly effective practice for finding specific risks and verifying specific behaviors. Which is great, but can make it hard to look at the big picture. It’s a wrench in the toolbox when we are discussing how to build a semi-truck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1D5LfVBg62bcmpfVkloyw.png" /></figure><p>There’s nothing better for answering specific questions like, “What happens when foo dependency fails?” or “Can we handle an extra 100ms of latency at our proxy layer?” But it doesn’t answer the question that the business really cares about: “Is our system reliable and will it work when we need it most?” Or the questions end customers care about, like, “Can I buy my groceries?” or “Can I pay this bill?” or “Will my flight be on time?”</p><p>Truth be told, the problems of reliability have grown beyond the scope of Chaos Engineering alone. And that’s a good thing! But it does mean we need to expand our approach to encompass everything reliability means to a business.</p><h3>Reliability programs are missing structure</h3><p>So how should we think about and evaluate the reliability of our systems? Fortunately, we don’t have to reinvent the wheel and can look at other industries for inspiration.</p><h4>Security</h4><p>Reliability risks and security vulnerabilities have a lot in common. A security vulnerability doesn’t mean you’ve been compromised and there’s an active breach — just the potential for one. Same with reliability risks. Sure, everything’s working now, but that risk could cause an outage at any moment.</p><p>So we can look at a couple best practices for security:</p><ul><li>Vulnerability analysis — Are we scanning for known vulnerabilities? Are we validating that we remain resilient to past vulnerabilities?</li><li>Pen Testing — Are we checking what happens when unexpected inputs are provided to our systems? (We tend to think about this in terms of fuzz testing, but what about unexpected load, i.e. DDOS?)</li></ul><h4>QA</h4><p>When you think of testing, where do you go? Probably QA testing. Let’s extend a couple of those ideas to testing for reliability of the system, not just the code you’re about to ship:</p><ul><li>Code Coverage — Have we tested all the things? If we don’t have time to test everything, have we tested the most important interactions?</li><li>Integration/Smoke testing — Are we testing the boundaries between our services? Are we validating our contracts? (In this case, SLAs and error conditions.)</li></ul><p>When it comes to reliability, vulnerability analysis and code coverage are similar to verification testing (like the pre-built Reliability Management Test Suite at Gremlin where you check your systems and code for known failure modes). On the other hand, pen testing and integration/smoke testing have more in common with exploratory testing where you look for the unknown failure modes — a specialty of Chaos Engineering.</p><p>These testing capabilities are then what we would consider part of a good reliability program: a set of processes meant to inform the business as to the current risks, past performance, and future mitigations that need to be taken in order to continue to provide a high quality experience.</p><p>But for this to be successful, it needs teeth. It’s that old idea of trust, but verify. We have a lot of trust and very little verification in this part of our industry. Teams are mostly left on their own to deal with reliability issues, and the most important matter at hand is to triage and restore service, regardless of how much duct tape and bailing wire is used.</p><h3>SREs can’t do it all by themselves</h3><p>Over the past few years, the hip solution has been to throw bodies at the reliability problem in the form of SRE/Ops teams. This approach has a few shortcomings.</p><p>First, it scales with headcount, so isn’t an efficient software answer. Second, SREs can only be experts in so many systems, so often they’re inheriting or operating on ‘Black Boxes.’ As a result, they have limited control. Sometimes they’re able to influence engineering teams to change their practices (e.g. more testing in the CI/CD pipeline, better error handling, circuit breakers, backoffs, etc.) and sometimes they’re left without the leverage or influence to ask those teams to make any code changes.</p><p>This is both a misalignment of concerns and control, leaving most of these teams as a glorified rebranding of the sysadmin role of days past.</p><p>We’re actually running into a deficiency with the SRE model. Don’t get me wrong: SREs are good at their job and have done wonders with what they’ve been given. But that’s the problem. They haven’t been given the organizational tools and authority to really make a huge difference that can work at scale.</p><p>What’s really missing is top down attention directly from the CTO (or other highest technical leader). Too often leadership hires a bunch of SREs, checks the box off, then forgets about reliability until there’s an outage. Under those conditions, how can an SRE get Engineering time to actually fix the problems they find?</p><p>When a CTO makes reliability a priority (and lets everyone know it by holding people accountable), it tells the business that software development cares deeply about the quality of the software that’s written. More importantly, it sets the expectation of highly reliable software. So when a reliability leader surfaces a risk, engineering knows that they should work on it.</p><p>Because the CTO is watching.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*10WmfFtqw3glSErdF3VEcw.jpeg" /></figure><p>When I first joined Amazon, I remember clearly being told, “Every great engineer owns the performance, availability and efficiency of their code. That isn’t some other team’s problem. It’s yours.” This told me early on that, yes, I should account for these problems. If I wrote poor quality code and it failed, I’d be held accountable. And it’s stayed with me to this day.</p><p>So yes, you need someone like SREs to own the testing and processes, but you also need engineers and team leaders to be accountable for the reliability of their systems. And you need leadership to set the standards for reliability so time and resources are actually spent remediating reliability risks. Once the standard has been set, they need tools to have visibility into these risks, the coverage, and the reliability of the system.</p><h3>Chaos Engineering is necessary, but not sufficient</h3><p>Chaos Engineering was designed to uncover the risk. And it’s still very good at that. But it isn’t a process for tracking and reporting on risk in a system.</p><p>You need both to improve reliability.</p><p>The engineer needs a tool and methodology to run experiments, verify behavior and uncover risks. At the same time, the CTO needs the metrics that show them how many risks exist in each deployed system, how many tests were run last week, and whether we’ve fixed what we previously found.</p><p>Without pairing tools with insight, we fall back to hope as a strategy. Without visibility, without accountability, without recognition for the good work done, teams won’t invest the time. And without doing the work, things will break, customers will be disappointed and ultimately, we’ll be back to discussing how we can get out of firefighting mode.</p><p>Which is the whole reason Chaos Engineering was created in the first place.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80fe5c947785" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/chaos-engineering-necessary-but-not-sufficient-80fe5c947785">Chaos Engineering: Necessary, but Not Sufficient</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gremlin’s Gameday: Breaking DynamoDB]]></title>
            <link>https://medium.com/gremlininc/gremlins-gameday-breaking-dynamodb-9c917de2c491?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/9c917de2c491</guid>
            <category><![CDATA[gremlins]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Philip Gebhardt]]></dc:creator>
            <pubDate>Mon, 18 Dec 2017 22:12:08 GMT</pubDate>
            <atom:updated>2017-12-19T01:44:57.898Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6XMWv7F4HP4VMlpTzDdsQ.jpeg" /></figure><p>By now, you might have read previous blog posts on <a href="https://blog.gremlin.com/it-is-gameday-c9b0c37e78a3">running a Gameday</a>. Better yet, your team has run a Gameday and learned something new about their services’ behavior during failure scenarios. But you might wonder, “What happens when Gremlin runs a Gameday on itself?”</p><p>To show you how we do this, we’ve put together a breakdown of our most recent Gameday. Of course, no Gameday would be complete without shattering our own assumptions about our system, so we’ll share some bugs we uncovered in our API service and how we fixed them.</p><p>Before we get into the nitty-gritty, Gremlin believes that the spirit of a Gameday is to challenge assumptions about a given service or distributed system through concrete discussion and thoughtful experimentation. Gamedays are not just for large organizations, or for dedicated SRE teams. Every team holds assumptions about a system’s behavior, and Gamedays are a useful tool for testing the assumptions you’re aware of, AND the ones you took for granted.</p><p>Assumptions aren’t all bad. Most of the time, they help us reason through our daily tasks without getting bogged down by a system’s complexity. But sometimes making the wrong assumption can blind us to problems that rear their heads during a 3am outage.</p><h3>We want everyone to join in</h3><p>We open our Gamedays up to the whole company (this is of course easier at a startup than a larger company). Inclusivity increases the diversity of perspectives, and expands the number of unique observations about our systems. Don’t be afraid to let wider teams participate.Everyone has a role to play when it comes to observing system behavior and identifying trends, and some of Gremlin’s team members have participated in successful Gameday events with hundreds of participants.</p><h3>Gremlin’s Gameday — Let’s Break DynamoDB</h3><p>Gremlin uses DynamoDB for its persistence layer for all stateful interactions with the API, including client operations that deal with launching attacks, updating templates, and registering new clients. DynamoDB is a solid choice for our elasticity requirements (we love autoscaling!), but this crucial dependency means any failures translate to wide-reaching impacts on Gremlin — and although Amazon built DynamoDB on top of years of experience with distributed hash tables, <a href="https://www.theregister.co.uk/2017/09/15/aws_brownout/">no system is too scalable to fail</a>. We want to make sure we fail gracefully, preventing inconsistencies in state and ensuring we are transparent to our users.</p><h3>Let’s break stuff</h3><p>The cadence of our experiments follow the “Chaos Experiment Lifecycle,” which is all about turning our assumptions about a system into hypotheses and then carrying out experiments that test them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p08K1QVVZF8OqNzYee-fgw.png" /></figure><h4>Begin with a Hypothesis</h4><p>At Gremlin, we formulate these hypotheses through a brainstorming session around “<a href="https://blog.gremlin.com/your-first-chaos-experiment-321ec4468bce#7401">What could go wrong?</a>” This session takes a lot of time. If you have tooling to help inject failures (like they are at Gremlin), the discussions, whiteboarding, and note-taking can be the most time-consuming part of a Gameday. That’s ok though because it’s also the most valuable artifact, partly because so many questions get presented to the whole team:</p><ul><li><em>So what do we think will happen when DynamoDB is unreachable? What does unreachable mean?</em></li><li><em>What sort of Exceptions do we expect our SDK to throw? Are we catching those explicitly?</em></li><li><em>What sort of API response is going to come back from these failures? What will the UI look like when displaying these responses?</em></li></ul><p>The biggest output of the discussion isn’t just talking, it’s a written record of outcomes (<a href="https://docs.google.com/spreadsheets/d/1tbs-B1LaQQM9oTqsUB_g4CJo9bzQBbvog6EgkHjWwjM/edit#gid=1413228070">Fill out the chart</a>). For example, here’s the output on our discussion around “What can go wrong when DynamoDB is unreachable?”</p><ul><li><strong>Failure Scenario:</strong> DynamoDB is unreachable</li><li><strong>Method:</strong> Run a <a href="https://help.gremlin.com/attacks/#network-gremlins">Blackhole Gremlin</a> on the EC2 instances hosting our API (drop all outgoing packets)</li><li><strong>Scope:</strong> Blackhole should impact port egress traffic to dynamodb.us-west-2.amazonaws.com:443</li><li><strong>Measurement:</strong> Datadog should report drop in requests per second, spike in HTTP 5XX errors</li><li><strong>Abort Conditions:</strong> We should not exceed 5% of API failures. In the event that we do, whoever is on call should get paged immediately. (The expectation that our on-call will get paged is another assumption that we’re testing)</li><li><strong>Abort Procedure: </strong>Stop the effect immediately — halt the Blackhole Gremlin</li><li><strong>Expected Measurements (hypothesis):</strong> Increase in HTTP 5XX errors due to DynamoDB client timeout errors</li><li><strong>Expected Observations (hypothesis): </strong>details around timeouts should be returned from the API responses</li></ul><h4>Experiment</h4><p>As stated above, we want everyone participating in the Gameday to have a role. So in addition to our hypothesis discussion, we also decide what real-time tests we want to carry out during the failure scenarios and who will be responsible for each test.</p><ul><li>UI Engineer: login/logout behavior</li><li>Client Engineer: client registration behavior from the CLI</li><li>API Engineer: API response times and failure metrics in datadog</li><li>Security Engineer: authentication workflow</li></ul><p>When we apply the DynamoDB failure scenario, we want to do so in well-defined durations, starting with something short-lived and increasing duration when we need to observe an effect for a longer period of time. This allows us to iterate quickly, validating correct behaviors and dialing up our blast radius to yield more results until we discover something novel about our system’s behavior. I.e., we discover a coupling that we can resolve!</p><p>We also started our Blackhole experiments against a single instance of our API server for 60 seconds, which yielded intermittent failures across all of our predefined tests as we expected.</p><h4>Wait, that’s not supposed to happen</h4><p>The interesting behavior only started happening once we applied impact on an entire availability zone for 10 minutes. Only after we had a large enough blast radius to fail hundreds of requests, our API autoscaling group started to behave unexpectedly: impacted EC2 instances started terminating due to ELB health check failures.</p><p>We designed the API health check specifically not to touch DynamoDB during each invocation. So it came as a surprise to learn that our ELB was encountering health check failures when DynamoDB couldn’t be reached. Potential reasons for this unexpected behavior ranged from nasty bugs to accidental features:</p><ul><li><em>Did we mistakenly blackhole traffic to/from our load balancer?</em></li><li><em>Did we accidentally introduce a dependency on DynamoDB in our health check?</em></li><li><em>Is this a feature or a bug? Don’t we want our load balancer to consider these instances unhealthy anyway?</em></li></ul><p>This health check behavior also came with a critical side effect. In most cases, the ELB health check failures would occur faster than the DynamoDB timeout failures, and so our instances were terminating faster than DynamoDB error events could be ingested into our alerting and monitoring systems. This means we weren’t getting alerted when our database was unreachable!</p><h4>Resolve</h4><p>At this point, we’re ready to call end-of-impact and shift focus to getting to the bottom of this unexpected behavior. If your Gameday has other experiments to run, this would be a good time to write down notes and move on to the next experiment. Issues like the one we found should be taken offline and remediated so that we can run the same experiment again come next Gameday.</p><p>As it turns out, our DynamoDB timeouts were indirectly causing ELB HTTP health check failures through <em>thread exhaustion</em>. Specifically, all API calls which depend on DynamoDB were spending so much time waiting for a connection that they occupied all available threads used by our HTTP servers. When this happens, any new requests (including health checks) get queued <em>indefinitely</em> until threads become available, which in our case is when calls to DynamoDB eventually timeout and free up.</p><p>Failing these database requests sooner should alleviate pressure on our servers, returning errors quickly so they show up as alerts as fast as possible instead of indirectly breaking health checks. But not all of Gremlin’s database operations fit this fail-fast requirement. We have larger database calls that run as batch jobs in background threads. These calls have a much higher cost when they fail and are completely independent of incoming HTTP requests, and so we want to give them as much time as reasonably possible to do their job. This means we need at least two profiles for our database timeouts.</p><p>The Java SDK for DynamoDB uses a default execution timeout of 30 seconds, which is an eternity when considering using DynamoDB as the persistence layer for an API server (this timeout is actually a bit longer than 30 seconds due to the SDK’s built-in backoff and retry algorithms). On the other hand, 30 seconds is reasonable for large queries or batch updates. We’ll gladly spend that time (especially in a background thread) if it means all of our data is processed. In both cases however, we set a very short <em>connection timeout </em>of 500ms. Whether we’re doing real-time or batch processing, we want to be alerted when connections start taking longer than a half-second to establish.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77fa9ea05fe727019357a34927ced1bd/href">https://medium.com/media/77fa9ea05fe727019357a34927ced1bd/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6672a54f771f71beb90854ea704d0b49/href">https://medium.com/media/6672a54f771f71beb90854ea704d0b49/href</a></iframe><h3>Looking ahead to the next Gameday</h3><p>This experiment will be one of the first on the list to rerun during Gremlin’s next Gameday so that we avoid drifting back into this strange health check behavior. We’ll use the hypothesis we’ve written down along with the rest of our notes, and start experimenting again from the top. Gremlin’s Schedules feature as well as its REST API also allow us to make sure we run this experiment regularly, even when we’re not running Gamedays.</p><p>This is the first in a series of posts about running Gamedays at Gremlin. We have so much ground to cover on this topic (like “What happens to Gremlin users when we run Gamedays?”), not to mention much more to discover about our own systems as we break them. Send us your ideas for Gameday content and check back here soon for new posts!</p><p>If you have any questions about Gamedays or would like Gremlin to work with your team on a Gameday of your own, join the <a href="https://slofile.com/slack/chaosengineering">Chaos Engineering Slack</a> and we’ll help you get started! .</p><p>Happy Gameday!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c917de2c491" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/gremlins-gameday-breaking-dynamodb-9c917de2c491">Gremlin’s Gameday: Breaking DynamoDB</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Gremlin: Orchestrating Chaos]]></title>
            <link>https://medium.com/gremlininc/introducing-gremlin-orchestrating-chaos-b137b74f2371?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/b137b74f2371</guid>
            <category><![CDATA[press-release]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Kolton Andrus]]></dc:creator>
            <pubDate>Tue, 12 Dec 2017 15:19:09 GMT</pubDate>
            <atom:updated>2017-12-12T16:23:24.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hhDFx_TowPaHMqcohRD6Rg.png" /></figure><p>Today is an exciting day for our team at Gremlin. After nearly two years in the making, we’re proud to share that Gremlin’s “Resilience as a Service” is publicly available! We’re also excited to announce our $7.5M Series A funding, led by Index Ventures and Amplify Partners, as well as several customers leveraging Gremlin to increase their system’s resilience, including <a href="https://www.twilio.com/blog/2017/11/chaos-engineering-ratequeue-ha.html">Twilio</a>, Expedia, Confluent, and Remind.</p><p>Our team has “grown up” on call, carrying a pager. At Amazon and Netflix, I served as a Call Leader, responsible for managing and resolving high-pressure incidents to keep our services up. We know the pain of waking up late at night to find and fix critical issues. And we’ve found a better way — preventing them from occurring in the first place.</p><p>Gremlin exists to help companies prevent downtime before it happens.</p><p>The solution is to use chaos engineering to break things on purpose, rather than waiting for things to go wrong. Chaos Engineering — <a href="https://blog.gremlin.com/the-discipline-of-chaos-engineering-e39d2383c459">thoughtful, planned experiments to introduce failure</a> into our systems to identify and fix unknown faults — helps us build robust and resilient systems. Think of it like a flu shot for your application.</p><p>While the roots of chaos engineering have been around for over a decade, and many great open source tools have helped pioneer the way, the industry needs more. At Gremlin, we’ve solved this problem before, so we’re able to provide exactly what engineers need to safely and securely execute these experiments, and then to automate them. We’ve also made this simple to do, so engineers can be up and experimenting in under five minutes.</p><p>We’re helping companies anticipate and mitigate failure before it happens.</p><p>Supporting a dozen different <a href="http://www.gremlin.com/product/">attack vectors </a>across three categories, Gremlin allows you to simulate a wide range of infrastructure failures. Recreate real-world incidents from datacenter and cloud provider outages to resource exhaustion and network issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*5fuGkXwp4szbQnM6PD2u2g.gif" /></figure><p>Built with safety in mind, Gremlin provides you with an ‘undo’ button and automatically cleans up if things go wrong. Security is built in from the ground up using least permissions, multi-factor authentication, auditing, and role based access control.</p><p><strong>Gremlin is a tool built by engineers for engineers who carry the burden of keeping the internet online.</strong></p><p>Behind the technology is a team who pioneered chaos testing at companies like Amazon, Netflix, Salesforce, and Dropbox, as well as <a href="https://blog.gremlin.com/and-now-for-something-completely-different-ecf678473d37">security</a> from RSA.</p><p>Today more than a dozen companies are using Gremlin to perform chaos experiments and build more resilient systems. More than providing a tool, we’re partnering with companies to teach this new discipline and help make the internet more reliable for all of us.</p><p>Get started with Chaos Engineering by running a <a href="https://blog.gremlin.com/it-is-gameday-c9b0c37e78a3">Gameday</a>, or requesting a <a href="http://www.gremlin.com/contact">demo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b137b74f2371" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/introducing-gremlin-orchestrating-chaos-b137b74f2371">Introducing Gremlin: Orchestrating Chaos</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[And now for something completely different…]]></title>
            <link>https://medium.com/gremlininc/and-now-for-something-completely-different-ecf678473d37?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/ecf678473d37</guid>
            <category><![CDATA[auditing]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Frederic Bull]]></dc:creator>
            <pubDate>Mon, 20 Nov 2017 18:01:01 GMT</pubDate>
            <atom:updated>2017-11-20T19:24:01.175Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*Z1_xqdFPNze0ycguIv8eWA.jpeg" /></figure><p>We wanted to take a break from our usual espousing of the value, benefits, and methods of Chaos Engineering and talk about something else near and dear to our hearts, security. We’ve been working behind the scenes to bring you new features and improvements that we’re just dying to share!</p><h3>Auditing &amp; Penetration Testing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*e78NXwaYEwFmUNgscTJhXA.jpeg" /></figure><p>Gremlin has always taken the security of our systems and infrastructure very seriously. Security is a pillar of our philosophy and an integral part of everything we do. But it’s not enough to think we’re accomplishing our goals and be satisfied with our own internal auditing and review efforts. Much like with chaos engineering, the efficacy of our systems must be rigorously tested in the real world, by real threat actors, before we can put faith in these systems and run them in production. To that end we partnered with a security firm, Bishop Fox, to perform an extensive security audit and penetration testing against all three components that make up the Gremlin service: Client, API, and Website App. We’re happy to say that after working with their team, we have passed with flying colors. Our auditors did their job well and identified several vulnerabilities which we quickly remediated. Those remediations underwent re-testing and were certified as fixed by our auditors (report available upon request).<br>Our experience working with an external security auditor has been invaluable. We will continue to have them regularly performed!</p><h3>Client Hardening</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*8LLiXtYIGb5KsGJ-tgKFMQ.png" /></figure><p>One of the results of our external audit experience was adding additional hardening to the Gremlin client. The Gremlin client has always had security at the heart of its design. From its Least Privileges principals, requiring only the permissions absolutely necessary to impart its impact, to its use of strong encryption and ephemeral keys. We’ve taken this a step further by locking down the binaries Gremlin depends on so we can be assured they have not been modified or redirected by a nefarious actor. We will continue to identify additional hardening that can be applied to Gremlin to ensure your experience is safe and secure!</p><h3>Multi-factor Authentication</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*OzI2YKTAKoaeYHtEyl0mBg.jpeg" /></figure><p>Passwords alone have always been an inherent weakness when authenticating users. Good passwords are difficult for people to remember, and those that are easy to remember are more often easy for a computer to guess. So we’ve added the ability to enable a second authentication factor using a Google-compatible authenticator (6 digit time-based tokens). This feature ensures that even if someone were to guess or remotely steal your password, your account will remain secure. Multi-factor authentication is available to all users and can be required for all users in your company by enabling the <em>force MFA</em> option available to super-users.<br>Gremlin recommends using an authenticator that cannot export their secret keys, such as the <a href="https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&amp;hl=en">Google Authenticator</a>. For more information visit our <a href="https://help.gremlin.com/user-management/#two-factor-authentication-mfa">help documentation</a>.</p><h3>SAML SSO Authentication</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*oKCz5Fp-kPVmAkXv2Tje2Q.png" /></figure><p>For those customers that desire more control over authentication and/or tighter integration with their existing identity management system we now offer SAML authentication. SAML allows your organization to retain strict control over how your users authenticate with Gremlin. In order to utilize SAML, your organization must have an identity provider that supports SAML SP initiated logins such as Google, AWS, Active Directory Federated Services (ADFS), OneLogin, or others (for a list see: <a href="https://auth0.com/docs/protocols/saml/samlp-providers">https://auth0.com/docs/protocols/saml/samlp-providers</a>). We are excited about the possibilities that SAML brings for the future and plan to continue expanding our support to include authorization controls.<br>For more information visit our <a href="https://help.gremlin.com/security/">help documentation</a>.</p><p>As always, if you have questions or concerns please reach out to us at <a href="/service/mailto:security@gremlin.com">security@gremlin.com</a> or visit us on <a href="http://tinyurl.com/chaoseng">Slack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecf678473d37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/and-now-for-something-completely-different-ecf678473d37">And now for something completely different…</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Chaos Experiments to Start With]]></title>
            <link>https://medium.com/gremlininc/4-chaos-experiments-to-start-with-cbb93e684730?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/cbb93e684730</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[resilience]]></category>
            <dc:creator><![CDATA[Matt Jacobs]]></dc:creator>
            <pubDate>Fri, 03 Nov 2017 16:01:01 GMT</pubDate>
            <atom:updated>2017-11-03T22:37:25.484Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ej_AuQBoknWL7S781ZwRIw.png" /></figure><p>The transition from learning about Chaos Engineering to practicing it can be difficult. Chaos Engineering as a concept has been around for a while now, and plenty of talks, papers, and books have talked about why it’s becoming essential to making distributed systems resilient and stable.</p><p>Introducing chaos to increase resilience seems like a contradictory concept at first. It’s important to recognize that <a href="https://blog.gremlin.com/the-discipline-of-chaos-engineering-e39d2383c459">Chaos Engineering is an engineering <em>discipline</em></a> where you carefully use experiments of isolated and controlled system failure as a method to identify couplings in your system, which will allow you to identify how to increase resilience.</p><p>The <a href="https://blog.gremlin.com/your-first-chaos-experiment-321ec4468bce">ideal chaos experiment</a> will introduce behavior into your system that is both likely to occur “in the wild”, and impacts core business functionality. This article provides a few examples of high-value chaos experiments that reproduce common outages we’ve experienced. They should be a good starting point for your chaos experimentation. In each experiment, I’ve included a “Gremlin Recipe”, if you want to use <a href="https://gremlin.com/">our tool</a> to conduct the experiment.</p><h3>Experiment 1: Resource Exhaustion</h3><p>Resources on computers are finite. A machine/VM/container will inevitably hit a resource limit at some point, and the application will be forced to handle the lack of a resource. Commonly, this is CPU, Memory, or I/O.</p><p>We can reproduce CPU exhaustion by conducting a chaos experiment. Running this experiment will consume CPU cycles, leaving the application with the same amount of customer-facing work, and less CPU to do it with. As always, we advocate starting small on a single instance, then increasing the blast radius as confidence grows. Common reactions to CPU exhaustion are an increase in errors and latency and a reduction in successful requests to customers.</p><h4>Gremlin Recipe</h4><ul><li><strong>Attack</strong>: CPU / Memory / Disk</li><li><strong>Scope</strong>: single instance</li><li><strong>Expected Results: </strong>Rate of good responses goes down, errors increase at all layers, <a href="https://en.wikipedia.org/wiki/Brownout_(software_engineering)">brownout mode </a>entered (if implemented), alerts fire (if configured at single-instance level), load balancer routes traffic away (if applicable)</li></ul><h3>Experiment 2: The Network is Not Reliable</h3><p>Network dependencies are a fact of life in a distributed system, and as distributed systems are growing in adoption AND complexity, chaos engineering becomes an optimal way to test for potential failures on the path to increasing resilience.</p><p>We know from the <a href="http://wiki.c2.com/?EightFallaciesOfDistributedComputing">Fallacies of Distributed Computing</a> that the network is unreliable. This implies that the success of our overall system will be determined by how many network calls we make, and how much we can shield the customer from the failures that do occur.</p><p>Applications generally have both internal and external network dependencies. Here, I’m defining internal dependencies to be those systems that are under our organization’s control. Someone is carrying a pager for that system and they work for the same company I do. We can (and should) expect that internal teams try to maintain availability, and use the proper tools and channels to notify them if they don’t.</p><p>When you do have a dependency that is out of your control, it’s critical to understand how the system reacts when it is unavailable. This is a network blackhole Chaos Experiment and will make the designated addresses unreachable from your application. Once you’ve applied the blackhole, you should check that your application starts up normally and is able to serve customer traffic without the dependency.</p><h4>Gremlin Recipe</h4><ul><li><strong>Attack</strong>: Network Blackhole / Latency</li><li><strong>Scope</strong>: single instance</li><li><strong>Expected Results: </strong>Traffic to dependency goes to 0 (or gets slow), startup completes without errors, application-level metrics in steady-state are unaffected, traffic to fallback systems shows up and is successful, dependency alerts and pages may fire (if scoped to single-instance)</li></ul><h3>Experiment 3: Datastore saturation</h3><p>All interesting applications have some sort of storage, and managing the relationship between application and datastore is critical to overall system health. There are a variety of ways that an application may overwhelm a data store (poor queries, lack of indices, bad sharding, upstream caching decisions, etc), but all of them result in what appears to be an unresponsive data layer.</p><p>It’s important to understand how datastore saturation manifests in your application. There are a few ways of modeling this with a Chaos Experiment. You can blackhole your datastore, making it appear completely unavailable. You can add latency to requests to your datastore, making it appear slow. Finally, you can consume I/O bandwidth to simulate a congested path to the datastore.</p><p>Since the datastore is a critical dependency, you should expect that some features of your application are slow or unavailable. Ideally, only the set of features backed by the datastore are affected. If the impact is wider than you expect, you should investigate for a hidden dependency on the datastore. This is also a great opportunity to tune timeouts to the datastore and test that they’re cutting off requests as you expect them to.</p><h4>Gremlin Recipe</h4><ul><li><strong>Attack</strong>: Network Blackhole / Latency / IO</li><li><strong>Scope</strong>: single instance</li><li><strong>Expected Results: </strong>Traffic to datastore is reduced or slower, application-level metrics in steady-state are unaffected to the degree possible, traffic to fallback systems shows up and is successful, timeouts and concurrency limits kick in when appropriate, alerts and pages may fire (if scoped to single-instance)</li></ul><h3>Experiment 4: DNS Unavailability</h3><p>It’s easy to forget the critical role that DNS plays in keeping our systems running. Many companies experienced customer-facing issues when a <a href="https://dyn.com/blog/dyn-statement-on-10212016-ddos-attack/">real-world DNS failure</a> occurred in October 2016. Because a failure like this is relatively rare, getting a recovery plan together is challenging.</p><p>The best way forward is to induce a DNS outage and understand how your application behaves. If you blackhole DNS traffic on a single instance, it will appear to that instance as if DNS is unavailable. The fixes will vary depending on the issue, but common solutions are to pass around IP addresses instead of hostnames for internal addressing and the use of a backup DNS provider. Once you’ve run this experiment, an exercise for the reader is to then consider what sort of damage an outage of an internal service discovery tool (like <a href="https://coreos.com/etcd/">etcd</a>, <a href="https://github.com/Netflix/eureka">Eureka</a>, <a href="https://www.consul.io/">Consul</a>, etc) would cause.</p><h4>Gremlin Recipe</h4><ul><li><strong>Attack</strong>: DNS blackhole</li><li><strong>Scope</strong>: single instance</li><li><strong>Expected Results: </strong>Inbound traffic may drop, traffic to external systems may fail, startup may not complete successfully</li></ul><h3>Next Steps</h3><p>This set of chaos experiments should help you get started. We’ll be discussing more sophisticated chaos experiments in future articles, but in the meantime, we’d love to hear about other interesting tests you’ve run or any chaos success stories you have. We’re active on <a href="http://tinyurl.com/chaoseng">Slack</a> and we look forward to continuing the conversation there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbb93e684730" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/4-chaos-experiments-to-start-with-cbb93e684730">4 Chaos Experiments to Start With</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Won’t Get Fooled Again]]></title>
            <link>https://medium.com/gremlininc/wont-get-fooled-again-95a3f2bb0a2e?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/95a3f2bb0a2e</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[gremlins]]></category>
            <category><![CDATA[resilience-testing]]></category>
            <category><![CDATA[continuous-chaos]]></category>
            <dc:creator><![CDATA[Matthew Fornaciari]]></dc:creator>
            <pubDate>Mon, 02 Oct 2017 17:21:13 GMT</pubDate>
            <atom:updated>2017-10-02T17:21:13.133Z</atom:updated>
            <content:encoded><![CDATA[<h3>Oh Noes! The Sky is Falling!</h3><p>It was relatively quiet, late one Friday afternoon, with many of the Gremlin employees gearing up for a long weekend filled with festivities, when all of a sudden alarms started blaring. Our service had decided to start serving `<strong>5xx</strong>` errors, more or less indiscriminately, just as everyone was getting ready to call it quits. I’m sure you can imagine our surprise, considering nobody was monkeying with the database, pushing code to production or doing anything of real consequence. Nevertheless, our service was on fire, and in desperate need of a little TLC, so we spun up a call and hopped on the offending instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bkJzceh1iCg1rJ6Dg8BUqQ.jpeg" /></figure><p>After a relatively brief bout of log diving <a href="https://medium.com/u/b90d8e0da301">Philip Gebhardt</a> was able to suss out the issue: log rotation wasn’t working properly and as a result the disk had run out of space. <strong>Face, meet palm</strong>. It seemed ludicrous that something so trivial could go unnoticed to the point where it interrupted our blissful slide into the weekend. We rotated the instances in the Auto Scaling Group and made a promise to ourselves that we wouldn’t get bit by this again.</p><h3>Fool Me Once</h3><p>Now I’ve made similar promises before, just to watch them fall by the wayside as priorities shifted. But at Gremlin, we vehemently believe in <strong>never</strong> failing the same way twice. After all, how would it look if we weren’t willing to practice what we preach?</p><p>Given our profound level of combined experience in systems engineering, it’s no surprise we’re huge proponents of the <a href="https://en.wikipedia.org/wiki/Unix_philosophy">unix philosophy</a>. As such, each Gremlin is a single-purpose task, meant to replicate a real world failure mode. Since disk exhaustion was an unprecedented failure mode for us, <a href="https://medium.com/u/fd7863d18421">Matt Jacobs</a>, our newest engineer, found himself with an initiation task, writing the Disk Gremlin (seen below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SoqL87ZqJ2y5kQMm." /><figcaption>Disk Gremlin Configuration</figcaption></figure><p>Instead of sitting back on our haunches and crossing our fingers that our newly created mitigation steps would work when called upon, we went proactive and decided to make sure there was no way they wouldn’t.</p><h3>What TODO?</h3><p>Easier said than done, right? It doesn’t have to be. Ensuring you don’t suffer from the same failure mode more than once boils down to four main steps.</p><ol><li><strong>Experiment</strong> — During the experimentation phase, we toyed with several different approaches to ensuring that a full disk doesn’t mean an angry API. We ultimately landed on publishing custom disk utilization metrics to CloudWatch (via <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html">mon-scripts</a>) and triggering an EC2 Stop Instance Event when disk utilization is approaching 100%.<br><strong><em>*NOTE*</em></strong> <em>Of course, this still leaves us susceptible to bugs that constantly overload our disks, so throwing an alarm to engage human investigation after several successive rotations was layered on as a catch all.</em></li><li><strong>Implement</strong> — We then baked the monitoring and alarming into our build process, to ensure that all future deployments come with full insight into disk utilization.</li><li><strong>Validate — </strong>After our first deploy, I ran a one-off attack using the newly created Disk gremlin, to ensure that the alarm fired, the service instance terminated and that the ASG replaced it with a shiny new instance, as expected.</li><li><strong>Automate </strong>— But the real magic lies in turning the one-off attack into a scheduled attack which runs daily, at random, against our production service. During business hours of course. In doing that, we ensure that we don’t become susceptible to a full disk as our code base and architecture continue to mutate. If we ever suffer a regression, the Disk gremlin will proactively trigger said alarms and we’ll be able to pinpoint the root cause of the regression.</li></ol><h3>Continuous Chaos</h3><p>You may ask if that truly frees a system from <a href="https://www.amazon.com/Drift-into-Failure-Components-Understanding/dp/1409422216">drifting into failure</a>, ultimately ensuring we never see the same failure mode twice? The answer, for the most part, is yes. Incrementally building a knowledge base of failure modes, creating tests that exercise these situations and then constantly executing them against your system is akin to regression testing and Continuous Integration, but with a slightly different goal in mind. Instead of testing in a perfect world, where network calls respond instantly and resources are never in contention, you’re simulating the real world, where chaos ensues and entropy abounds. We call it Continuous Chaos, and it’s not all that different from many best practices you’ve likely already adopted.</p><p>I’m sure some of you are thinking “<em>That’s all well and good Forni, but not all of us can regularly be slowing down network connections or pushing our disk to 95%, and </em><strong><em>DEFINITELY</em></strong><em> not in production</em>” to which I say, “<em>Why not?</em>” It’s going to happen one day, one way or another. Instead of dealing with disaster as you’re counting down the minutes to a long weekend, wouldn’t you rather it be on your own terms?</p><p>Interested in unleashing a little hell at your company?</p><p>Feel free to shoot us an <a href="/service/mailto:sales@gremlininc.com">email</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95a3f2bb0a2e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/wont-get-fooled-again-95a3f2bb0a2e">Won’t Get Fooled Again</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s Gameday]]></title>
            <link>https://medium.com/gremlininc/it-is-gameday-c9b0c37e78a3?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/c9b0c37e78a3</guid>
            <category><![CDATA[antifragile]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Kolton Andrus]]></dc:creator>
            <pubDate>Tue, 11 Jul 2017 16:47:45 GMT</pubDate>
            <atom:updated>2017-07-11T17:35:21.018Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s here, Gameday has arrived. You’ve prepared diligently — studied the <a href="https://blog.gremlininc.com/the-discipline-of-chaos-engineering-e39d2383c459">principles</a>, accepted that failure <em>will</em> happen, injected <a href="https://blog.gremlininc.com/your-first-chaos-experiment-321ec4468bce">failure</a> yourself. You are ready to prove that you have what it takes — to survive an outage in production. It’s time to run a Gameday.</p><h4>Practice makes perfect</h4><p>Gamedays are like fire drills — an opportunity to practice a potentially dangerous scenario in a safer environment. They are the capstone which allows us to measure the resilience of a system. Running a Gameday tests our company — from engagement to incident resolution, across team boundaries and job titles. It verifies the system at scale, ultimately in production. By proactively testing these events, we can choose the terms of engagement — the time, the place and the root cause.</p><p>The first step to any drill is knowing what to practice. Are you evacuating one of your data centers or cloud provider’s regions? Validating that the loss of a key dependency won’t bring you down? Planning out the Gameday is a great opportunity to collaborate with other parts of the company, share context, and learn about the system as a whole.</p><h4>Communicate</h4><p>When in doubt, over communicate. Especially when you’re going to be breaking things. Let everyone interested or involved know your plan. Have a <em>Command Center — </em>a chat room, conference bridge, or a conference room — where anyone can check on the status of the Gameday. Share your success and abort criteria, as well as any dashboards you are watching. Having many eyes on the problem can speed up detection if things go wrong.</p><h4>What happens if things go wrong?</h4><p>Always have a rollback plan and a set of abort criteria. Often if there is any customer facing impact (beyond what is expected), then the impact is reverted and an investigation begins. Monitoring, alerting, and engagement are key parts of your system to verify.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TpJc9X2D_kx8viE8IWQaFg.jpeg" /><figcaption>Blast Radius — The potential impact from a failure test.</figcaption></figure><h4>Start small</h4><p>It is critical to understand and minimize the <em>blast radius</em> of an exercise. Run your Gameday first in a test or staging environment. Start with the smallest blast radius that will teach you something about your system. This may be breaking a single container, degrading a single instance, or injecting failure into a single request. Next fail the entire service, zone, or a percentage of requests. At each step you either gain confidence in your system or find an issue which needs to be fixed.</p><h4>Dial it up</h4><p>There is a benefit to starting small and dialing it up — different scales teach us different characteristics of our system. At small scale we test the functional: Do we handle exceptional cases correctly? Is our system usable in a degraded state? At large scale we learn about resource constraints and cascading failure: Do we protect ourselves if traffic builds up? Are our timeouts set aggressively enough? Only by testing the small <strong>and</strong> the large scale will we be prepared for what will occur in the real world.</p><h4>Run in Prod</h4><p>The end goal of any Gameday is to run in Production.</p><blockquote>Fail services regularly. Take down data centers, shut down racks, and power off servers. Regular controlled brown-outs will go a long way to exposing service, system, and network weaknesses. Those unwilling to test in production aren’t yet confident that the service will continue operating through failures. And, without production testing, recovery won’t work when called upon.</blockquote><blockquote>James Hamilton, <a href="https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/hamilton_html/index.html">On Designing and Deploying Internet-Scale Services</a></blockquote><p>It’s production that matters at the end of the day. That’s where customers live, where the money is made. It’s production’s configuration that counts if things go wrong. Many systems are tuned for the ‘happy-case’, and find themselves woefully unprepared when failure strikes.</p><h4>Train your team</h4><p>Furthermore, when failure <em>does strike,</em> there is little time for learning during the event. Don’t train your on-calls by handing them a pager and wishing them good luck. Teams need to proactively test their reactive skills! By regularly testing important scenarios, your teams will build muscle memory and be able to act quickly and confidently in a crisis. De-mystify and de-stress your incidents by practicing in advance!</p><h4>Start today!</h4><p>At <a href="https://www.gremlininc.com/">Gremlin</a>, we believe strongly in testing in production. In order to do that, we’ve built a system that makes it safe, simple and secure to run Gamedays. You can test a wide range of failure scenarios out of the box! Contact <a href="/service/mailto:sales@gremlininc.com">sales@gremlininc.com</a> today to get started.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9b0c37e78a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/it-is-gameday-c9b0c37e78a3">It’s Gameday</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your First Chaos Experiment]]></title>
            <link>https://medium.com/gremlininc/your-first-chaos-experiment-321ec4468bce?source=rss----94cbb73a7983---4</link>
            <guid isPermaLink="false">https://medium.com/p/321ec4468bce</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[failure]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Kolton Andrus]]></dc:creator>
            <pubDate>Thu, 01 Jun 2017 18:06:20 GMT</pubDate>
            <atom:updated>2017-06-02T15:25:00.688Z</atom:updated>
            <content:encoded><![CDATA[<p>OK, so you’ve decided that <a href="https://blog.gremlininc.com/the-discipline-of-chaos-engineering-e39d2383c459">Chaos Engineering</a> sounds like a good idea. How do you get started? We get that question a lot, and we wanted to outline some tips for implementing these practices in your environment.</p><h4><strong>Chaos</strong></h4><p>†A quick aside. Chaos is a cool name, but it is a misnomer in the best way to approach failure testing. Sometimes a design decision like enabling <a href="https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey">Chaos Monkey</a> in a new environment can be a great way to enforce realistic constraints on teams operating there. It can be a bit daunting however to apply a random strategy when dealing with an existing environment. We think the best way to get started is a thoughtful, planned experiment to validate expected behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*aJcEHSPLZOub8o2jJzycbA.jpeg" /><figcaption>Good news everyone! We’re going to run an “Experiment”!</figcaption></figure><h4><strong>Planning your First Experiment</strong></h4><p>One of the most powerful questions in Chaos Engineering is <em>“What could go wrong?”</em>. By asking this question about our services and environments, we can review potential weaknesses and discuss expected outcomes. Similar to a risk assessment, this informs priorities about which scenarios are more likely (or more frightening) and should be tested first. By sitting down as a team and white-boarding your service(s), dependencies (both internal and external), and data stores, you can formulate a picture of “<em>What could go wrong?</em>”. When in doubt, injecting a failure or a delay into each of your dependencies is a great place to start.</p><h4><strong>Creating a Hypothesis</strong></h4><p>You’ve got an idea what can go wrong. You’ve chosen a scenario — the exact failure to simulate — to inject. What’s happens next? This is a excellent thought exercise to work through as a team. By discussing the scenario, you can hypothesize on the expected outcome when running live. What will be the impact to customers, to our service or to our dependencies?</p><h4><strong>Measuring the Impact</strong></h4><p>In order to understand how your system behaves under duress, you need to measure it. It’s good to have a key performance metric that correlates to customer success (such as orders per minute, or <a href="https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a">stream starts per second</a>). As a rule of thumb, if you ever see an impact to these metrics, you want to halt the experiment immediately. Next, is measuring the failure itself, you want to verify (or disprove) your hypothesis. This could be the impact on latency, requests per second, or system resources. Lastly, you want to survey your dashboards and alarms for unintended side effects.</p><h4><strong>Have a Rollback Plan</strong></h4><p>Always have a plan in case things go wrong. Know going in that sometimes even the backup plan can fail. Talk through the ways in which you’re going to revert the impact. If you’re running commands by hand, be thoughtful not to break ssh or control plane access to your instances. One of the core aspects of <a href="http://www.gremlininc.com/">Gremlin</a> is safety. All of our attacks can be reverted, allowing you to safely abort if things go wrong.</p><h4><strong>Go fix it!</strong></h4><p>After running your first experiment, hopefully, there is one of two outcomes. You’ve verified either that your system is resilient to the failure you introduced, or you’ve found a problem you need to fix. Both of these are good outcomes. On one hand, you’ve increased your confidence in the system and its behavior, on the other you’ve found a problem before it caused an outage.</p><h4><strong>Have Fun</strong></h4><p>Chaos Engineering is a tool to make your job easier. By proactively testing and validating your system’s failure modes you will reduce your operational burden, increase your availability, and sleep better at night. Gremlin makes it safe and simple to get started, <a href="/service/mailto:kolton@gremlininc.com">email us</a> to get started today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=321ec4468bce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gremlininc/your-first-chaos-experiment-321ec4468bce">Your First Chaos Experiment</a> was originally published in <a href="https://medium.com/gremlininc">Gremlin</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>