<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Beal on Medium]]></title>
        <description><![CDATA[Stories by Jeff Beal on Medium]]></description>
        <link>https://medium.com/@jbbeal?source=rss-36a2b6ef26f7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nmN7eHohHhxxVg7nXez4kg.jpeg</url>
            <title>Stories by Jeff Beal on Medium</title>
            <link>https://medium.com/@jbbeal?source=rss-36a2b6ef26f7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:21:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jbbeal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reliability from Unreliability]]></title>
            <link>https://medium.com/@jbbeal/reliability-from-unreliability-cdc4d2e0718?source=rss-36a2b6ef26f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdc4d2e0718</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jeff Beal]]></dc:creator>
            <pubDate>Mon, 05 Dec 2022 07:09:13 GMT</pubDate>
            <atom:updated>2022-12-05T07:09:13.163Z</atom:updated>
            <content:encoded><![CDATA[<h4>A particularly unexpected failure scenario causes some grief</h4><figure><img alt="A cat sleeping on a green leather chair. Completely irrelevant to the story, but who doesn’t like pictures of cats?" src="https://cdn-images-1.medium.com/max/1024/1*I4nq1Ua-fWnv3Xn27CpEfw.jpeg" /></figure><p>From 2008 through 2010, I worked on the team that built and launched Amazon’s Relational Database Service (RDS). We built RDS on top of existing AWS services, including EC2, EBS, and S3, with SQS and Simple Workflow powering some of our systems behind the scenes. Cloud Computing was still very early in its development, and none of these systems were nearly as robust as they are today. Our challenge, put simply, was to find ways to build a reliable and robust database system, on top of a shaky and unreliable foundation.</p><p>At the time — and to an extent still today — the prevailing advice on how to architect your applications to run on EC2 hardware was to assume that your servers are unreliable, and design accordingly. Netflix built systems like Chaos Monkey to intentionally increase the likelihood of failure, to help their teams and systems be prepared to handle major systemic failures gracefully. We knew, when building RDS, that our systems had to offer better reliability to our customers than EC2 did.</p><p>There were a lot of types of failure we were able to anticipate. We knew that EC2 and S3 APIs would fail and timeout, so we planned for failure handling and retry strategies from the beginning. (This is where SWF and SQS were really helpful, and will maybe be the topic of a later post.) We knew that servers would crash, so we built monitoring systems to validate connectivity to each database server, and built workflows to move the database to a different machine when we lost connectivity. One (of many) failure scenarios we did not anticipate was Silent Success.</p><p>Very early in the project, we set up a series of end-to-end tests that were continuously running from a Jenkins server. These tests were exercising any functionality we offered on RDS — creating a database, scaling it up, scaling it down, increasing storage, taking a backup, restoring from backup, etc. — and running those scenarios against real AWS services and hardware. These tests launched hundreds of RDS instances — and, therefore, hundreds of EC2 instances — every day.</p><p>We started out with a standard AWS account[1], same as any AWS customer would have. At the time, the default EC2 instance quota was about 20 instances. We could have requested an increase to our instance quota (and maybe we did a little?), but since our test team followed good hygiene and cleaned up after each test, we didn’t think it was necessary. As I’m sure you’ve figured out because I’m bringing it up, we started to run into cases where tests were failing due to hitting our EC2 instance quota.</p><p>Some of the failures could be traced to cleanup issues in our code, which we worked to address. (I sometimes wonder if one of my most significant contributions to RDS, along with Nick Hertl, was in making our “Delete DB Instance” workflow almost perfectly robust to any type of failure.) However, no matter how many issues we fixed on our end, it still seemed inevitable that we would eventually hit quota, despite not having any previously reported errors in our tests we could find.</p><p>We eventually found that the EC2 instance IDs that were filling our quota weren’t anywhere in our database. We had our own administration database where we tracked all of the data about all customer database instances (including our test instances), and it simply had no record of every creating these EC2 instances. So, we added more logging statements, thinking that if we made sure to log every EC2 response, we’d find out where these orphan EC2 instances were coming from.</p><p>The instance IDs weren’t in logs, either.</p><p>With more investigation (likely with some help from the EC2 team; it was over a decade ago, I don’t remember <em>all</em> the details), we realized these EC2 instances were from AWS requests that that were being reported as failures to our system. We were reporting a service timeout — either returned to us from a load balancer, or based on a client-side timeout on our end — but the underlying EC2 API was actually, eventually, successful. Since EC2 instance IDs are randomly generated with each request, the only way to get an instance ID was from a successful API request. We didn’t get an instance ID back, so treated it as just another failure, retried the request, and carried on. With no record of the instance ID, our system didn’t know it had to delete it, so it just stuck around until we accumulated enough of them to fill up our quota.[2]</p><p>Once we identified the problem, the resolution was relatively straightforward. We added an hourly cron job to list all of our EC2 instances on our account, and delete any for which we didn’t have records. (We did ask the EC2 team to improve their APIs to reduce occurrence of this issue, but they had other priorities at the time. I suspect that the current ClientToken field on their API was added as a permanent solution to prevent similar issues. This was added after my time on the team, though.)</p><p>Some problems are a challenge to diagnose; others are a problem to solve. This doesn’t stand out as very significant in either of those aspects, but to me it stands out as one of the most unexpected failure scenarios I’ve run into. One where the system essentially lies to you, telling you it failed, when in fact it had succeeded. It’s one thing to write code that handles every failure scenario, but this opens up the possibility that even the failure scenarios are things you cannot trust to be what they at first appear.</p><p>I’m reluctant to overgeneralize, but wanted to share this as an example of the type of unexpected issues you will encounter when trying to build perfectly reliable software. I think being a good engineer requires you to have simultaneously pessimistic and optimistic outlooks. Your pessimistic side needs to believe there is at least one bug in every line of code you or anybody else has ever written; your optimistic side needs to know that every problem is solvable.</p><p><strong>Fake Footnotes[3]</strong></p><p>[1] Worth pointing out that our test account was separate from our production account, which, since this was happening before we launched, didn’t really matter.</p><p>[2] If we were a normal AWS customer, we might have noticed this on our bill and deleted them sooner. There were some advantages to being internal AWS customers.</p><p>[3] Real footnotes would be cases where the publishing platform actually supported footnotes as a feature.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdc4d2e0718" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Not again, but at last]]></title>
            <link>https://medium.com/@jbbeal/not-again-but-at-last-dfcd7e1522ee?source=rss-36a2b6ef26f7------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfcd7e1522ee</guid>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[2016-election]]></category>
            <category><![CDATA[poetry]]></category>
            <dc:creator><![CDATA[Jeff Beal]]></dc:creator>
            <pubDate>Sun, 05 Feb 2017 21:23:45 GMT</pubDate>
            <atom:updated>2017-02-06T01:42:48.549Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNuObiQZbg2s24zyzRk9yQ.jpeg" /><figcaption>Flags at Memorial Day display in Eagle, Idaho; May, 2011</figcaption></figure><p>Was America great when Columbus arrived,<br> When we brought Christ’s light to the land?<br> When we raped and we pillaged to plunder their gold, <br> was it then that our greatness is found?</p><p>Was America great when the Mayflower came,<br> When pilgrims could worship in peace?<br> When we hung the witches on Salem’s hill,<br> Was it then that our greatness is found?</p><p>Was America great when our nation was born<br> When farmers bought freedom with blood?<br> When slaves toiled our fields and were counted as less,<br> Was it then that our greatness is found?</p><p>Was America great when we ended the war<br> When we finally freed those same slaves?<br> When we sent Jim Crow to push them back down,<br> Was it then that our greatness is found?</p><p>Was America great when the bold and the brave<br> Toppled the Nazi regime?<br> When separate but equal still ruled back home,<br> Was it then that our greatness is found?</p><p>Was America great when we stepped on the moon,<br> When science took us to the heavens?<br> When MLK’s murder put doubt to his dream,<br> Was it then that our greatness is found?</p><p>Was America great when the wall came down,<br> And the Soviet Union crumbled at last?<br> When we killed Matthew Shepard, and said ‘don’t ask, don’t tell’<br> Was it then that our greatness is found?</p><p>Was America great when we marched to Iraq<br> When we freed them from Saddam’s grasp?<br> When at Columbine we murdered kids for fun,<br> Was it then that our greatness is found?</p><blockquote><strong>Author’s Note: </strong>There’s an obvious structure in this piece of two lines talking about a great accomplishment or event in America’s history, followed by two lines that, in my opinion, describe something we should all be ashamed of. I also wanted to include some mention of the unlawful and unjust internment of Japanese citizens during World War II, but had a hard time fitting it in. I made a few attempts at writing that stanza and couldn’t come up with something that fit the pattern. Also, the idea for this poem came while watching the film <em>Loving</em>, so I felt it necessary to keep the mention of “separate but equal” where it is. If I figure out a way to fit Japanese internment into the piece, I may publish a revised version.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfcd7e1522ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>