<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Keith Pitt on Medium]]></title>
        <description><![CDATA[Stories by Keith Pitt on Medium]]></description>
        <link>https://medium.com/@keithpitt?source=rss-bbc15949f05------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SW_3TBCRoZSs4rGycsuW-A.jpeg</url>
            <title>Stories by Keith Pitt on Medium</title>
            <link>https://medium.com/@keithpitt?source=rss-bbc15949f05------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:18:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@keithpitt/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[5 ways we’ve improved flakey test debugging]]></title>
            <link>https://medium.com/building-buildkite/5-ways-weve-improved-flakey-test-debugging-4b3cfb9f27c8?source=rss-bbc15949f05------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b3cfb9f27c8</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[rspec]]></category>
            <dc:creator><![CDATA[Keith Pitt]]></dc:creator>
            <pubDate>Mon, 03 Jul 2017 21:31:04 GMT</pubDate>
            <atom:updated>2017-07-04T00:53:15.497Z</atom:updated>
            <content:encoded><![CDATA[<p>Flakey tests are <em>the worst</em>. Sometimes when debugging flakey tests, we (the Buildkite development team) would say things like “wouldn’t it be cool if the test suite did this” or “I’d be able to debug this failure <em>so</em> much easier if I just had access to the database”. So we tried to build some of those things, and turns out, it wasn’t as hard as we thought it’d be.</p><p>We use RSpec as our test framework, and it has a ton of hooks and features that allow us to extend it in cool and useful ways. So here’s the top 5 things we did that turned out to be awesome.</p><h3>Change #1 — Saving database state for failed tests</h3><p>One of the more tricky types of test failures to debug are when you expect the database to be in one state, but it’s weirdly in another. Perhaps there are extra or missing records from the database that’s making all your tests fail. They’re even harder to debug on CI when the database state is wiped after every test and destroyed at the end of the test suite.</p><p>To combat these types of error, when a test fails, we capture the state of the database at that time and store it as a simple JSON file. We then have a mechanism to load that same state into a temporary database on our development machines so we can see what was going on in the database when the test failed.</p><p>Here’s a gist of the code we use to save and load database state from Rspec: <a href="https://gist.github.com/keithpitt/c124eec848c6b40ee4a5f1f1ec9f9cc9">https://gist.github.com/keithpitt/c124eec848c6b40ee4a5f1f1ec9f9cc9</a></p><p>Once you’ve downloaded a dump of the database state, loading it into a temporary database is super easy with script/load_test_database_state</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJVMjy9GvGcBeI0yh2B-0A.png" /><figcaption>Now I can inspect the database state at the time of the error</figcaption></figure><p>As a bonus, we upload the database state as a <a href="https://buildkite.com/docs/builds/artifacts">Buildkite Artifact</a> so we can quickly download the state for each failed test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIykrjYMTfCq3brwOow3ZQ.png" /><figcaption>Uploading the database state to Buildkite as an artifact</figcaption></figure><h3>Change #2 — Our own simple database cleaner</h3><p>Now that we had the database state saver, I decided to have a go at trying to fix some of the flakey tests in our test suite. The next time I saw them fail, I downloaded the state of the database for that test to figure out what was going on.</p><p>Looking through the downloaded data, I noticed that there were extra records in the database from previous tests runs that shouldn’t have been there. After more digging, it turned out that database_rewinder (the gem we use to clean the database between test runs) had a pretty major caveat: it doesn’t clean tables that you insert data into usingActiveRecord::Base.connection.execute %{INSERT ...} (which our application does a lot of for performance reasons).</p><p>After spending some time trying to figure out to how fix the issue within database_rewinder, I decided that it may be easier to just write our own database cleaning code. It ended up being much simpler than I thought it would be — and less lines too!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/352329e28528fa0767aac4aaaa75b11c/href">https://medium.com/media/352329e28528fa0767aac4aaaa75b11c/href</a></iframe><h3>Change #3 — Extracting logs from failed tests</h3><p>When tests fail, it’s hard to find the specific place in the logs where that spec ran. We made it easier for ourselves by splitting out the log with the failed test.</p><p>You can find the code we use to split logs here: <a href="https://gist.github.com/keithpitt/74950281d01b11bac7b076fb1d3eda1a">https://gist.github.com/keithpitt/74950281d01b11bac7b076fb1d3eda1a</a>’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*XiLgBiVdszFZK9IXLIOHdw.png" /><figcaption>The Rails Log Splitter also shows the name of the current spec</figcaption></figure><p>Just like we did the database state files, we upload all log/**/*.log files as <a href="https://buildkite.com/docs/builds/artifacts">Buildkite Artifact</a>s so we can easily download them from the Buildkite UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ae2Emqpkg-NIOmcQsMVeA.png" /><figcaption>Easy access to log of a failed spec</figcaption></figure><h3>Change #4 — Tracing ActiveRecord IDs in logs</h3><p>A common thing we found ourselves doing locally when trying to debug database related test failures was this:</p><pre>it &quot;returns does something&quot; do<br>  user = User.create(...)<br>  Rails.logger.info &quot;User created and had ID: #{user.id}&quot;<br>  ...<br>end</pre><p>We found this a little repetitive, and we wanted a mechanism that not only worked when debugging failures locally — but also when we were trying to debug tests that failed on CI.</p><p>To replace this repetitive manual step, we invented the ActiveRecordIDTracer that outputs the ID of the last inserted record into log/test.log</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*ASNkgWh47Fpm8RL1TtFloA.png" /><figcaption>ActiveRecordIDTracer as seen in log/test.log</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0928342360c9857fab482422be260689/href">https://medium.com/media/0928342360c9857fab482422be260689/href</a></iframe><h3>Change #5—Resetting database ID sequences</h3><p>We came across a weird test failure once where the test would only fail if the 2 records we were testing had the same ID (yeah, weird…). The annoying part was the first time we’d run the test locally it would fail. But subsequent test runs would succeeed.</p><p>The first step to debugging any weird test failure was to try and get it to fail each time. One we realised what the problem was, we whipped up this little class that would reset all the ID sequences in PostgreSQL after each test run.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b80e6b86f4e40c0286893b10793712d/href">https://medium.com/media/6b80e6b86f4e40c0286893b10793712d/href</a></iframe><h3>Takeaway: Treat your test code just like production code</h3><p>We change production code all the time, but tests end up rotting away without much improvement. Our light bulb moment was when we stopped treating our tests as this weird ball of code that couldn’t be shaped.</p><p>After we started treating test code like production code, debugging test failures got a whole lot easier (<em>how cool is it downloading database test state to your dev machine!?)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b3cfb9f27c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/building-buildkite/5-ways-weve-improved-flakey-test-debugging-4b3cfb9f27c8">5 ways we’ve improved flakey test debugging</a> was originally published in <a href="https://medium.com/building-buildkite">Building Buildkite</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Outage Post-Mortem for August 22nd]]></title>
            <link>https://medium.com/building-buildkite/outage-post-mortem-for-august-23rd-82b619a3679b?source=rss-bbc15949f05------2</link>
            <guid isPermaLink="false">https://medium.com/p/82b619a3679b</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[postmortem]]></category>
            <dc:creator><![CDATA[Keith Pitt]]></dc:creator>
            <pubDate>Tue, 23 Aug 2016 14:28:57 GMT</pubDate>
            <atom:updated>2016-08-24T01:04:00.489Z</atom:updated>
            <content:encoded><![CDATA[<p>On the 22nd of August 2016 at 17:21 UTC, <a href="https://buildkite.com/">Buildkite</a> suffered a severe unplanned outage. During this period Buildkite was still able to run builds and update GitHub/Bitbucket Pull Request statuses, but no one was able to login, view build logs or read documentation. This is a full account of what happened, what changes we’ve made and lessons we learned.</p><h4>What a morning…</h4><p>The Buildkite team are currently based out of Melbourne, Australia — which meant we were all asleep when we first went offline (3:21 AM AEST). Usually if this happens our primary on-call person would get a phone call from PagerDuty letting them know that Buildkite is having problems, but due to mis-configured settings on PagerDuty and our phones being on “silent”, that didn’t happen.</p><p>We woke up at 21:00 UTC almost 4 hours after we went offline to see our phones full of emails, tweets and Slack messages letting us know Buildkite was down. Many expletives were yelled as we all raced out of bed, opened laptops, and started figuring out what was going on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*kf6VqoEqI5u4H3rYtSb3uA.png" /><figcaption>SMS from Tim this morning when he discovered the outage</figcaption></figure><p>It took us a while to figure out what was going on, and at 22:28 UTC we made some temporary fixes to get <a href="https://buildkite.com/">buildkite.com</a> back online. Although we were able to start servicing web requests, our response times were really bad and some requests were just getting dropped.</p><p>We finally recovered at 23:10 UTC and spent the rest of the day working on making sure this doesn’t happen again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*njs1XqqcbYgJ03ZZYzKvYA.png" /><figcaption>Buildkite’s uptime over the last 24 hours — it’s not great…</figcaption></figure><h4>Trying to reduce our AWS bill</h4><p>About a year ago we were accepted into the AWS Activate program and awarded a bunch of credits to spend on AWS infrastructure. Since then we’ve been growing fast and used the age old method of “just buy more hardware” to solve our scaling problems. This was a cheap solution in terms of time and allowed us to get on with building out the rest of the product.</p><p>About 2 weeks ago we discovered that those credits were about to run out, and we needed to start aggressively working on reducing our AWS bill. We paused product development and spent some time coming up with a plan to reduce the cost of the next bill.</p><p>The main culprit was our <em>m4.10xlarge Multi-AZ RDS PostgreSQL</em> database. We looked at some graphs and decided to downgrade to a more affordable<em> r3.2xlarge Multi-AZ RDS PostgreSQL </em>instance.</p><p>Since we felt pressure to get these changes out sooner rather than later, we didn’t do our due diligence to figure out if this new database would perform under heavy load.</p><blockquote><strong>Lesson #1:</strong> Keep an eye on AWS credits so you’re not rushed to make significant infrastructure changes in a short period of time.</blockquote><blockquote>AWS Billing Reports show $0 bills while you’re spending credits, so it’s really hard to know what your actual costs are.</blockquote><h4>Database performance issues</h4><p>We scheduled a maintenance window on the weekend to downgrade the database. The downgrade was successful and everything was looking great. The performance of the new database looked fine and we were happy with the results.</p><p>Our largest customers are in US timezones, so we wouldn’t really have known how the database was going to hold up under heavy load until Monday morning in the US (Monday night for us). We monitored it into our evening and what we thought was the beginning of the US peak period and everything seemed to be tracking okay, so we went to bed.</p><p>Since the new database didn’t have as much memory as the old one, we also moved to a centralised pgbouncer to pool connections across all of our servers in hopes to reduce the per-connection memory overhead.</p><p>At around 16:00 UTC the database started struggling. It couldn’t keep up and its CPU started maxing out. These performance problems started a cascade of other issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XEKhNkDU2xXuYcmtry1NcQ.png" /><figcaption>CPU Utilization of our new database</figcaption></figure><blockquote><strong>Lesson #2: </strong>Do load testing after significant infrastructure changes. Due to the tight time constraints before the next AWS bill, we didn’t load test the new database or the centralized pgbouncer setup.</blockquote><blockquote>We’ve already started planning how this could work and ways we can do it without impacting others using the system.</blockquote><h4>Failing health checks</h4><p>Buildkite has 3 major Elastic Load Balancers and Auto Scaling Groups. One handles all requests to the Buildkite Dashboard, one handles all requests to our REST API, and the other handles all Agent API traffic. This separation works really well because we can scale each one individually since they all have very different traffic profiles.</p><p>The ELBs perform health checks by making HTTP requests to our main application every 30 seconds. If those checks return HTTP 500 errors in succession, it automatically removes those servers from the ELB and marks them as “OutOfService”.</p><p>Our health check endpoints hit our 2 SQL databases along with our Redis and Memcache servers to ensure the application is running smoothly. It essentially looks like this:</p><pre>class HealthCheckController &lt; ActionController::Base<br>  class UnhealthyError &lt; RuntimeError; end</pre><pre>  def index<br>    [ ActiveRecord::Base, Job::Log::Chunk::Shard::Two ].each do |ar|<br>      ar.connection.select_value(&quot;SELECT 1&quot;) == &quot;1&quot; or<br>        raise UnhealthyError, &quot;Database ping failed&quot;<br>    end</pre><pre>    $redis.ping == &quot;PONG&quot; or<br>      raise UnhealthyError, &quot;Redis ping failed&quot;</pre><pre>    Rails.cache.dalli.version.values.all?(&amp;:present?) or<br>      raise UnhealthyError, &quot;Memcached ping failed&quot;</pre><pre>    render plain: &quot;OK&quot;<br>  end<br>end</pre><p>When the database performance problems started, the health check endpoints started returning HTTP 500 errors that it couldn’t connect to the database. The failed health check threshold is much lower for the ELB servicing dashboard requests, so it automatically started removing servers. The ELBs for the Agent API and the REST API were having some issues, but the servers remained in the ELB.</p><p>Normally when servers are removed from the ELB due to health check failures, they’re replaced with new ones. When the new ones have finished launching, the ELB starts performing health checks. Due to an error in how we bootstrapped these new servers and some of our recent infrastructure changes, the health checks failed which meant no new servers could come online to replace the ones that were removed.</p><p>Over the course of a few hours, this caused a cycle of new servers to launch and terminate instantly due to failing health checks.</p><blockquote><strong>Lesson #3: </strong>Re-think what a health check should actually do. Our first iteration checked that it could connect to all dependent servers. We didn’t consider what happens if _all_ servers in an ELB failed health checks.</blockquote><blockquote>We need to figure out the trade offs here: would we prefer all servers to get dropped out of an ELB? Or would we prefer to just throw HTTP 500 errors if it can’t connect to our database.</blockquote><blockquote>We’ve switched it now to just return “OK” if it can hit the app. We don’t check database connectivity in this request anymore.</blockquote><blockquote>We’re also considering adding another health check endpoint that Pingdom can hit which makes sure the application can connect to the database and any dependent services.</blockquote><h4>Replacement servers not going healthy</h4><p>New servers that are launched by the Auto Scaling Group are based off an AMI that we only build when we make changes to server configurations. This AMI has a version of the codebase baked into it at the time it was built — so when they come online they need to figure out what the latest version of the codebase is and deploy that to itself.</p><p>Our first version of “how do we find out what Git revision is deployed to production” was to cURL a special endpoint that we created that returned the current revision: https://buildkite.com/_secret/version</p><p>This works really well if buildkite.com is online… We always had the intention of improving this but other stuff just got in the way and it dropped off our radar.</p><p>Since it couldn’t grab the latest version, it just rolled with what ever version of the codebase that was baked into the AMI. The health check in this version referenced an old database `Job::Log::Chunk::Shard::One` which has since been decommissioned. This meant all health checks failed against new servers and they never got added to the ELB.</p><blockquote><strong>Lesson #4:</strong> Don’t do this.</blockquote><blockquote>We’ve now switched to storing the latest deployed revision as file on S3 and we reference that when newly launched servers need to self-deploy.</blockquote><h4>Our “on-call” setup failed</h4><p>While all of this was happening, the Buildkite team were fast asleep.</p><p>We have various services watching Buildkite all the time: New Relic, Pingdom, Datadog, Calibre and CloudWatch. If any of them start seeing issues on Buildkite, they create a PagerDuty alert which phone calls the developer who’s been designated “on-call”. This week it was my turn.</p><p>I still haven’t figured out how this happened, but the “Immediately phone me” rule was somehow dropped as a “Notification” from my settings. I have “Do Not Disturb” turned on my phone in the evenings, but I allow PagerDuty to call me if anything comes up. Since I didn’t get a phone call (just push notifications which don’t make any sound) I didn’t get woken up.</p><p>We have escalation policies setup so if someone doesn’t acknowledge a PagerDuty alert it goes to someone else. The other team members had their phones on “silent” since they weren’t designated as the primary “on-call”, so they didn’t get woken up either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2GSY9TGHTxs6YCM71p2lw.png" /><figcaption>My updated PagerDuty settings</figcaption></figure><blockquote><strong>Lesson #5:</strong> Ensure all on-call team members have the correct PagerDuty and iPhone settings before going on call. We’ve all done manual tests to ensure calls are getting through. We’d like to investigate whether or not we can trigger company-wide tests to make sure everyone’s setup is correct on a regular basis.</blockquote><blockquote>Another idea is to have PagerDuty call team members when they go “on-call” — or have a requirement that all Notification Rules contain a “Immediately phone” rule. We haven’t looked into whether or not PagerDuty supports this out of the box, but we’ll be reaching out to them to see what we can do.</blockquote><h4>AWS issues prevented us from fixing anything</h4><p>Once we were all online and had finished identifying what the problems were, we decided the first thing we wanted to do was upgrade the database.</p><p>We logged into the AWS Console and tried upgrading the database, but due to IAM issues AWS was experiencing, the interface wouldn’t let us make the changes. We checked the AWS Status Page and noticed they were having problems, so we decided to skip this step for now and go onto the next thing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4BKEa4uBz72UriTCWLfLA.png" /><figcaption>Screenshot of AWS Status Page showing IAM issues</figcaption></figure><p>We needed to push out a new AMI to our servers that didn’t reference https://buildkite.com/_secret/version. We made the necessary changes, built the AMI and decided to test it out before deploying it to the rest of our infrastructure. When we tried to launch it, we discovered the IAM role selector wasn’t working either. We kept getting “Internal server error” in the AWS Console — which we guess to be IAM related.</p><p>At this point we stole some servers from the Agent API and added them to the ELB that services requests for the dashboard. This allowed us to bring buildkite.com back online — although requests were slow due to the database still being under heavy load.</p><p>Once the IAM issues had subsided we tried launching new EC2 servers, but we got hit by another AWS issue: EC2 request limits. Because servers had kept coming up, failing health checks, and going down again, we had reached out EC2 request limit. Our only option here was to wait it out.</p><p>After some time we were eventually able to upgrade the database to an instance type that could handle the load and roll out new servers that fixed self deploys. We had to use the aws-cli to do these last steps since the AWS Console was still having issues.</p><p>Along with upgrading the database, we also took this opportunity to remove the centralized pgbouncer server. We previously used to have pgbouncer installed on each host and pool from there. Since we knew the previous setup had worked well, we reverted back to that until we had capacity to investigate if the new pgbouncer setup was working.</p><blockquote><strong>Lesson #6: </strong>When you see high churn issues in AWS make sure to stop your health check and autoscaling rules so that you don’t run into future EC2 request limits. When we hit ours we changed the Auto Scaling Groups to EC2 health checks and locked in desired counts.</blockquote><blockquote>As part of writing this postmortem, we discovered you can “Suspend and Resume Auto Scaling Processes” directly from within AWS — which would have been very useful if we’d had known about it:</blockquote><blockquote><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/as-suspend-resume-processes.html">http://docs.aws.amazon.com/autoscaling/latest/userguide/as-suspend-resume-processes.html</a></blockquote><h4>Moving forward</h4><p>Tim, Sam and I would like to personally apologise for the downtime. We know exactly what it’s like when services you rely upon have technical issues (AWS had problems when we were trying to fix ours). It’s incredibly frustrating and at times infuriating. We don’t want to be that sort of company and we don’t want our customers and friends to feel like that.</p><p>We’d also like to personally thank you all for your understanding during the downtime. It’s was very heart warming and encouraging to see such lovely tweets while we were working to get everything back online.</p><h3>Nathan Anderson on Twitter</h3><p>opshug to @buildkite ops team</p><h3>David Goodlad on Twitter</h3><p>hugops to my friends at @buildkite</p><h3>Dave Slutzkin on Twitter</h3><p>Sending hearts out to @keithpitt, @toolmantim, @sj26 - looks like a tough morning for the BK team.</p><h3>Matt Allen on Twitter</h3><p>Big props to @keithpitt, @toolmantim and @sj26 for dealing with multiple levels of outages all at once today. The team at @buildkite is</p><p>We can’t promise that we won’t go down again, but we can promise that we won’t make the same mistakes twice. Not only are we stronger technically, we’re a stronger team now because of it.</p><p>If you have any questions or would like to know more about what happened, feel free to email me directly keith@buildkite.com</p><p>❤ Keith, Tim and Sam</p><p><em>Buildkite is a CI &amp; CD platform that helps you speed up and automate your software development workflows. We value transparency and support, and are dedicated to providing powerful and reliable build tools your team can depend upon. If you’d like to learn more about us and what we do, check us out at (the now snappy) </em><a href="https://buildkite.com/"><em>buildkite.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82b619a3679b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/building-buildkite/outage-post-mortem-for-august-23rd-82b619a3679b">Outage Post-Mortem for August 22nd</a> was originally published in <a href="https://medium.com/building-buildkite">Building Buildkite</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>