<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Christian Gregg on Medium]]></title>
        <description><![CDATA[Stories by Christian Gregg on Medium]]></description>
        <link>https://medium.com/@CGA1123?source=rss-134dccaae9fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tR-gw-bSFpKOpLidr2thjw.jpeg</url>
            <title>Stories by Christian Gregg on Medium</title>
            <link>https://medium.com/@CGA1123?source=rss-134dccaae9fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:33:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@CGA1123/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ Scaling Ruby on Rails on Heroku with Connection Pooling]]></title>
            <link>https://medium.com/@CGA1123/scaling-ruby-on-rails-on-heroku-with-connection-pooling-4fa2e9b02e72?source=rss-134dccaae9fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fa2e9b02e72</guid>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[pgbouncer]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Christian Gregg]]></dc:creator>
            <pubDate>Sat, 06 Feb 2021 21:51:52 GMT</pubDate>
            <atom:updated>2021-02-07T20:47:10.689Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbxPeAL7ZmmXo7ylL7vfQA.png" /><figcaption>The result of one trial in the experiment I ran to test out Connection Pooling, depicting P99 duration and queue times as well as average request success rate.</figcaption></figure><p>Heroku recently made Connection Pooling for Heroku Postgres databases <a href="https://devcenter.heroku.com/changelog-items/2011">generally available</a>. This is exciting! I decided to take a weekend to see how this might affect our ability to scale our Ruby on Rails applications at <a href="https://carwow.co.uk/">carwow</a>.</p><p>In this post I want to go over the evolution of scaling an application from one dyno to many as your application gains traction. We’ll cover different resource constraints that you might encounter, some ways which we’ve overcome them, and finally how Connection Pooling could help us to increase our application capacity by ~4x.</p><p>I created a <a href="https://github.com/carwow/pgbouncer_test">dummy application</a> and ran a series of <a href="https://github.com/carwow/pgbouncer_test/blob/f6993f07fadd1245a2f6b3beb16072fcfad41476/test.sh">trials</a> against that application with a varying number of dynos, concurrent requests, and either using or not using connection pooling.</p><p>Each trial ran for 45s, the application always did a 500ms pg_sleep on a standard-0 Heroku Postgres instance and 500ms sleep on the ruby side. This simulates a 50/50 split of DB IO and other IO work which is <em>fairly </em>representative of the workloads we see at carwow<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6RsnYb7Hin3SzQjrLRJhA.png" /><figcaption>Table showing P99/90/75/50 percent of time which a request is waiting on the database vs total request runtime across three production Ruby on Rails application (results over a 14-day time period).</figcaption></figure><p>Throughout this post I will be referencing this application and metrics pulled from the experiments run on this application to step through the simulated increases in traffic over time.</p><p>One of the key metrics to know about is “request queue time” which is what we use to to autoscale our dyno formations.</p><blockquote><strong>Request Queue Time </strong>is<strong> </strong>the amount of time between a request being ingested by Heroku’s routing layer (as timestamped by the <a href="https://devcenter.heroku.com/articles/http-routing#heroku-headers"><em>X-Request-Start</em></a> header) and the request arriving at our applications middleware. This indicates the amount of time which a request was fully ready to be processed but was waiting on resources on your application to be free. A persistently high/increasing value in this metric signals that your application is under-provisioned for the current level of traffic.</blockquote><p>A high or persistent increase in the value of this metric is an indication that our application is under-provisioned and is not able to keep up with the amount of traffic being directed to it. If this metric continues to grow it can lead to cascading performance degradation or failure as downstream applications fail to access data and Heroku drops requests as they exceed the 30s timeout mark.</p><p>Cool, let’s get going! 🚀</p><h4>Starting with a single dyno</h4><p>Let’s start at the very beginning with a single dyno running a single process and 10 threads. The ActiveRecord connection pool is also set to 10. What happens if we start to send more than 10 concurrent requests to the application?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nYRV-J-iqhZwwwe06TDHWw.png" /><figcaption>Showing an application running one dyno with a constant flow of 5/10/15/20 concurrent requests being made to it</figcaption></figure><p>The app.test_name column is formatted as NN-dyno-MMM. NN is the number of dynos currently running on the application (in this case 1), MMM is the configured number of concurrent requests (in this case, 5, 10, 15, 20). The tool that creates load (<a href="https://github.com/rakyll/hey">hey</a>) will continually make up to MMM concurrent requests, whenever one request completes it will start a new one.</p><p>We can see that as the number of concurrent requests exceeds the number of threads configured on the application the request queue starts to increase. This is to be expected. We only have 10 threads available to process work so we can only process a maximum of 10 requests at a time. Any additional request will need to be queued and wait until a thread is free in order to be processed.</p><p>There are a number of ways to improve this situation, we can either try to make every request faster or we can scale our application to have more threads available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ywNtUCJtGatdp-3ZQM6YKA.png" /><figcaption>Focusing in on request queue time, showing an increase as the number of concurrent requests exceeds the number of threads available</figcaption></figure><p>For the sake of this blog post we’ll assume that the application is already fully optimised¹. This leaves us only with the option to increase the amount of threads available. We can do this by increasing the number of puma workers, the number of threads, or both! I think for the majority of applications the main things restricting the number of threads and processes will be available RAM.</p><p>The more threads and processes running, the more RAM you will need available. There is a <a href="https://www.schneems.com/2019/11/07/why-does-my-apps-memory-usage-grow-asymptotically-over-time/">great blog post</a> on memory usage over time by Richard Schneeman (schneems) which is well worth a read.</p><p>At a high number of threads per process you might also have to start worrying about CPU contention as MRI/CRuby is restricted to running on a single CPU core due to the Global VM Lock (GVL).⁷</p><p>Let’s assume that we’ve maxed out the RAM available on our dyno and we either already have or don’t want to scale vertically anymore (scaling vertically = provisioning larger dynos sizes to get access to more resources).</p><p>In this scenario,<strong> </strong><em>our only path forward is to scale horizontally</em>, spinning up more dynos to <em>increase the total number of concurrent requests our application can handle</em>. At work we do this automatically (we ‘auto-scale’) using <a href="https://www.hirefire.io/">HireFire</a> based on the request queue metric across all our dynos.</p><h4>Scaling horizontally to multiple dynos</h4><p>So autoscaling has kicked in and we are now running 5 dynos, traffic has increased as well now steadily with 30, 40, 50, and 75 concurrent requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TMejFW_-BxL0_btHpkb7lg.png" /><figcaption>Scaling our application to five dynos with 30/40/50/75 concurrent requests</figcaption></figure><p>There are some spikes in queue time on the levels of traffic that we wouldn’t expect², but they are not persistent so it isn’t much of an issue. The reason for these spikes is because Heroku routes requests randomly across the whole dyno formation. This means that it is entirely possible that over a smaller timeframe the distribution is not uniform causing one or two dynos to be routed more requests than others, thus increasing the queue time on those dynos. If we look at the heatmap we can see this a little bit more obviously. Looking at the P99 doesn’t always give us the full picture!</p><p>That being said in general the same pattern observed with one dyno shows with five — queue time increases as the number of concurrent requests increases and exceeds to total number of web threads available. Albeit with a little bit more noise in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WMRWFATSW4ucp5j5WgYtBg.png" /><figcaption>Scaling to fifteen dynos continues with the trend shown with one and five dynos</figcaption></figure><h4>The limits of horizontal scaling</h4><p>Horizontal scaling looks to be very promising, our scaling problems are solved! We can continue to provision more dynos whenever we notice request queuing increasing and persisting. This <em>could</em> be true if your application doesn’t make use of any external resources. However, in our case each dyno connects to the database and at some point we’ll start to exceed the provisioned resources on the database.</p><p>In the case of <em>this</em> application, the primary DB resource we are constrained by is the total number of available <em>connections</em>.</p><p>A standard-0 Heroku Postgres instance has a limit of 120 connection which can be opened at any one time. Our application will open up to 10 connections per dyno. In theory, this means that our application can only scale up to 12 dynos before it exceeds this limit. In practice, it seems that the hard limit is around double that (240 connections, 24 dynos).</p><p>What happens when we exceed this limit?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*668y0mISq5pogY_H8YOd6w.png" /></figure><p>The graph above shows this scenario, as we increase our number of dynos and concurrent requests, the pattern established previously persists. What is new is that the average rate of successful HTTP responses has deteriorated as we’ve scaled our dyno formation in excess of the available database connections.</p><p>As we exceeded our connection limit the Rails application fails to connect to our database. Raising a PG::ConnectionBad error with this message FATAL: remaining connection slots are reserved for non-replication superuser connections.</p><p>How do we get around this problem? We’ve already optimised the heck out of our app and on this database plan we can’t scale horizontally anymore without seriously affecting request success rate as we run out of connections.</p><p>We can increase our Heroku Postgres plan to get access to more connections. The standard-2 (there is no standard-1) plan increases our available connections to 400 and standard-3 to 500. However, after 500 connections we can’t get anymore! Let’s assume we’ve forked over the cash and delayed this issue. We’re running a standard-3 database, we’re now able to run ~50 dynos³ and are hitting issues with scaling. What do we do now?</p><p>Before connection pooling was available one solution to this issue, which we’ve gone with at carwow, is to create a read-only replica of our application⁴.</p><p>This approach works really well, we’ve been running it on two of our largest apps for over a year. It has a fairly high initial cost of set up, but once your CI/CD pipeline has been configured properly and you have it initially set up in Terraform, it isn’t too much of a burden to manage and replicate to new applications, especially if the alternative is to move all of our infrastructure off of Heroku. 😄</p><h4>Using Connection Pooling to enable more horizontal scaling</h4><p>Now that Connection Pooling is available there is a much less involved way to enable our application to scale more horizontally without causing large amounts of requests to fail!</p><p>The graphs below show comparisons of running our test application with 25, 50, and 75 dynos with and without Connection Pooling (pgbouncer) enabled.</p><p>Notice any differences? 😛</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJ0RIzPxG5oG1WMLzR4dPQ.png" /><figcaption>Running with twenty-five dynos two trials each with and without connection pooling (pgbouncer) enabled</figcaption></figure><p>We’re back up to a 100% success rate! Connection Pooling allows us to scale our application horizontally way past what we used to be able to. By adding a layer of indirection over the underlying ‘physical’ database connections and sharing them between all of the client connections we can increase the utilisation of the underlying connections and avoid total failure to acquire a connection.</p><p>At the beginning of this post I mentioned that each request spends 500ms sleeping on the database and 500ms sleeping in the Ruby thread, which was more-or-less representative of database usage seen on production applications at carwow.</p><p>Based on this, we can say that the database connection utilisation over the lifetime of a request in our test application is 0.5 or 50%. We’re holding on to the database connection for the lifetime of the request but only using it for 50% of that time, what a waste!⁵</p><p>Connection Pooling works at the database transaction level, each transaction can run on any one of the underlying connections, this means that over the lifetime of a web request you might use as many different connections as you have transactions. In between those queries you aren’t holding on to that physical connection, other clients and web threads are free to use it!</p><p>This allows us to make much better use of the underlying connections (increasing utilisation), allowing us to scale horizontally with minimal impact on overall request latency. We are able to allocate connections at a higher level of granularity, while also making it impossible to the exceed the total number of allocated connections and therefore the possibility of hard failure. We are trading system failure for (potential) performance degradation when connection utilisation exceeds 100%.</p><p>This works remarkably well on our test application as we increase to running 50 and 75 dynos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rcBV3X-6wBVpUuycZn-7Kw.png" /><figcaption>Application running fifty dynos</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LlmFoVK7m_NEO3zNZ8nLNA.png" /><figcaption>Application running seventy-five dynos</figcaption></figure><p>We’ve scaled an application running a standard-0 database with 10 connection per dyno to 75 dynos and no pesky PG::ConnectionBad errors to be seen!</p><p>The cynics among you will probably be asking: what’s the catch?</p><p>And there is a catch or two with this. If you are eagle-eyed you will have noticed in the later trials (running 50 and 75 dynos) the duration_ms metric starts to creep up over 1s with a pretty significant spike actually in one of the trials running 75 dynos. This is most likely explained due to contention on the underlying database connections, as well as a potential overhead to using the connection pool itself.</p><p>The connection pooler still only has 90 connections available to it (Heroku only makes 75% of the total allocated connection pool available to the pooler, pgbouncer). If all 90 of the underlying connections are used, any additional transaction will be queued until a connection becomes available, much like what happens with our web requests. As queueing increases so does overall latency. This could be an issue if it persists over time but is vastly better than complete failure! Make sure to set appropriate timeouts.</p><p>Some more caveats are that because connections are shared on a transaction basis, session variables will leak as there is no guarantee that a client will be assigned the same connection across transactions. This means you cannot make use of some PG features such as session variable and prepared statements. You should consult the <a href="https://www.pgbouncer.org/features.html">pgbouncer documentation</a> for more details.</p><p>Overall, however, very promising! With the increased utilisation of database connections and no longer needing to reserve connections for prebooting we’ve successfully increased the number of dynos we can scale by over 4x.</p><p>We should <em>really</em> have only been running ~6 dynos on this application in theory in order to stay well within connection limits and to have reserved enough connections to preboot properly.⁶</p><p>In the end we’ve managed to scale up to 75 dynos before seeing any significant performance impact on total request duration due to contention on the underlying database connections.</p><p>We’re still working out the finer details of this feature and will be running a couple more tests to make sure that timeouts work as we expected them to before trying this out on a couple of production applications. I’ll update this post if with any additional learnings during that process.</p><p>In the meantime, I hope that this was informative and that you were able to take away some learnings for your, your teams, or your company’s journey to scaling your applications to handle more load. It’s a very nice problem to be having, but one that ideally you already have an idea how to solve!</p><p>Feel free to reach out with more questions, clarifications, or feedback. You can find me on twitter <a href="https://twitter.com/CGA1123">@CGA1123</a>.</p><p>Until next time! 👋</p><h4>Footnotes</h4><p>[1] Worth noting that it might be wise to look into other more common performance issues before exploring connection pooling as a solution! Fix your N+1 queries, investigate sensible places where caching could help (at all the different layers of your stack), reduce JSON blobs, use optimised collection rendering in Rails, and so on!</p><p>[2] A small note on other anomalies you might see. If you are running your application on Heroku’s Common Runtime (running Standard-1X or Standard-2X dynos) you might every so often get individual dynos which spike in request duration and queue time, this is usually down to ‘noisy neighbours’ hogging resources as you are running on shared infrastructure, these spikes usually resolve themselves as the noisy neighbour calms down or Heroku automatically shuts them off as part of their automatic abuse detection. If you need to get things running smoother faster, restarting the dyno in question will get it back up running on another physical host.</p><p>[3] It’s worth noting that you usually wouldn’t be able to run your application at full dyno capacity during normal working hours! You usually need to reserve up to 50% of your pool for pre-booting application to ensure you have a smooth rolling deploy (assuming that you deploy regularly throughout the working day like we do at carwow). You also likely need to reserve some connection for your Sidekiq workers, Heroku Scheduler jobs, and the odd production console here and there.</p><p>[4] We create a follower database (a read-only database to which writes to the lead instance are replicated to) and a new Heroku application running the same application code. We then route ~50% of all safe requests (GET, HEAD, OPTIONS) to that read-only application. The majority of our requests aren’t writing to the database and we can tolerate a little bit of lag on writes in most circumstances (we can also set a cookie after each write to ensure that a client reads from the lead database after a recent write as well).</p><p>[5] Using Connection Pooling should allow most applications to run <em>at least </em><strong>double the number of dynos </strong>they are currently restricted to as removes the need to reserve connections during the preboot phase of a release. This is assuming that you are using preboot (which you should be, if you are releasing on a regular basis). Any additional horizontal scaling opportunity afforded by Connection Pooling will rely on the current connection utilisation of you application. If you already spend 100% of the time in your requests doing database work, connection pooling is not going to allow you to make better or more efficient use of your connections, they are already being optimally utilised.</p><p>If you only spend ~50% then you have the potential opportunity to allow more clients to connect and share the underlying pool to increase the utilisation. The lower your current utilisation to more opportunity Connection Pooling offers your application to scale horizontally passed its current threshold.</p><p>[6] As alluded to in [5], during the preboot window of your application you could run up to double the number of dynos in your formation. The outgoing dynos might not stop doing work as soon as the incoming dynos are started, this can lead to brief periods of time where your application requires DB connections that exceeds how many are available leading to potentially large increases in errors during deploys. If you deploy often this can affect reliability significantly. For this reason it is usually wise to limit the maximum number of dynos you scale to in order to accommodate for this. You might also want to avoid having ‘double deploys’ to avoid overlapping preboot windows. You can usually do this by queuing deploys and grouping multiple commits merged within some specified time window together so they are part of the same release and therefore preboot window.</p><p>[7] In a previous version of this post this was tacked on to the end of this paragraph:</p><blockquote>although I think in most cases Rails applications spend the large majority of their time on I/O rather than CPU, so it’s probably not something people in general need to worry too much about</blockquote><p>This is highly dependent on the workloads of your application and may not be true for you, and after taking the time to look at the data from applications in production at carwow, it seems it wasn’t true for those applications either. Which showed a ~50/50 balance between CPU work and IO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RqAao92K-_dpGUCIG7usCw.png" /></figure><p>You should be able to measure the general split of IO vs CPU on your application by subscribing to the process_action.action_controller ActiveSupport::Notification and measuring the proportion of cpu_time vs total_time. This should give you a good <em>indication </em>of the split of work being done on your application if enough samples are collected. If you want more accurate measure you might need to roll your own middleware and measure the difference between cpu_time and total_time yourself.</p><p>cpu_time can be found as the difference of Process.clock_gettime(Process::THREAD_CPUTIME_ID) at the start and end of your request.</p><p>total_time can be found as the difference of Process.clock_gettime(Process::CLOCK_MONOTONIC) at the start and end of your request.</p><p>(Thanks to u/jrochkind on Reddit for <a href="https://www.reddit.com/r/ruby/comments/le7uxr/scaling_ruby_on_rails_on_heroku_with_connection/gmgqvfx?utm_source=share&amp;utm_medium=web2x&amp;context=3">pointing this out</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fa2e9b02e72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing Rails Memory Usage by ~15%]]></title>
            <link>https://medium.com/@CGA1123/reducing-rails-memory-usage-by-15-56090b6294bd?source=rss-134dccaae9fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/56090b6294bd</guid>
            <category><![CDATA[puma]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[Christian Gregg]]></dc:creator>
            <pubDate>Sun, 24 Jan 2021 12:50:48 GMT</pubDate>
            <atom:updated>2021-02-07T12:11:35.736Z</atom:updated>
            <content:encoded><![CDATA[<p>We recently deployed a change to our Puma configuration that led to a ~15% reduction in memory usage across all of our Ruby on Rails applications! In this post, I want to go over what that change was as well as a bit of a dive into why it worked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6LAp97KmTJxylB_YacM8A.png" /><figcaption>Memory usage on one of our application</figcaption></figure><p>If you are already familiar with Puma and the differences between running in single vs cluster mode then the TL;DR of this post is that we used to run puma in cluster mode with a single worker and have now changed that to running in single mode, simple as that 😄</p><p>So why does running puma in cluster mode use more memory? Before answering that question, here is a brief overview of how Puma does work.</p><h4>How Puma does work</h4><p>Puma uses a thread pool to do work, it is implemented in <a href="https://github.com/puma/puma/blob/master/lib/puma/thread_pool.rb#L17">Puma::ThreadPool</a> which is the context within which our application code is actually executed.</p><p>However, puma doesn’t just pass incoming requests directly to this thread pool. Before passing a request to the thread-pool for work puma buffers the requests in a queue. There is a nice overview of how and why this is done in the <a href="https://github.com/puma/puma/blob/master/docs/architecture.md">Puma Architecture</a> document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*UPvPnhR14LLJz64Tr7Ao6Q.png" /><figcaption>The Puma Connection Flow (<a href="https://github.com/puma/puma/blob/master/docs/architecture.md">source</a>)</figcaption></figure><p>The main gist of it is that there are 3 queues on which requests can be placed before being processed by our application:</p><ol><li>The Socket Backlog queue which holds an OS level-queue of network connections on the socket that the puma process is listening (outside of puma)</li><li>The Receive queue which accepts connections from the sockets and buffers requests until they have been fully read (this prevents issues such as “slow-client” DDoS attacks) before pushing them into the “todo” queue</li><li>The “todo” queue is the queue which the actual thread-pool mentioned above consumes from, it will pop off requests from the queue, call our underlying application, and write a response.</li></ol><p>As an aside, the time which a request waits in the “todo” queue is the metric we use in order to trigger auto-scaling of our web dynos. This is the amount of time which a request which was fully ready to be processed had to wait before being processed, if it trends upwards it means we aren’t spreading the load across enough instances and so we (automatically) provision more!</p><h4>Cluster Mode vs Single Mode</h4><p>Puma has a concept of (and by default runs in) “cluster-mode” whereby a number of separate ruby processes can listen to a socket and process requests as described above. Running puma in cluster mode has an overhead of running n+1 processes, where n is the number of workers you would like to run. This is because puma needs to run an additional process to manage the pool of workers. This is called the “Runner”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*TUA5gnA9wrcP-3W1r5me6Q.png" /><figcaption>Puma’s General Architecture (<a href="https://github.com/puma/puma/blob/master/docs/architecture.md">source</a>)</figcaption></figure><p>The “Runner” is responsible for managing the “Workers”, it starts, restarts, and stops them. Workers are created by “forking”. Forking is a way by which a currently running process can create a new “child” process. When a process is forked its memory is copied to the new process and it continues execution from the same place. For example:</p><pre>#!/usr/bin/env ruby</pre><pre>puts &quot;I am process ##{Process.pid}&quot;<br>pid = fork</pre><pre><strong>if</strong> pid<br>  puts &quot;I am the parent ##{Process.pid}. My child is ##{pid}&quot;<br><strong>else<br>  </strong>puts &quot;I am the child  ##{Process.pid}&quot;<br><strong>end</strong></pre><pre># Output:<br># =&gt; I am process #50601<br># =&gt; I am the parent #50601. My child is #50749<br># =&gt; I am the child  #50749</pre><p>This is how Puma works and why we use things such as preload_app!, before_fork and on_worker_boot in our Puma configuration to manage how this behaves.</p><p>preload_app! is especially interesting as it exists in order to make use of an optimisation that OSes can provide when forking processes called <a href="https://en.wikipedia.org/wiki/Copy-on-write">“Copy-on-Write” or CoW</a>.</p><p>CoW delays the allocation of memory to the child process until the memory being used has been written to and becomes “dirty”. This allows multiple processes to use the same memory until it is required that they have their own memory. By “preloading” the application, the runner process loads up our rails app and allocates all the required boot-up memory (including running our initializers) before forking that memory onto the worker. This reduces boot-up time on the worker processes and should also reduce overall memory footprint.</p><p>In single-mode puma bypasses all of this and only runs a single worker process that has no “runner” orchestrating it. It does not preload_app! or have before_fork and on_worker_boot hooks, because they are not necessary.</p><p>Because our configuration only runs a single worker in cluster mode we create additional unnecessary memory allocations. By simply turning this off we save a pretty handsome amount of memory!</p><h4>Results</h4><p>You can see a reduction in usage after boot and then over time as well from around ~12–1 PM in the following graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxqH5DjCxmRuWJ63H_VS5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6LAp97KmTJxylB_YacM8A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ADt0nHagUCCh71UChh2UFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKvM7uRUOlqXioco1XUulg.png" /></figure><p>The good thing about this reduction is that it brings our dynos under the 100% (512MB) limit that Heroku has on our Standard-1X dynos.</p><p>While this is a ‘soft’ limit in that Heroku won’t kill our process if we exceed it (it will only kill a process that is using over 200% its expected memory) it does result in the usage of “swap” memory. In the graphs above you can see the use of “swap” in dark-purple. The use of swap occurs as soon as we exceed the 512MB limit.</p><p>“swapped” memory are chunks of memory that a process holds that are removed from RAM and dumped onto disk by the OS when there is not enough memory available in RAM. This can cause significant performance impact when the process tries to access memory that has been swapped to disk as it requires the OS to dump some other memory into swap and load the requested memory from disk again. This is orders of magnitude slower than having that memory in RAM (as people who studies this lovely memory hierarchy diagram below will already know):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*SSwhR6KJec40Wxcwh7BxdA.png" /><figcaption>Memory Hierarchy, Cost and Speed are inversely proportional to the size/capacity</figcaption></figure><h4>Summary</h4><p>Hopefully, this was a pretty decent intro into how puma works in general and the differences between cluster and single-mode. As well as touching a little bit on OS processes, forking, and memory hierarchy, which are all pretty interesting topics which can fill a full semester worth of learning. 😄</p><p>I stumbled across this idea to try out changing our configuration after reading <a href="https://github.com/puma/puma/issues/2393">puma/puma#2393</a> which describes how some people saw puma double in memory usage after upgrading to 5.0.0 which was caused by an (inadvertent) change in puma which led to properly respecting the WEB_CONCURRENCY environment variable and running multiple workers.</p><p>This isn’t the first decent performance improvement that I’ve found by fixing/changing really old configuration, it’s always worth investigating and experimenting with options that are set in your production configuration and initializers, things might have changed significantly since you first wrote or generated that configuration, especially if the project was started many major versions ago.</p><p>As always, thank you very much for taking the time to read this post!</p><p>If there are any questions, comments, or feedback let me know by responding below or on Twitter @CGA1123 🚀</p><p>Until next time 👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56090b6294bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Speeding up our Rails apps with asynchronous cache writes]]></title>
            <link>https://medium.com/carwow-product-engineering/speeding-up-our-rails-apps-with-asynchronous-cache-writes-3eaab3e1c26?source=rss-134dccaae9fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eaab3e1c26</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[caching]]></category>
            <dc:creator><![CDATA[Christian Gregg]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 13:34:38 GMT</pubDate>
            <atom:updated>2020-08-21T13:34:38.520Z</atom:updated>
            <content:encoded><![CDATA[<p>At carwow our applications can be chatty, making one or several API calls to each other in order to fulfil a request. In fact, around 48% of all user-facing requests in the last 7 days made at least one API request to another one of our applications before responding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZA2xittO-8Z3QTmGq5hDVw.png" /><figcaption>Average % of user-facing requests making at least 1 (internal) API call over a 7 day period (48%)</figcaption></figure><p>One of the tools we use to reduce the strain we put between our applications is caching. We cache both at the client-side (using Rails.cache), as well as on the server-side (by putting Fastly ‘in front’ of our web servers). This allows us to reduce network overhead on the client-side as well as reduce the load on the server-side.</p><p>Around 73% of our requests end up being served from the client-side Rails.cache and while investigating some particularly slow endpoints I found myself unable to account for ten-to-a-hundred-ish milliseconds of processing time. Using honeycomb to investigate a couple of specific traces I could <em>see </em>our applications spending some amount of time between getting a response and continuing on with execution, I just didn’t know what it was spending it on! As it turns out, there are two main things we do via middleware on our HTTP Clients: caching and parsing. I’m going to focus on the caching part in this post, but the parsing part had just as much to do with these unaccounted milliseconds (perhaps more on that in a separate blog post 😉).</p><p>Something I’ve learned to do during optimisation work is to think of all the changes we intend to make as experiments, and that means taking a lot of measurements. It&#39;s only by measuring that I can have confidence that the changes I made had the intended impact. So how can we measure the impact of cache writes?</p><p>By making use of the <a href="https://edgeguides.rubyonrails.org/active_support_instrumentation.html#cache-write-active-support">cache_write.active_support</a> notification and pushing the results of that to honeycomb, I was able to get a view of how long we usually spend on caching.</p><p>It turns out that actually it wasn’t awful<em>. </em>As the figure below shows, the P99 was around 15ms, but wouldn’t it be nice if that was closer to 0ms?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LcQgmvG-xvx9OtHyjPV4Mw.png" /><figcaption>Time taken for cache writes</figcaption></figure><p>We shouldn’t be holding up any requests in order to update our cache, it should be best-effort and eventually consistent.</p><p>In order to facilitate this behaviour, we built ActiveSupport::Cache::AsyncWriteStore which wraps an existing cache-store implementation and overrides its#write method to push its work to a thread-queue instead of executing synchronously.</p><p>To test the effectiveness of this change we rolled it out to 50% of our application servers and had a look at overall performance impact on web requests. We saw some pretty good gains!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9N3GR1ZbqeqhX-MDOFcOA.png" /><figcaption>Results of an even/odd test, odd dynos (false / orange) had asynchronous cache-writes enabled via ActiveSupport::Cache::AsyncWriteStore</figcaption></figure><p>Overall P99 was ~100ms faster and P95 ~20ms. Every little bit helps!</p><p>Writing to cache asynchronously may be a nice and relatively easy way to chop off some of the overhead of caching while still maintaining the benefits!</p><p>One thing to take into account is that this adds another pool of threads for your Ruby process to manage, which <em>may </em>have an adverse effect on overall performance as ruby still needs to do those writes as some point. Effectiveness might vary based on specific use-cases for your application!</p><p><em>Thanks for taking the time to read this post! Let me know what you think, and please leave any questions in the comments below or on twitter @CGA1123 </em>😃</p><p><em>Until next time</em> 👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eaab3e1c26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carwow-product-engineering/speeding-up-our-rails-apps-with-asynchronous-cache-writes-3eaab3e1c26">🗂 Speeding up our Rails apps with asynchronous cache writes</a> was originally published in <a href="https://medium.com/carwow-product-engineering">Carwow Product, Design &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backfilling 50 million records quickly]]></title>
            <link>https://medium.com/carwow-product-engineering/backfilling-50-million-records-quickly-eaa04ba5617f?source=rss-134dccaae9fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaa04ba5617f</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[backfill]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Christian Gregg]]></dc:creator>
            <pubDate>Thu, 11 Jun 2020 15:51:52 GMT</pubDate>
            <atom:updated>2020-06-11T20:34:01.404Z</atom:updated>
            <content:encoded><![CDATA[<h3>🔙 Backfilling 50 million records, quickly</h3><h4>Getting large datasets consistent quickly, safely, and without downtime</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZVsV3q90HmkxECgJtwodcg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@ruchindra?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ruchindra Gunasekara</a> on <a href="https://unsplash.com/search/photos/data-warehouse?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Over the last few months at carwow, a small team have reworked the way in which we process and propagate pricing changes made by dealerships, manufacturers, and governments, and the user-facing quotes which depend on them.</p><p>The database table which holds user-facing quotes (known as the factory_order_quotes table) is one of the largest, oldest, and most-used at carwow. Part of the work we had to carry out involved running numerous “backfills” in order to migrate specific enum values for the status column. The data the we needed in order to know how to migrate those records came from three different databases accessible only feasibly by making multiple internal HTTP API calls.</p><p>We needed to come up with a solution that allowed us to update ~50 million records, relying on data coming from three external sources, quickly, safely, with zero-downtime, and without significant impact on user experience and other production systems. And we did! 😄</p><h3>How did we do it?</h3><p>The major obstacle that we faced was being able to have access to the data we needed in order to decide which new status a record should be moved to. Running additional requests or queries to fetch that data live as we were backfilling was out of the question: it would introduce significant overhead slowing down the backfill (we were a bit impatient) and also introduce a significant amount of additional load on our other applications.</p><p>One important attribute of this backfill was that we had already done the work to update the live logic to update the status column appropriately, so only old records existed to be migrated. We also knew that the ‘destination’ status for any given record we wanted to backfill was stable, and wouldn’t change based on when we ran the migration. This was a <em>key</em> property of the backfill we were running. Knowing this meant that we could precalculate all of the records that needed to be moved to a new status and what that status was. And best of all, we could calculate this mapping using our data warehouse, bringing the impact to users and production systems to zero 🥳</p><p>This also allowed us to remove the need for cross-application API requests and remove the need for additional database queries on the host applications, we could instead run only UPDATE queries, reducing the amount of work we had to do ‘live’ by orders of magnitude!</p><p>So what did this process look like?</p><ul><li>Run a query on Snowflake (our data warehouse) to pull in all of the required data from databases powering several distinct production applications</li><li>Dump the data into a temporary S3 bucket, split per 10k rows</li><li>In a background job, download the file, load the IDs and run an UPDATE</li></ul><p>You can see the script used to run the query, split, and upload to S3 <a href="https://gist.github.com/CGA1123/cd52d744da70ee562814d44f7c7eaf49">here</a>.</p><p>The splitting of files and enqueuing as background jobs was important. It meant that we could easily and quickly stop and start the backfill if we needed to (if the database was on fire) in a recoverable manner. We basically had ‘checkpoints’ at every 10k records. It also meant that we could run multiple streams of jobs, each operating on a specific file or sequence of files.</p><p>If you are curious, the implementation used to run these jobs looked a little (a lot) like <a href="https://gist.github.com/CGA1123/143ca9376397c0f09c96e39313479281">this</a>.</p><h3>Summary</h3><p>The method we used allowed us to update ~48.7 million records (twice actually, because we replicate this particular table to a different database) in just under 6 hours! This is mostly due to removing the need for multiple batched reads from the database and API calls to other services before updates. Instead, we did this all those reads pre-emptively outside of our production environment, allowing us to more or less consistently pump updates to the database.</p><p>There are a few important caveats which should be taken into account before attempting a similar approach:</p><ul><li>Make sure that all the records in the initial set fetched from your warehouse <em>will always remain within that set</em> (if more records could potentially get added, that’s fine) i.e. the set of records you want to migrate should be fixed at the point you generate the mapping from your data warehouse (if it is possible for records to be removed from that set, this method is not directly applicable!).</li><li>You may need to throttle between writing runs, maybe adding some breathing period between enqueuing subsequent jobs to allow for propagation of writes to following databases.</li><li>Make sure that the writes you are making would not usually also trigger some other events (jobs, event publishing, etc.).</li><li>If there are some events that need to occur or may automatically be triggered, make sure that downstream services are capable of handling the additional load incurred by a backfill, or whether another solution is required (we had to run a secondary backfill on another of our applications following this one).</li><li>Make sure you have good monitoring and observability around your system to make sure it is coping well — it’s good to know when to give your system a break; before it breaks 😅</li><li>We found <em>significant </em>performance improvements in our backfill runs when ordering the dumped set of IDs we wanted to update in order (i.e. ORDER BY id). I assume this is because it causes less thrashing (page faulting) of memory on the database, although I couldn’t find documentation to support this. YMMV.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oe4ITebzYsJBTgym9YkJzA.png" /><figcaption>A graph of 5-minute load average on our leader database — updates took place between 12 pm and 6 pm on the 23rd</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HvKmKbMx9hdst_TpwCX2w.png" /><figcaption>A graph of 5-minute load average on our follower database — updates took place between 12 pm and 6 pm on the 23rd</figcaption></figure><p>Overall we found this method to be a great improvement on previous backfilling strategies we had developed, and have since used it multiple times (on the same table, and others) to enable us to evolve our data models as the business grows, requirements change, and we hit scaling issues.</p><p>There were some minor issues (which you may have deduced from the graphs above) related to database load that we had to keep a close eye on and stop our backfill for a little bit in order to allow our following/replica databases to catch up on commits. Thanks to the idempotent nature of the jobs we ran and the monitoring we had available to us we could proactively pause the backfill before having any impact on user-facing services.</p><p><em>Thanks a lot for taking the time to read this. I hope that perhaps you or your team might find the method above applicable in helping to facilitate the evolution of your company’s data model as your project grows and the requirements (functional and non-functional) change! If you have any questions or feedback feel free to ping me in the comments or on twitter @CGA1123</em></p><p><em>Until next time! </em>👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaa04ba5617f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carwow-product-engineering/backfilling-50-million-records-quickly-eaa04ba5617f">🔙Backfilling 50 million records quickly</a> was originally published in <a href="https://medium.com/carwow-product-engineering">Carwow Product, Design &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Shadow requesting for great good]]></title>
            <link>https://medium.com/carwow-product-engineering/shadow-requesting-for-great-good-92cde331363a?source=rss-134dccaae9fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/92cde331363a</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Christian Gregg]]></dc:creator>
            <pubDate>Fri, 13 Dec 2019 16:04:47 GMT</pubDate>
            <atom:updated>2019-12-13T22:53:13.972Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build confidence in how your web applications perform under additional load, safely</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4DHJmpMTEeRkX06zymQZPQ.png" /><figcaption>Graph showing shadow (purple) vs real (orange) requests during a recent test</figcaption></figure><p>Encouragingly, traffic to the carwow applications has grown significantly over the last couple of years, and of course we hope this trend continues. Our Software Engineering Tools &amp; Infrastructure (SETI 👽) Team has the task of ensuring that our applications, infrastructure, and tooling continue evolving to support the business as it grows.</p><p>Recently, SETI has been investigating how our applications perform under significant increases in traffic. One of the methods we’ve used is commonly known shadow requesting, and it’s been useful enough to us that I’d like to share it here.</p><h4>So what is shadow requesting?</h4><p>At its core, shadow requesting means replaying live requests shortly after they have completed. Either to the same host (in order to simulate additional load), or to a different host (to verify parity of responses).</p><h4>Why shadow requests?</h4><p>One of the benefits of load testing via shadow requests is that we are simulating real traffic patterns without any configuration overhead; by replaying the requests almost in real-time we create a lifelike scenario of additional users on our systems without needing to maintain a repository of user flows and their associated HTTP requests.</p><h4>How are we doing this?</h4><p>We developed a tool, <a href="https://github.com/carwow/umbra">umbra</a>, to facilitate shadow-requesting. It works as a Rack middleware, which sends incoming requests, via a Thread Queue, to a Redis pub/sub channel. In a separate process (or processes), a subscriber listens to that same channel and replays the request.</p><p>I’ve added the current umbra configuration for one of our applications in this <a href="https://gist.github.com/CGA1123/93a8a46b1f61c0470a57446341f5e52b">gist</a>.</p><h4>What do we need to watch out for?</h4><p>We currently configure umbra only to shadow idempotent requests. As a general rule this means only GET requests, however, we also use an allow/deny list of endpoints in order to ensure we are not inadvertently manipulating production data.</p><p>We also made sure thatumbra guarantees not to replicate its own requests, preventing an exponential increase in requests being made and an eventual self Denial of Service.</p><p>We have rich monitoring and alerts, powered by <a href="https://www.honeycomb.io/">Honeycomb</a>. This enables to monitor our applications and the point at which they start to ‘sweat’. Without good monitoring, we would not be able to gain any benefits or insights about how our applications behave under the artificially increased load we add via shadow requesting.</p><h4>What are some of the outcomes?</h4><p>We recently tested a new middleware using umbra. One of our applications receives around 50% of its traffic from internal clients. We already set client-side timeouts on internal requests in order to mitigate against cascading performance degradation.</p><p>When under significant load it is possible that our servers queue requests for longer than the client&#39;s timeout period. When this occurs, those requests would usually still be processed by the server, even though the client has already ‘given up’.</p><p>We added a middleware that would drop any requests that took longer to reach our application — from initial ingestion at the routing layer to reaching this middleware — than the timeout set by the client (via a X-Client-Timeout header set by all our internal HTTP clients). Using umbra we were able to test this middleware under load safely, and the results were positive!</p><p>We enabled the middleware only for web processes/dynos that were evenly numbered, and only for shadow requests, allowing us to compare the performance of web requests being processed with and without the new load-shedding middleware enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fr9EP6ft3rJm8DXD_zNm3A.png" /><figcaption>Results of even/odd test of load shedding on shadow-traffic only, purple lines depict those requests processed with load-shedding enabled, orange without.</figcaption></figure><p>The key takeaways from this test were:</p><ul><li>We were able to process 190% the number of requests with load-shedding enabled vs. without</li><li>Our Apdex score was 10% with load shedding enabled</li><li>P95 and P90 request queuing was significantly lower with load shedding enabled (and less significantly for P99)</li><li>Server error rates were around 1/2 for requests being processed with load shedding enabled</li></ul><p>The new load-shedding middleware looks to be a success!</p><p>Thanks to shadow requesting we were able to safely put our system under additional load and create the environment under which we expect this middleware to help us. This gave us the confidence in knowing that this change would help us to provide a more resilient service when our application comes under significant load. Thanks to that, we have since deployed the middleware to all of our applications.</p><p>Over the coming months, we will continue to work on testing out new ideas to improve the performance of our applications in preparation for the projected growth in our user base. We’ve open-sourced umbra on <a href="https://github.com/carwow/umbra">GitHub</a> under the terms of the <a href="https://github.com/carwow/umbra/blob/master/LICENSE.txt">MIT License</a>, and published on <a href="https://rubygems.org/gems/umbra-rb">Rubygems.com</a>.</p><p>We will continue to use it to test new solutions and to benchmark the performance of our applications. This will help us to identify where there are improvements to be made and enable us to give more accurate and better-informed assessments of the capabilities of our applications and surrounding infrastructure. umbra is now another tool we can use to ensure that carwow is able to deliver the best possible car buying experience to our growing user base.</p><p>Perhaps shadow requesting is something that your teams can look into in order to build confidence in the performance of your applications. If you are running a Rack-based application (such as Rails) then maybe umbra is a good fit for your team too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92cde331363a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carwow-product-engineering/shadow-requesting-for-great-good-92cde331363a">🌒 Shadow requesting for great good</a> was originally published in <a href="https://medium.com/carwow-product-engineering">Carwow Product, Design &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Speeding up our Heroku deploys by ~35%]]></title>
            <link>https://medium.com/carwow-product-engineering/speeding-up-our-heroku-deploys-by-35-percent-f9fa6f6cf404?source=rss-134dccaae9fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9fa6f6cf404</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[deployment-pipelines]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[Christian Gregg]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 14:55:38 GMT</pubDate>
            <atom:updated>2019-08-15T14:55:38.189Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀 Speeding up our deploys by ~35%</h3><h4>Speeding up our CI/CD pipeline by precompiling Heroku slugs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n88ZUj6mUE6GeDcyK8Ne2Q.jpeg" /></figure><p>Having a Continuous Integration/Continuous Deployment pipeline is the industry standard nowadays. CI/CD is great and has a very well documented list of benefits. However, one of the drawbacks is how slow the pipeline can become as your codebase grows.</p><p>At carwow, we have already taken many steps to try and ship faster, without compromising on the reliability and safeguards that a good CI/CD pipeline offers us.</p><p>Some of the things we do involve:</p><ul><li>Precompiling our assets in order to stop ad-hoc compilation occurring while our test suite runs</li><li>Extensive use of caching on assets and dependencies</li><li>Parallelisation of our test suite</li><li>Optimising dependencies between jobs in our workflow (i.e. workflow steps should start as soon as they can — they should not wait for any step that they do not rely on)</li></ul><p>All of these optimisations have helped to increase the speed of our CI/CD pipeline, and in turn enabled us to get changes — and fixes — out the door that much quicker. These optimisations are worth it because we deploy fairly regularly; anything between 15–35 times a day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dn24l0yXlw9A4iShF6M6kQ.png" /><figcaption># of deploys to production per day per country</figcaption></figure><p>That being said, there is always some room for improvement. All of our production applications follow the same general pattern for their workflow (and I assume this is fairly universal):</p><ol><li>Install dependencies</li><li>Compile assets and run code linters</li><li>Run our test suite</li><li>Deploy</li></ol><p>A quick look at recent builds on our main applications showed that we spend a lot of time during our deploy phase. But why?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTRP_-l_cspSENbrRMHDxg.png" /><figcaption>An overview of one of our master branch workflows, notice time spent in deploy step</figcaption></figure><h4>A brief overview of Heroku’s slug compilation and release process</h4><p>We deploy all our applications to Heroku and make use of their <a href="https://devcenter.heroku.com/articles/buildpacks">buildpacks</a> in order to <a href="https://devcenter.heroku.com/articles/slug-compiler">compile our source repositories into slugs</a> that Heroku can run.</p><p>The <em>de facto</em> way of deploying to Heroku is by pushing to their git remote. Doing so will lead to the following:</p><ul><li>Receive new commit</li><li>Sequentially run each buildpack via <a href="https://githooks.com/">git hooks</a>. In general, this involves installing dependencies and compiling code/assets</li><li>Bundle up the result of building into a slug</li><li>Run an optional release command</li><li>Restart all dynos with new slug</li></ul><h4>How we sped up our deploys</h4><p>Now that we understood a bit more about what exactly Heroku was doing, we could start to find out why it was taking so long and come up with a solution.</p><p>Earlier in this post, I mentioned that one of the ways in which we have previously improved our build times was by <em>“Optimising dependencies between jobs in our workflow”, </em>which means ensuring we aren’t making any steps wait for things they don’t depend on. Our previous deployment process was a great example of us <strong>not</strong> doing this.</p><p>Deploying to Heroku is split into two steps bundled together: build and release. The release step has a dependency on our test suite passing — we don’t want to release any code that doesn’t pass our tests. The build step, on the other hand, does not. The compilation and release of the slug have different pre-requisites to execution.</p><p>Conceptually what we want to do is start the time-consuming process of compiling our slug as early as possible and separate it from the release step. We just had to find a way to do it.</p><p>There are a few ways of decoupling the build and release when using Heroku. There are two main alternatives that they recommend: <a href="https://devcenter.heroku.com/articles/container-registry-and-runtime">Deploying with Docker</a>, or <a href="https://devcenter.heroku.com/articles/platform-api-deploying-slugs">building your own tarball</a>.</p><p>These two methods are quite involved and would likely mean reimplementing Heroku’s slug compiler, as well as changing the way in which we configure which buildpacks we use, and most likely a bunch of other stuff we hadn’t even considered.</p><p>There <em>had</em> to be a simpler way. There must be a way in which we can run Heroku’s own build process without releasing the slug to production, right? This would allow us to kick-off building a new slug and wait for our test suite to pass, at which point we can simply promote the slug to production.</p><p>Unfortunately, there is no way to do that, at least directly. If you dig deep enough into Heroku’s documentation you will find this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f-r-14ZFCO3mXiFSmAbCXg.png" /><figcaption>🎉A very helpful note from Heroku’s docs that took me far too long to find (<a href="https://devcenter.heroku.com/articles/build-and-release-using-the-api">source</a>)</figcaption></figure><p>So that’s what we did! We now do a normal Heroku deploy via git, except to a non-public instance of the app. This runs through the normal build process and performs an empty release. When — and if — the test suite succeeds, our deploy step uses <a href="https://github.com/heroku/heroku-api-plugin">Heroku’s Platform API</a> to find the appropriate slug from our compile app, and promote it to the production app. Our script looks <a href="https://gist.github.com/CGA1123/41a8901b82a14c9f2a77d8f9663d7d19">something like this</a>.</p><p>All of this means that our pipeline now looks like this (notice the compile_slug steps in the bottom left).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rixjPR7vNWRsgekNSRaEGg.png" /></figure><p>This allows us to save a <em>significant</em> amount of time in our builds by parallelising the slug compilation process to run while our test suite runs — we’ve seen speedups of 30–45% on our master branch builds. And all it took was some configuration changes and a little bit of bash 🎉.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3pnLUK21eD3GNLRHRVC5g.png" /><figcaption>Graph showing the average duration (minutes) of successful master builds on one of our applications</figcaption></figure><p>We use Terraform to manage our infrastructure, which made the process of creating a new (free, zero dyno) Heroku application for each production app simple; just a change in a variable once we had initially configured it.</p><p>There are some things to watch out for though:</p><ul><li>Make sure that you don’t run any release commands you shouldn’t on the compilation app. We did this by short-circuiting the release command with [ ! -z $COMPILE_APP ] || release</li><li>You may need to attach databases, other add-ons, and set some environment variables on your compilation app in order for it to work; in Rails at least, asset compilation requires loading your whole application.</li></ul><p><strong>I hope this information is useful and that you might also attempt the same or a similar approach in order to speed up your deploys, and hopefully, you see the same speedups as we have! If you know of any other interesting alternatives or decide to start your own project that enables building tarballs/slugs based on a list of buildpacks, please let me know in the comments.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9fa6f6cf404" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carwow-product-engineering/speeding-up-our-heroku-deploys-by-35-percent-f9fa6f6cf404">🚀 Speeding up our Heroku deploys by ~35%</a> was originally published in <a href="https://medium.com/carwow-product-engineering">Carwow Product, Design &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>