<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gavin Morrice on Medium]]></title>
        <description><![CDATA[Stories by Gavin Morrice on Medium]]></description>
        <link>https://medium.com/@gavinmorrice?source=rss-73d4a00c10dd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VlzH-xtYvHPqmH3LIgS6fw.jpeg</url>
            <title>Stories by Gavin Morrice on Medium</title>
            <link>https://medium.com/@gavinmorrice?source=rss-73d4a00c10dd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:15:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gavinmorrice/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding n+1 query problems in Ruby on Rails]]></title>
            <link>https://sourcediving.com/understanding-n-1-query-problems-in-ruby-on-rails-b2ea4be86a2d?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2ea4be86a2d</guid>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[database-optimization]]></category>
            <category><![CDATA[web-performance]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Wed, 07 Dec 2022 14:52:49 GMT</pubDate>
            <atom:updated>2022-12-07T21:07:58.588Z</atom:updated>
            <content:encoded><![CDATA[<p>n+1 queries are a common problem in web application development. This common pattern (or antipattern) can degrade application performance by increasing server response times and by placing additional strain on limited database resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJuNcRPQSgDccWSQ1BntGA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@hansenhouse?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Glenn Hansen</a> on <a href="https://unsplash.com/s/photos/engines?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>However, n+1 queries are <em>sometimes</em> desireable, and can be used along with other design patterns to <em>improve</em> performance. Because this pattern can have both positive and negative effects, it’s important that developers have a good understanding of n+1 patterns and their implications in the context of their application.</p><p>This guide aims to explain n+1 queries in the context of modern Ruby on Rails web applications.</p><h3>What is an n+1 query?</h3><p>An n+1 query occurs when we are loading a collection of records from a database, and also one related record for each record in that collection. The first query loads the collection of records from the database, and then one additional query is executed for each <em>n</em> record. Hence, the total number of queries executed is <em>n + 1</em>.</p><blockquote><em>💡 Tip</em></blockquote><blockquote><em>It sometimes helps to think of this as a </em>1+n<em> query, since the 1 (the query that loads the collection) is executed first, and then an additional </em>n<em> queries are executed thereafter.</em></blockquote><h3>A simple example</h3><p>Suppose we have a collection of recipes and each recipe belongs to a user. In our app, we wish to display a list of each recipe’s title and the name of the user who published the recipe next to it:</p><pre>&quot;Mac and cheese by Marco&quot;<br>&quot;Lobster thermidore by Joe&quot;<br>&quot;Fish tacos by Betty&quot;</pre><p>We might choose to implement this in Ruby like this:</p><pre>def title_and_author(recipe:, user:)<br>  &quot;#{recipe.title} by #{user.name}&quot;<br>end</pre><p>…and call our new helper method like so:</p><pre>Recipe.limit(3).each do |recipe|<br>  puts title_and_author(recipe: recipe, user: recipe.user)<br>end</pre><p>If we were to look at the application logs in this hypothetical example, we would expect to see something like the following:</p><pre>SELECT &quot;recipes&quot;.* FROM &quot;recipes&quot; LIMIT 3<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 1 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 2 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 3 LIMIT 1</pre><p>In this case, <em>n</em> = 3 (the number of recipes). For each recipe, an additional query is executed to load the related user record.</p><p>1 query to load the recipes. 3 queries to load each user.</p><p>total queries = n + 1 = 3 + 1 = 4</p><p>4 queries in total.</p><blockquote>📝 <em>Note</em></blockquote><blockquote><em>n+1 has become a common term to describe additional queries being executed for a set of n records. In practice, it is often more than simply 1 extra query per record. Sometimes the correct expression might be </em><em>2n+1 or </em><em>3n+2. We typically still refer to these as being &quot;n+1 problems&quot;.</em></blockquote><h3>What causes n+1 query problems?</h3><p>n+1s are a problem caused by human error, and tend to happen in applications that employ <a href="https://en.wikipedia.org/wiki/Lazy_loading">lazy-loading</a> to improve performance.</p><p>Because Rails’s lazy-loading is so easy to use, and will defer the loading of resources until they are called in code, it’s common for developers to introduce n+1 queries to a feature without realising it.</p><p><strong>Using common code design patterns and best-practices can sometimes make it <em>harder</em> to recognise an n+1 query</strong> by reading the code alone, because these encourage code to be separated out into many different places and reused in different contexts.</p><p>Here are some examples of code patterns that might introduce n+1 queries…</p><h4>Iterating over a collection</h4><p>n+1s, by definition, appear when we are working with collections of records. Often, they can appear when we call methods within iteration blocks, without pre-loading all of the data those blocks rely on.</p><p>Sometimes these patterns are very easy to spot, like in the following example:</p><pre>&lt;!-- app/views/recipes/index.html.erb --&gt;</pre><pre>&lt;% Recipe.limit(3).each do |recipe| %&gt;<br>  &lt;%= recipe.user.name %&gt;<br>&lt;% end %&gt;</pre><p>But sometimes the patterns are less obvious. For example:</p><pre>@recipes = Recipe.limit(3)<br>@user_names = @recipes.map(&amp;:user).map(&amp;:name)</pre><p>In both of these examples, just the @recipes collection has been loaded from the database to begin with. Calling user on each recipe iteratively will trigger an additional query to load each recipe&#39;s user individually.</p><h4>Separating code across architectural layers</h4><p>Often these patterns can be hidden behind a series of different methods in different application layers. These instances are difficult to spot by just reading the code, especially when we’re only focussing on one particular area of the code (such as the view templates):</p><pre>&lt;!-- app/views/recipes/index.html.erb --&gt;<br>&lt;h1&gt;Latest users to publish recipes&lt;/h1&gt;<br>&lt;%= latest_recipe_user_names(@recipes) %&gt;</pre><pre><em># app/helpers/recipes_helper.rb</em><br>module RecipesHelper<br>  def latest_recipe_user_names(recipes)<br>    recipes.map { |recipe| recipe.author_name }<br>           .to_sentence<br>  end<br>end</pre><pre><em># app/models/recipe.rb</em><br>class Recipe &lt; ApplicationRecord<br>  belongs_to :user</pre><pre>  def author_name<br>    user.name<br>  end<br>end</pre><h4>Metaprogramming / calling relations dynamically</h4><p>Metaprogramming can make it harder still to recognise an n+1 by reading the code alone. Study the following example, and try to spot how it might introduce n+1 queries:</p><pre>def collect_associated_names(collection, assoc_name)<br>  association = collection.model.reflect_on_association(assoc_name)<br>  unless association&amp;.klass.instance_methods.include?(:name)<br>    return Array.new(collection.count, nil)<br>  end</pre><pre>  collection.map do |item|<br>    if association.belongs_to? || association.has_one?<br>      item.public_send(assoc_name).name<br>    else<br>      item.public_send(assoc_name).map(&amp;:name)<br>    end<br>  end.flatten<br>end<br>@recipes = Recipe.limit(3)<br>collect_associated_names(@recipes, :user) <em># =&gt; [&quot;Betty&quot;, &quot;Joe&quot;, &quot;Marco&quot;]</em></pre><p>If we were to inspect the application logs in each of the above examples, we would see the same n+1 pattern in each case:</p><pre>SELECT &quot;recipes&quot;.* FROM &quot;recipes&quot; LIMIT 3<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 1 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 2 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 3 LIMIT 1</pre><h4>Talking to strangers</h4><p>n+1s can show up when an object tries to talk to a <em>stranger</em>.</p><p>A <em>stranger</em>, as defined in the <a href="https://en.wikipedia.org/wiki/Law_of_Demeter">Law of Demeter</a>, is an object that is only related to the current object through one of its dependencies.</p><p>Consider our earlier example:</p><pre>&lt;!-- app/views/recipes/index.html.erb --&gt;</pre><pre>&lt;% @recipes.each do |recipe| %&gt;<br>  &lt;%= recipe.user.name %&gt;<br>&lt;% end %&gt;</pre><p>In this case, our current context is the view template object. @recipes, and the local variable recipe are both <em>friends</em> or direct dependencies of the current view template object. But user has no direct relationship with the current view template object, and is only related to the template through the intermediary object recipe. In this example, user is a <em>stranger</em> to the view template.</p><p>By <em>talking</em> to user (by calling the method #name), this code may invoke n+1 database queries, because each recipe&#39;s user has not been explicitly eager-loaded as a dependency of the view template&#39;s dependency @recipes.</p><p>To state this problem more generally: <em>n+1 queries are likely to occur when an object calls methods on other objects that are not direct dependencies of this object.</em></p><p>This tends to happen because, as we move further away from objects within our current working context, we tend to think less about their properties and how they are defined. While we’re working on the recipes index page, we’re less likely to be thinking about how the Recipe model is defined, and less likely still to think about Recipe’s relationships and dependencies.</p><h4>Delegates pattern</h4><p>A common pattern to avoid <a href="https://en.wikipedia.org/wiki/Law_of_Demeter"><em>Law of Demeter</em></a> violations, is to define delegates within a class’s public interface.</p><p>For example:</p><pre>class Recipe &lt; ApplicationRecord<br>  belongs_to :user<br>  delgate :name, to: :user, prefix: true<br>end</pre><pre>&lt;% @recipes.each do |recipe| %&gt;<br>  &lt;%= recipe.user_name %&gt;<br>&lt;% end %&gt;</pre><p>This code is Demeter compliant because user_name has been defined as part of the public interface of Recipe, rather than us having to call recipe.user.name. By looking at the code in this example, there is no reason to suspect that this code will have any negative consequences.</p><p>However, behind the scenes, Rails’s delegate method is calling #user on each recipe individually, creating another n+1 problem—just as we saw in the previous example.</p><p>Delegates can help us write cleaner public interfaces, but we have hidden the complexity of our model’s relationship to the database behind another layer of abstraction, which makes it more difficult to recognise that an n+1 query has been introduced.</p><h4>In summary</h4><p>To summarize, this section:</p><ul><li>There are various ways we can introduce n+1 problems into our application</li><li>n+1s can arise as a result of following established code-design best-practices</li><li>It’s often difficult to recognise an n+1 query simply by reading the code</li></ul><h3>How do I recognise an n+1 query?</h3><p>As described in the section above, it’s difficult to spot an n+1 query by simply reading our application code.</p><p>Instead, <strong>the best way to find n+1 queries is by reading your application’s log</strong>. Here, you can see a record of every database query being made as part of each request, and look for instances that might indicate an n+1 query that can be optimised.</p><h4>A simple n+1 log example</h4><p>Typically, an n+1 query will look like many similar queries being executed one after the other:</p><pre>SELECT &quot;recipes&quot;.* FROM &quot;recipes&quot; LIMIT 3<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 1 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 2 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 3 LIMIT 1</pre><p>These instances are the simplest, and are usually the easiest to detect and solve.</p><p>Sometimes, though, they are not so easy to identify, and require a more thorough read of the log files.</p><h4>A non-sequential n+1 log example</h4><p>Sometimes the repetitive n+1 queries are separated by other queries, and so they don’t stand out as obviously from scanning the application logs:</p><pre>SELECT &quot;recipes&quot;.* FROM &quot;recipes&quot; LIMIT 2<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 1 LIMIT 1<br>SELECT &quot;ingredients&quot;.* FROM &quot;ingredients&quot; WHERE &quot;ingredients&quot;.&quot;recipe_id&quot; IN (1,2)<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 2 LIMIT 1</pre><p>In this example, the n+1 problem on users is separated by the SELECT queries on the ingredients table (the ingredients are properly eager-loaded as a collection). It’s important to review <em>the entire log</em> for a given request to look for similar queries becase they might not always be grouped together.</p><h4>Analogous query example</h4><p>Sometimes, we can load the same resources in different ways. This can lead to us to make more database queries than are necessary, but these aren’t always easy to spot in the log.</p><p>For example:</p><pre>&lt;% @recipes.each do |recipe| %&gt;<br>  &lt;% if recipe.comments.any? %&gt;<br>    &lt;%= render recipe.comments %&gt;<br>    &lt;div&gt;Showing a total of &lt;%= pluralize(&quot;comment&quot;, recipe.comments.count) %&gt;&lt;/div&gt;<br>  &lt;% end %&gt;<br>&lt;% end %&gt;</pre><pre><em>--- log output ---</em><br>Recipe Load (0.0ms)  SELECT &quot;recipes&quot;.* FROM &quot;recipes&quot;<br>Comment Exists? (0.1ms)  SELECT 1 AS one FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 1 LIMIT 1<br>Comment Load (0.1ms)  SELECT &quot;comments&quot;.* FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 1<br>Comment Count (0.1ms)  SELECT COUNT(*) FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 1<br>Comment Exists? (0.0ms)  SELECT 1 AS one FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 2 LIMIT 1<br>Comment Load (0.0ms)  SELECT &quot;comments&quot;.* FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 2<br>Comment Count (0.1ms)  SELECT COUNT(*) FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 2<br>Comment Exists? (0.0ms)  SELECT 1 AS one FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 3 LIMIT 1<br>Comment Load (0.0ms)  SELECT &quot;comments&quot;.* FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 3<br>Comment Count (0.0ms)  SELECT COUNT(*) FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; = 3</pre><p>In this example, the logs show 3n+1 queries, because for each recipe we are:</p><ul><li>Checking for the existence of any related comments</li><li>Loading all of the related comments</li><li>Counting the related comments</li></ul><p>… and each of these executes a separate query.</p><p>It’s important to understand that sometimes a query can look different from other functionally similar queries, and that this can hide opportunities for optimisation.</p><p>The above example could be optimised to execute only two queries:</p><pre>@recipes = Recipe.limit(3).includes(:comments)</pre><pre>&lt;% @recipes.each do |recipe| %&gt;<br>  &lt;% if recipe.comments.load.any? %&gt;<br>    &lt;%= render recipe.comments %&gt;<br>    &lt;div&gt;Showing a total of &lt;%= pluralize(&quot;comment&quot;, recipe.comments.length) %&gt;&lt;/div&gt;<br>  &lt;% end %&gt;<br>&lt;% end %&gt;</pre><pre><em>--- log output ---</em><br>Recipe Load (0.0ms)  SELECT &quot;recipes&quot;.* FROM &quot;recipes&quot;<br>Comment Load (0.1ms)  SELECT &quot;comments&quot;.* FROM &quot;comments&quot; WHERE &quot;comments&quot;.&quot;recipe_id&quot; IN (1,2,3)</pre><h3>When should I fix an n+1 query (and when should I not)?</h3><p><strong>When optimising a particular endpoint for performance, the ultimate goal should be to improve overall user experience.</strong> With this goal in mind, there are various factors you should consider, before you decide to address a particular n+1 query, including:</p><ol><li>Does this endpoint currently meet our SLOs?</li><li>Do we cache any aspects of this endpoint?</li><li>Does eager-loading really improve performance?</li></ol><h4>Does this endpoint currently meet our SLOs?</h4><p>Service Level Objectives (SLOs) are the objective metrics we use to determine whether or not a part of our application is performing to the desired standards we have defined. These should be a strong consideration when deciding whether or not to fix performance issues such as n+1 problems. Software teams have limited resources to apply to solving problems, and often an endpoint’s performance is <em>Good Enough™</em>, even when it isn’t perfect.</p><p>If an endpoint has n+1 queries, but is currently performing within your SLOs, it might be reasonable to focus on other optimisation opportunities elsewhere.</p><p>If an endpoint is performing below your team’s SLOs, you should probably raise this as a problem to be addressed with your team.</p><h4>Do we cache any aspects of this endpoint?</h4><p>Caching — specifically fragment caching or <em>Russian Doll</em> caching — can drastically change whether or not it pays to fix n+1 problems. In some cases, code that results in n+1 queries can actually be beneficial to the overall performance of an endpoint.</p><p>Consider the following example:</p><pre>&lt;% @recipes.each do |recipe| %&gt;<br>  &lt;% cache(recipe) do %&gt;<br>    &lt;% cache recipe.user do %&gt;<br>      &lt;h3&gt;Written by: &lt;%= recipe.user.name %&gt;&lt;/h3&gt;<br>    &lt;% end %&gt;<br>    &lt;%- @recipe.steps.each do |step| %&gt;<br>      &lt;% cache(step) %&gt;<br>        &lt;%= step.description %&gt;<br>      &lt;% end %&gt;<br>    &lt;% end %&gt;<br>  &lt;% end %&gt;<br>&lt;% end %&gt;</pre><p>In this example, this endpoint would execute at least two n+1 queries. For each recipe, we’re also loading its associated user and steps. If the collection @recipes contained ten recipes, this code could trigger twenty-one database queries.</p><p>However, because we are caching the recipe content as a fragment of HTML, the methods #user and #steps will not be called on subsequent page loads—the recipe&#39;s HTML content would be returned from the cache instead. This means that, as long as this recipe is not updated, this endpoint will only execute one query per request to load this content.</p><p>If just one of the recipes is updated, the records associated with that recipe would trigger additional queries again in subsequent loads, but this would likely still be fewer than the original twenty-one queries.</p><p>Since we’re serving mostly cached content when our recipes page is loaded, eager-loading the related user and steps might actually be detrimental. With a warm cache, we only need to load the recipes collection from the database on each page load, and subsequently loading the users and steps would execute additional queries without providing any performance benefit.</p><p>Because of this effect, it is important to review the application logs with caching turned on in development mode, to see what effect caching has on the queries that are executed. This should be a factor in whether or not n+1 queries should be <em>fixed</em> or implemented.</p><p>See <a href="https://www.youtube.com/watch?v=ktZLpjCanvg&amp;t=267s">this interview</a> with DHH and Nate Berkopec for more information.</p><h4>Does eager-loading really improve performance?</h4><p>As developers, we are taught to think of n+1s as always causing slower responses, and increasing database burden. Sometimes, paradoxically, it can be faster to execute multiple, simple queries than one more complex query across joins and with other conditions.</p><p>Consider the following example:</p><pre>@recipes = Recipe.eager_load(:user)<br>@recipes.each do |recipe|<br>  puts recipe.user.name<br>end</pre><pre><em>--- log output ---</em><br>SELECT &quot;recipes&quot;.&quot;id&quot; AS t0_r0, &quot;recipes&quot;.&quot;user_id&quot; AS t0_r1, &quot;recipes&quot;.&quot;title&quot; AS t0_r2, &quot;recipes&quot;.&quot;created_at&quot; AS t0_r3, &quot;recipes&quot;.&quot;updated_at&quot; AS t0_r4, &quot;users&quot;.&quot;id&quot; AS t1_r0, &quot;users&quot;.&quot;name&quot; AS t1_r1, &quot;users&quot;.&quot;created_at&quot; AS t1_r2, &quot;users&quot;.&quot;updated_at&quot; AS t1_r3 FROM &quot;recipes&quot; LEFT OUTER JOIN &quot;users&quot; ON &quot;users&quot;.&quot;id&quot; = &quot;recipes&quot;.&quot;user_id&quot;</pre><p>Eager-loading users along with recipes creates a fairly lengthy sql query with an outer join. It is possible that, in a large production dataset with tens of millions of records, this query actually takes longer to execute than the alternative solution shown below, which performs three simple queries on the primary key index of a single table:</p><pre>@recipes = Recipe.all<br>@recipes.each do |recipe|<br>  puts recipe.user.name<br>end</pre><pre><em>--- log output ---</em><br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 1 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 2 LIMIT 1<br>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 3 LIMIT 1</pre><p>The important point to iterate here, is that performance optimisation should be data-driven and outcome focussed. Sometimes, changes you make might not have the positive impact you expect them to, and so one should, if possible, benchmark changes with a production-like dataset before merging them; or monitor the impact of these changes shortly after deploying them; in order to validate that they are indeed improvements.</p><h3>Summary</h3><p>To summarise the above:</p><ul><li>To identify n+1 query problems before they are deployed, you should review the application log to see how a given endpoint interacts with the database</li><li>Review the log both with caching disabled, and with caching enabled. Be sure to run a few requests to see how caching changes the database usage</li><li>Whether to spend time on resolving an n+1 or not should be a decision that is informed by our SLOs, and reviewed with your team</li><li>Resolving n+1 queries isn’t always best for optimal performance</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2ea4be86a2d" width="1" height="1" alt=""><hr><p><a href="https://sourcediving.com/understanding-n-1-query-problems-in-ruby-on-rails-b2ea4be86a2d">Understanding n+1 query problems in Ruby on Rails</a> was originally published in <a href="https://sourcediving.com/">Source Diving</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[This Rails cache is not your friend!]]></title>
            <link>https://sourcediving.com/this-rails-cache-is-not-your-friend-512871c138aa?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/512871c138aa</guid>
            <category><![CDATA[rails-performance]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[activerecord]]></category>
            <category><![CDATA[caching]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Fri, 16 Sep 2022 08:39:21 GMT</pubDate>
            <atom:updated>2022-09-21T08:27:25.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vf7QUJljX-pNccSN0ToSyQ.jpeg" /></figure><p>I recently learned that some of the assumptions I had made about one of Rails’ features were completely wrong, and this would sometimes lead me to write sub-optimal code. This was a delightfully frustrating discovery to make, having working with Rails for so long.</p><p>I had always been aware of this feature, but hadn’t spent much time thinking or caring about how it works and the implications of its use. And so I was very impressed to learn that it is not as it seemed…</p><blockquote>The ActiveRecord::QueryCache is actually quite expensive!</blockquote><h4>What is the ActiveRecord::QueryCache?</h4><p>As the name suggests, the ActiveRecord::QueryCache (I&#39;ll just write ARQC from here on) caches database queries made to the database by ActiveRecord. Within a particular request (or job or Rake task), the results from each database query are cached as a string in memory. Subsequent, identical queries fetch their results from the memory-store, rather than the database.</p><p>This has the beneficial effect of reducing the number of database queries that are made during a request. Since database read is a major bottleneck for most web applications, the ARQC helps to reduce the burden on limited resources and boost response speed.</p><p>The ARQC is enabled by default in ActiveRecord.</p><h4>A code example</h4><p>Consider the following code, which provides helpers to a view that loads the curently authenticated user:</p><pre>class ApplicationController &lt; ActionController::Base<br>  helper_method :current_user, :signed_in?<br><br>  private<br><br>  def current_user<br>    User.find_by(id: session[:user_id])<br>  end<br><br>  def signed_in?<br>    !!current_user<br>  end<br>end</pre><pre>&lt;% if signed_in? %&gt;<br>  Welcome &lt;%= current_user.name %&gt;! &lt;%= link_to(&quot;My profile&quot;, user_path(current_user)) %&gt;.<br>&lt;% end %&gt;</pre><p>This seemingly ordinary Rails code calls the current_user method three seperate times. In each case, a database query is called to load a user record from the database using the ID stored in the session cookie.</p><p>The Rails logs for this request might contain something like this:</p><pre>User Load (0.5ms)  SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = ? LIMIT ?  [[&quot;id&quot;, 1], [&quot;LIMIT&quot;, 1]]<br>CACHE User Load (0.0ms)  SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = ? LIMIT ?  [[&quot;id&quot;, 1], [&quot;LIMIT&quot;, 1]]<br>CACHE User Load (0.0ms)  SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = ? LIMIT ?  [[&quot;id&quot;, 1], [&quot;LIMIT&quot;, 1]]</pre><p>Notice the “CACHE” prefix at the start of the second and third lines. These indicate to us that these two queries were fetched by the ARQC and did not query the database directly.</p><h4>A slightly more complicated code example</h4><p>The above example is very simplistic. But consider this slightly more complicated case. What if the authenticated user were to click the “My profile” link and view the own profile page?</p><p>The users controller to load a profile might look something like this:</p><pre>class UsersController &lt; ApplicationController<br>  def show<br>    @user = User.find(params[:id])<br>  end<br>end</pre><p>Here we are using a different method (find vs find_by) and a different source of data (params[:id] vs session[:id]) for the user ID, but if a user views their own profile then the resulting database query might be the same in both cases:</p><pre>SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = 1 LIMIT 1;</pre><p>It may not always be clear from reading the code alone that duplicate queries are being executed, but these should always be reported in the development logs.</p><h4>Issues with ActiveRecord::QueryCache</h4><p>Typically, when we hear the word “cache”, we think of some means of storage that is optimised for performance. I believe this leads a lot of engineers (including me) to assume that the ARQC is fine to use often. This assumption is reinforced by the Rails log output, which always reports a load time of 0.0ms when returning records via the ARQC (see the log output example above).</p><p>However, this reported load time of 0.0ms may be misleading. While it is indeed correct to report that 0.0ms of time was spent loading these results <strong>from the database</strong>, it is not also the case that 0.0ms was spent to load the records <strong>from the query cache</strong>.</p><p>I’ve demonstrated this in a demo application which you can view here: <a href="https://github.com/bodacious/query-cache-demo">Query Cache Demo</a>.</p><p>In this application, two versions of the same page are provided. One fetches records from the ARQC and one <em>memoizes</em> records — storing them in memory within the code.</p><p>The benchmarks demonstrate a striking difference between these two implementations!</p><p>Running on my own MacBook, the difference between cached and memoized was 19.7s and 7.8s respectively. In other words: the endpoint that relied on the ARQC took more than 2.5x longer than the optimised endpoint to serve the same number of requests.</p><p>I encourage you to run these benchmarks for yourself, and explore the code to better understand the differences.</p><h4>Why so slow?</h4><p>While the ARQC offers an improvement over making multiple calls to the database, it still adds overheads to the cost of serving a response. This is because the cache only stores the SQL string that is returned from the database software. Each time a cached query is called again, Rails still has to go through the effort of converting that SQL string into ActiveRecord objects. This process includes instantiating a new instance or collection of instances, casting and setting their attributes, and also running the after_initialize and after_find callbacks.</p><p>When we compare the number of objects allocated in each experiment branch of the demo application, <strong>we can see a stark difference in the number of allocations</strong>. The following excerpt is from the development logs for each endpoint:</p><pre>Started GET &quot;/memoized/users/1/recipes&quot; for ::1 at 2022-08-16 20:15:55 +0100<br>  ...<br>  Rendered collection of recipes/_recipe.html.erb [15 times] (Duration: 12.2ms | Allocations: 833)<br>  Rendered recipes/index.html.erb within layouts/application (Duration: 23.1ms | Allocations: 2045)<br>  ...<br>  Rendered layout layouts/application.html.erb (Duration: 25.5ms | Allocations: 3271)<br>Completed 200 OK in 34ms (Views: 29.9ms | ActiveRecord: 0.8ms | Allocations: 4183)<br><br><br>Started GET &quot;/cached/users/1/recipes&quot; for ::1 at 2022-08-16 20:16:05 +0100<br>  ...<br>  Rendered collection of recipes/_recipe.html.erb [15 times] (Duration: 5.9ms | Allocations: 3539)<br>  Rendered recipes/index.html.erb within layouts/application (Duration: 7.6ms | Allocations: 4470)<br>  ...<br>  Rendered layout layouts/application.html.erb (Duration: 9.3ms | Allocations: 5680)<br>Completed 200 OK in 11ms (Views: 9.5ms | ActiveRecord: 0.6ms | Allocations: 6347)</pre><p><em>Note: Ignore the render times in these examples. The can vary widely in a local development environment.</em></p><p>These numbers demonstrate the <em>hidden</em> costs of ARQC, and are something we should be mindful of when optimising our code.</p><h4>What’s the alternative?</h4><p>There are likely to be a few ways you can re-write your code to avoid the additional delays caused by re-initializing records multiple times per request. The simplest and most obvious one deserves mentioning though: memoization. By memoizing methods that trigger SQL queries, we can store the resulting record in-memory, rather than having to fetch a SQL results string from the query cache. The following code snippet refactors the above code, removing the reliance on the ARQC:</p><pre>class ApplicationController &lt; ActionController::Base<br>  helper_method :current_user, :signed_in?<br><br>  private<br><br>  def current_user<br>    @_current_user ||= User.find_by(id: session[:user_id])<br>  end<br><br>  def signed_in?<br>    !!current_user<br>  end<br>end</pre><pre>&lt;% if signed_in? %&gt;<br>  Welcome &lt;%= current_user.name %&gt;! &lt;%= link_to(&quot;My profile&quot;, user_path(current_user)) %&gt;.<br>&lt;% end %&gt;</pre><p>Note that it was not neccesary to memoize signed_in?. Only current_user actually calls the database.</p><p>The Rails logs for this request should now look something like:</p><pre>User Load (0.5ms)  SELECT &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = ? LIMIT ?  [[&quot;id&quot;, 1], [&quot;LIMIT&quot;, 1]]</pre><p>Now there is only one query described. The two CACHED queries in the previous example are no longer present.</p><h4>Conclusion</h4><p>The ARQC is a useful addition to the Rails suite. It can make hacking and prototyping faster, without us having to worry about making many redundant queries to the database.</p><p>However, as the described benchmarks demonstrate, this cache is not free, and adds a degree of <em>technical debt</em> to an application (much like n+1 queries do).</p><p>Developers who are seeking to optimise a particular part of their code should be mindful of the overheads that ActiveReccord::QueryCache adds, and look for alternative ways of expressing their solution that do not rely on it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=512871c138aa" width="1" height="1" alt=""><hr><p><a href="https://sourcediving.com/this-rails-cache-is-not-your-friend-512871c138aa">This Rails cache is not your friend!</a> was originally published in <a href="https://sourcediving.com/">Source Diving</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we improved our Rails app’s performance with Conditional Get Requests]]></title>
            <link>https://sourcediving.com/how-we-improved-our-rails-apps-performance-with-conditional-get-requests-35a7a472a0b9?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/35a7a472a0b9</guid>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[web-performance]]></category>
            <category><![CDATA[rails-performance]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Tue, 31 May 2022 11:17:21 GMT</pubDate>
            <atom:updated>2022-12-02T17:39:43.859Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZqmemPKq8b1PJByRfMnaGg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@veri_ivanova?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Veri Ivanova</a> on <a href="https://unsplash.com/collections/yVu6sldRUVI/stopwatch?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>HTTP provides a method of client-side caching known as <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests">Conditional Get Requests</a>. This style of caching allows a client to cache the content of a response locally (in your browser cache or mobile device). When the client makes a subsequent request to your application for the same resource, it includes a token or timestamp from their previous request. Based on this token or timestamp, if your application determines that the response body would be the same as the last request, the server may respond with a short, head-only, 304 Not Modified response. This instructs the requesting client to fetch the response body from its local cache store instead.</p><p>This method of caching can reduce latency and server burden, since it saves our application from having to fetch many records from our database and from having to render view templates when a resource hasn’t changed. For resources that tend not to change too often, or are requested multiple times by the same client without changing, <em>Conditional Get Requests</em> can be a simple and powerful tool for improving performance.</p><p>In May 2022, we conducted an experiment to determine if we could use this method of caching at <a href="https://cookpad.com/">Cookpad</a> to improve response times on selected API endpoints. This post documents how we used this feature to shave up-to 150ms off our response times.</p><h4>Beyond default behaviour</h4><p>Modern web frameworks and networking libraries are quite full-featured. Our mobile clients (<a href="https://play.google.com/store/apps/details?id=com.mufumbo.android.recipe.search&amp;hl=en_GB&amp;gl=US">Android</a> and <a href="https://apps.apple.com/us/app/cookpad-find-share-recipes/id585332633">iOS</a>) were already sending the required headers for <em>Conditional Get Requests</em> with every GET request to our backend, but our API—a Ruby on Rails app—was effectively ignoring these and rendering responses afresh with each request.</p><p>Rails does, of course, support <em>Conditional Get Requests</em> straight out of the box, and our application would respond with a 304 Not Modified if it recognised a resource hadn&#39;t changed since the previous request. But our default implementation was still doing the work of fetching the same records from the database and rendering the same view template, regardless of the <em>Conditional Get</em> headers, so the performance benefit was minimal. To reap the benefits of client-side caching required some thoughtful configuration and refactoring.</p><h4>Choosing our test subject</h4><p>To find good candidates for client-side caching, we used <a href="https://www.charlesproxy.com/">Charles Proxy</a> to monitor our own personal use of the mobile application. Certain endpoints stood out as being likely to be visited by a browsing user more than once in a short period of time, without the response content having changed. This made them a good candidate for client-side caching.</p><p>We nominated the /user/:user_id/recipes endpoint as a test subject. This is an endpoint in our API that presents all of the recipes belonging to a given user. Next, we discussed <em>which method</em> we would use to determine if a resource was changed or not.</p><p>HTTP supports more than one way of validating whether a resource has changed or not (called <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests#validators">validators</a>), and more than one validation method can be used in the same request. In this particular case, this recipes endpoint can present different data based on query parameters (such as page and per_page), so we decided that <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag">ETags</a> were a more accurate way to validate if a resource had changed than a <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified">last modified</a> timestamp.</p><p>Rails sends an ETag header with each response by default, which represents a hash digest (like a fingerprint) of the response body that is essentially unique. As mentioned before, our mobile clients were already sending this ETag back to the server in subsequent requests (as a header named HTTP-IF-NONE-MATCH). All that was required then, was that we update our controllers to only load resources and render views <em>if</em> the hash digest was different from the one expected.</p><h4>Updating our controllers</h4><p>To add this behaviour, we used two Rails methods: <a href="https://api.rubyonrails.org/classes/ActionController/ConditionalGet.html#method-i-stale-3F">#stale?</a> and <a href="https://api.rubyonrails.org/classes/ActionController/ConditionalGet/ClassMethods.html#method-i-etag">#etag</a>.</p><h4>#stale?</h4><p>The #stale? method provided by Rails will compare the hash digest of a given value or object with the value provided in the HTTP-IF-NONE-MATCH request header. If the two values are the same, #stale? will return false and the action can respond with a 304 Not Modified without having to load and render additional data. If the two values differ, #stale? returns true and the page should be loaded and rendered in full.</p><p>Here’s an example to help you visualise how this might look:</p><pre>def index<br>  @user = User.find(params[:user_id])<br>  if stale?(etag: @user.updated_at)<br>    @recipes = @user.recipes<br>  else<br>    # head 304<br>  end<br>end</pre><p>In this example, the ETag validator includes the current user’s updated_at timestamp. <a href="https://github.com/rails/rails/blob/cb52350f3829803b7408c230268224ef8b92daef/actionpack/lib/action_dispatch/http/cache.rb#L134-L136">Behind the scenes</a>, Rails will add this (and potentially other) values to an array and then hash all of those values to generate a unique hexadecimal string-the ETag. If that string matches the value in the HTTP-IF-NONE-MATCH header, #stale? will return false and the controller will respond 304 Not Modified.</p><h4>#etag</h4><p>Calculating whether a response matches an ETag or not is not always as straightforward as checking just a simple timestamp. Some endpoints include query parameters (such as pagination), and some endpoints render a resource differently depending on which user is making this request.</p><p>To factor these variable values into the entire digest for a response ETag, Rails provides the etag method. Simply call this method in the controller, with a block, to add any additional validators to the array of values for the final ETag digest.</p><p>For example:</p><pre>etag { params[:page] }<br><br>etag { Current.user.id }<br><br>def index<br>  # ...<br>end</pre><p>To make our lives easier, we added an additional class method to our controllers .include_params_in_etags to simplify adding parameters and their values to the validators array.</p><pre>module ETagParamKeys<br>  extend ActiveSupport::Concern<br><br>  module ClassMethods<br>    private<br><br>    def include_params_in_etags(*param_names)<br>      Set.new(param_names).each { |p| include_param_in_etags(p) }<br>    end<br><br>    def include_param_in_etags(param_name)<br>      etag { params.key?(param_name) &amp;&amp; &quot;#{param_name}-#{params[param_name]}&quot; }<br>    end<br>  end<br>end</pre><h4>A complete example</h4><p>Here’s a complete example that connects the three concepts described above:</p><pre># The value of these parameters will be included as part of<br># the array used to create the ETag digest<br>include_params_in_etags :page, :per_page<br><br># The current User&#39;s ID will be added to the ETag, making it unique<br># per authenticated User<br>etag { Current.user&amp;.id }<br><br>def show<br>  if stale?(etag: etag_value)<br>    # Slow query we&#39;d rather not make if we can avoid it<br>    @recipes = user.recipes.published.page(params[:page]).per(params[:per_page])<br>  else<br>    # head 304<br>  end<br>end<br><br>private<br><br>  def etag_value<br>    # This value is store in an in-memory cache and is super quick!<br>    @_etag_value ||= user.latest_recipe_updated_at<br>  end<br><br>  def user<br>    @_user ||= User.find(params[:user_id])<br>  end</pre><p>Our implementation in this experiment was essentially the same as described above, but with some additional lines of code for collecting performance metrics and other such things.</p><h4>To invalidate or not to invalidate?</h4><blockquote><em>There are only two hard things in Computer Science: cache invalidation and naming things.</em></blockquote><blockquote><em>— Phil Karlton</em></blockquote><p>Implementing these changes in the code was fairly straightforward. The real difficulty came in determining the most effective yet reliable way of invalidating our cache.</p><p>Our initial PR was blocked, because other engineers started to spot potential cases where the response data might have changed and our ETag value wouldn’t detect it. This led to quite a long and detailed discussion in which we tried to identify as many potential cases as possible. In the end, we opted for a cached timestamp that would be invalidated every time a user or any of their recipes changed. This additional method call added an extra few milliseconds to the response time, but not enough to outweigh the overall benefit.</p><h4>Results</h4><p>We used Prometheus to record the response times for each request to this endpoint, and also to track whether the ETag was stale or not (a cache “miss” or a cache “hit”).</p><p>The results of this experiment are shown in the charts below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jxfyDeJVftmzdnQHOMbwhg.png" /><figcaption>Chart showing the “hit”, “miss”, and average response times (ms) for this endpoint</figcaption></figure><p>In this first chart, the Y-axis shows the response time (ms), and the X-axis is the time. The three lines indicate cache hits, cache misses, and the average response time for the endpoint (both hits and misses).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JoM-3nTYbx-_rq1a4o7VlQ.png" /><figcaption>Chart showing the cache hit rate (%) for this endpoint</figcaption></figure><p>In this second chart, the Y-axis shows the % cache hits, and the X-axis is the time.</p><p>As you can see from these results, cache <em>miss</em> responses tended to range between 220ms and 310ms. Whereas cache <em>hits</em> were consistently around 95ms. The cache hit rate was around 25%, but we saw this growing steadily over time. This had the effect of bringing the <strong>average response time down between 30ms and 70ms</strong>, roughly 10–20%, and that improvement has increased with the cache hit rate.</p><p>Adding <em>Conditional Get Requests</em> improved response time <em>dramatically</em> for about 25% of requests, and improved the average response time for all requests to this endpoint. Not bad for a few hours of work!</p><h4>Caveats</h4><p>Implementing Conditional Get Requests in Rails is surprisingly simple.</p><p><em>However</em>, it is important to ensure that the logic for determining a stale vs fresh cache is sound, and covers all scenarios. It is very easy to miss certain edge cases or certain combinations of variables, and doing so could mean serving stale, out of date content to a user.</p><p>We ended up writing several tests to thoroughly cover the cases we thought of, and to provide a layer of insurance that future changes wouldn’t break the cache behaviour.</p><h4>Cache lifespan</h4><p>Cached responses may live on a device for a long time. This means that improperly configured ETag values could result in users experiencing bugs or viewing out-of-date information for several days or weeks. Since these errors take place on the client-side, and not server-side, it is very difficult to know if problems arise unless our users start to complain!</p><p>This issue can be mitigated by customising the :cache_control option in the #stale? method to set a shorter cache expiry time, but that is outside of the scope of this article.</p><h4>Proxies and privacy</h4><p>It’s possible that, depending on the cache-control headers you set, caching a response body may also happen in one of the proxies along a network. You should ensure you understand the cache-control headers, and how these are configured for your application. Make sure this also aligns with your company’s privacy policy and other legal requirements too; if in doubt, err on the side of caution.</p><h4>YMMV</h4><p>The above described experiment demonstrates an average improvement of 10%-20% in response times for this endpoint. However, the impact of conditional get requests is likely to vary significantly for each endpoint. <em>Your Mileage May Vary</em>. I would encourage teams to A/B test the effectiveness of their implementation as part of their workflow, and compare the results with the default implementation, to prove that the overall benefit justifies developing and managing this feature.</p><h4>Conclusion</h4><p><em>Conditional Get Requests</em> offer an easy and potentially powerful way to improve the response times for a given endpoint. In our experiment, we were impressed by how high the cache hit rate was, and how much faster cached responses were. However, it is essential that you ensure the conditions you have identified for cache validity are realistic and comprehensive.</p><p>Teams are advised to measure and benchmark their implementation to gain a deeper understanding of the impact (if any) their changes have made.</p><p>Remember to share your findings with others 🤓</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35a7a472a0b9" width="1" height="1" alt=""><hr><p><a href="https://sourcediving.com/how-we-improved-our-rails-apps-performance-with-conditional-get-requests-35a7a472a0b9">How we improved our Rails app’s performance with Conditional Get Requests</a> was originally published in <a href="https://sourcediving.com/">Source Diving</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to host a static website on AWS S3 with automated CircleCI deploys]]></title>
            <link>https://medium.com/@gavinmorrice/how-to-host-a-static-website-on-aws-s3-with-automated-circleci-deploys-771e7953a0a7?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/771e7953a0a7</guid>
            <category><![CDATA[jekyll]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloudflare]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Fri, 24 Jul 2020 00:00:00 GMT</pubDate>
            <atom:updated>2020-07-28T14:53:24.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLKyxlZ7sp9fucqksxV5GQ.jpeg" /></figure><p>Jekyll is a framework for quickly building static HTML websites that supports SASS, reusable templates, and layouts. It’s built in Ruby and can be extended with third-party plugins and your own code.</p><p>Amazon’s AWS offers super-simple, super-cheap, and efficient static website hosting.</p><p>CircleCI is a continuous integration platform that can automatically build and deploy your code any time you push out changes to GitHub.</p><p>In this post, I’ll talk you through the steps for hosting your static Jekyll website on AWS, and setting it to automatically deploy every time you push changes to your git repo.</p><h3>Before we get started</h3><p>This post assumes you already have:</p><ul><li>created your Jekyll website (if not, you can <a href="https://jekyllrb.com/docs/">follow this tutorial</a>)</li><li>set up a Git repository for your website on GitHub</li><li>purchased a domain for your website (if not, a subdomain will do, or you can <a href="https://aws.amazon.com/route53/">buy a domain here</a>)</li><li>some familiarity with AWS and CircleCI</li></ul><p>Once those items are in place, we’re ready to get deployin’…</p><h3>Creating an S3 bucket</h3><p>If you haven’t already created an AWS account, <a href="https://portal.aws.amazon.com/billing/signup#/start">you can do so here</a>. If you have, sign in to your account and navigate to the S3 section within the console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/0*u3NRiXsZAgVl15vR.png" /></figure><p>Create a new Bucket by clicking on the “Create bucket” button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*FBx_coFRtvGzYVV_.png" /></figure><p>Configure your bucket as shown below. <strong>Be sure to name your bucket as the domain name that you’re hosting your website on</strong>. This is a requirement set by AWS, and the hosting won’t work without it.</p><p>Choose the region that makes the most sense for you. For us, this is the <strong>eu-west-2</strong> (London) region, because we’re based in Scotland and so are most of our clients. This part isn’t <em>super</em> important if you’re using a CDN like CloudFlare though, since a distributed network will have cache locations close to your visitors, wherever they might be.</p><p>You’ll notice I’ve unchecked the public access checkboxes. This is because your bucket will require public access in order to make the HTML, CSS, JS and other asset files of your website visible to the public. AWS will ask you to confirm this before creating the bucket. <strong>It’s important that you do not store any sensitive documents in this bucket</strong> since they will be openly available to the public for viewing.</p><p>Leave all of the other settings as their default values for now, and click the “Create bucket” button to proceed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v3HFVNqeJtqSNT23.png" /></figure><p>Now click on your new bucket in the list of all your buckets. The empty bucket page should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A_eWbYmLxq_YYr4n.png" /></figure><p>Next, we want to tell AWS that we intend to use this bucket to host a static website. Do that by clicking on the “Properties” tab and then clicking “Static website hosting”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oEn6mjgPEW_8vWmM.png" /></figure><p>Click on the radio button labelled “Use this bucket to host a website” and fill in the form fields that appear. The default values index.html and error.html should be fine for now-It&#39;s unlikely that your index file will have a different name. You&#39;ll want to configure an error file on your page to handle cases like <strong>404 Page Not Found</strong> errors. By default, this is called error.html, but you can name it something else if you&#39;d prefer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8IFdb8SMiMWvJQ1r.png" /></figure><p>Click the <strong>Save</strong> button, and your bucket should now be (almost) set up and ready to serve your static pages. There’s just one more bucket configuration we need to change, and that’s to create a “Bucket Policy” that allows public GET requests to the objects within our bucket. In other words, we need to tell AWS that members of the public may access our site’s files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i6JLDPKTzBaBY--L.png" /></figure><p>Here’s the code from the Bucket Policy. I’ll explain what it means below…</p><p>This <em>Policy Statement</em> is how we tell AWS that we want to <em>allow</em> all <em>GET</em> requests on objects (files) within the Resource named “arn:aws:s3:::[your-bucket-name]/*” (our bucket). This should permit people to view the content of our website, but not add, remove, or modify any of the files without the proper access.</p><h3>Test that it works!</h3><p>Now we need to test that the steps we’ve just implemented have worked, and our bucket is configured properly to serve static files.</p><p>To do this, we’re going to create an index.html file and manually upload it to the bucket.</p><p>In your favourite text editor, create a file named index.html, and edit it with the following:</p><p>Save this file (it doesn’t matter where), and then upload it to your bucket by clicking on the “Upload” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BZ0yip7A_7MSqyd-.png" /></figure><p>It’s fine to tap the “Next” button here and use AWS’s default settings for uploading this file.</p><p>Once your file has been uploaded, you should be able to access it via its AWS URL. This will be unique to your own settings, but should be in the format of: [bucket name].s3-website.[aws region].amazonaws.com. In my case, my bucket name is jekyll-aws-demo.katanacode.com and my region is eu-west-2, so the URL of my website is: <a href="http://jekyll-aws-demo.katanacode.com.s3-website.eu-west-2.amazonaws.com/">http://jekyll-aws-demo.katanacode.com.s3-website.eu-west-2.amazonaws.com</a></p><p>When I visit that URL <a href="http://jekyll-aws-demo.katanacode.com.s3-website.eu-west-2.amazonaws.com/">http://jekyll-aws-demo.katanacode.com.s3-website.eu-west-2.amazonaws.com</a>, I can see that it’s working because my “Whew! It works” index file is loading!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vY8lUBAc5RzIvmR8.png" /></figure><p><strong>Note:</strong> AWS doesn’t support the “https” protocol (we’ll fix that later), so make sure you’re using “http” for now.</p><h3>Setting up permissions to upload your site</h3><p>We’ve set a policy for the public to read files from our site, and now we can see that AWS is serving static files from our newly created bucket. But we also need to set some permissions that allow us to programmatically upload, edit, and remove files from our bucket so we may make changes to our site.</p><p>The best way to do this is to create an IAM policy and assign this policy to an AWS IAM user account. To do this, click on the services tab and choose the IAM option from the list of services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ECFcRaNfhAN-Dsnm.png" /></figure><p>First, we need to create a policy. Do that, by clicking on the “Policies” tab, and then clicking the “Create policy” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B2jEJAZ9aWm52qB7.png" /></figure><p>Click the <strong>Service</strong> tab, and type “S3” to specify that we want our policy to allow S3 access. You’ll then be able to choose permissions from a long list of options in the <strong>Actions</strong> tab. As a rule, we want to limit the permissions to only those we require. This will help keep our site, and the other services we use more secure. The only boxes you need to check for this policy are the following:</p><ul><li>List &gt; ListBucket</li><li>Read &gt; GetBucketLocation</li><li>Read &gt; GetObject</li><li>Write &gt; DeleteObject</li><li>Write &gt; PutObject</li></ul><p>Next, expand the <strong>Resources</strong> tab and specify the bucket name. This should be the name of your bucket, which you should remember, will be the same as the domain of your website. Lastly, specify the object name as “*”. By setting these values, we’re telling AWS that the permissions we’re asking for should only apply to this bucket and any of the objects inside it.</p><p>If you’ve set all that up correctly, it should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V1NK_f-vpv7u2fg8.png" /></figure><p>Click “Review policy” and give your policy a name and helpful description. I’ve called mine “jekyll-demo-policy”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TSMV0532CCchttXM.png" /></figure><p>Click the “Create policy” button to finalise completion of your policy.</p><p>Now, we need to create a <em>User</em>, so that we can generate an access key and secret. These will allow us to authenticate ourselves and access our bucket so we can make changes to our website programmatically.</p><p>Click the “Users” tab, and then the “Add user” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vOe1KCo5ywrmCYT2.png" /></figure><p>Give your user a name (I’ve called mine “jekyll-demo-user”) and choose <strong>Programmatic access</strong> for the <strong>Access type</strong>. Then click the “Next: Permissions” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P53LjBt9J_Vi833Q.png" /></figure><p>Here, you should choose the option to “Attach existing policies directly”, and filter the list of available policies to only include “Customer managed” (the policies you own). This way we can directly assign the new policy we just created with the user we’re creating, without having to create permission groups etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-l0HV-yYCMqvBi1U.png" /></figure><p>From there, you should see your own, newly created policy in the list. Check the checkbox next to that item, and click “Next: Tags”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qVcSw1P2DcfAk1GP.png" /></figure><p>We don’t need to add any tags here, so just click the “Next: Review” button to move on to the final screen. Make sure the details there look correct, and if you’re happy, click the “Create user” button.</p><p>Copy the Access key ID and Secret access key, and store these someplace safe. You won’t get the chance to view that secret access key again, so it’s important you don’t lose it. Also, <strong>make sure to protect these credentials, and do not share them with anyone you do not trust!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pEQiWq2EoTcxASpU.png" /></figure><p>Now we can test our credentials are working using the AWS CLI, which is a tool that lets us communicate with the AWS API straight from the command line. If you don’t already have this installed, head over to the <a href="https://aws.amazon.com/cli/">AWS CLI page</a> now and follow the installation instructions.</p><p>Assuming you have the AWS CLI installed, let’s try uploading our Jekyll site to our bucket!</p><p>The AWS tool we’re going to use for uploading out site to our S3 bucket is s3 sync.</p><p>The full syntax is aws s3 sync [origin] [target] [options].</p><p>First, in order to make the next few steps a bit easier, let’ss set our newly created access key and secret access key as environment variables in our bash terminal. You might want to create a profile on your local machine and store them long term. If so, the instructions for doing so can be found <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html">here</a>.</p><p>So in my case, I would type:</p><pre># I&#39;ve hidden parts of my key ID and access key for security export AWS_ACCESS_KEY_ID =AKIA54LT4OEXXXXXXXXX export AWS_SECRET_ACCESS_KEY =ZYe4cF3ydDfaNPe8XXXXXXXXXXXXXXXXXXXXXXXX export AWS_DEFAULT_REGION =eu-west-2</pre><p>Now, assuming you’re in the root directory of your Jekyll site, you should be able to upload the whole site to your S3 bucket by running the following two commands:</p><pre># Make sure the site is built in _site jekyll build # Upload the site files aws s3 sync _site s3://your-bucket-name</pre><p>If that works, you should see a list of the files that have been uploaded from your local _site directory to your S3 bucket:</p><pre>jekyll-demo(master) ✗:aws s3 sync _site/ s3://jekyll-aws-demo.katanacode.com upload: _site/404.html to s3://jekyll-aws-demo.katanacode.com/404.html upload: _site/about/index.html to s3://jekyll-aws-demo.katanacode.com/about/index.html upload: _site/assets/main.css to s3://jekyll-aws-demo.katanacode.com/assets/main.css upload: _site/feed.xml to s3://jekyll-aws-demo.katanacode.com/feed.xml upload: _site/index.html to s3://jekyll-aws-demo.katanacode.com/index.html upload: _site/jekyll/update/2020/07/24/welcome-to-jekyll.html to s3://jekyll-aws-demo.katanacode.com/jekyll/update/2020/07/24/welcome-to-jekyll.html upload: _site/assets/minima-social-icons.svg to s3://jekyll-aws-demo.katanacode.com/assets/minima-social-icons.svg upload: _site/assets/main.css.map to s3://jekyll-aws-demo.katanacode.com/assets/main.css.map</pre><p>Finally, you can check that the files are being served properly by going back to your bucket’s public URL: http://[your bucket].s3-website.[your region].amazonaws.com</p><p>You should no longer see “Whew! It works”, and instead you should see your own site.</p><h3>Setting up CircleCI</h3><p>Now that we are able to update the contents of our bucket with a single command, we have the opportunity to delegate this task to an automation service like CircleCI. The benefit of this is not only that we can save ourselves a bit of time and effort deploying, but also that we can run additional checks (such as linters), and schedule nightly builds (see why later).</p><p>If you don’t already have an account with CircleCI, you can <a href="https://circleci.com/signup/">set up a free account here</a>.</p><p>Once you’ve signed up or signed in to your CircleCI account, click the “Add Projects” button and choose your Jekyll website’s git repo from the list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RNHZopM5oBVlz14f.png" /></figure><p>CircleCI will then give us two options. For our project to build properly on CircleCI, our repository needs to have a config.yml at .circleci/config.yml with instructions for how to create the environment and build the site. CircleCI offers to add a boilerplate config file for us, or we can choose to add one of our own config files. Click &quot;Add manually&quot; so we can add our own config file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D7z1Fv3lSaDfb9Ok.png" /></figure><p>Click “Start building”.</p><p>Now, in your favourite text editor, copy and paste this YAML code, then save it in your site’s repository as .circleci/config.yml. (Don&#39;t forget the &quot;dot&quot; at the start of the directory name).</p><p>If you have some experience with CircleCI, then this should hopefully make sense to you. But if not, I’ll try to summarise what’s included here.</p><p>The references: block defines some reusable <em>mixins</em> that are used across more than one job. In this case, we&#39;re defining the directory name for our repo, and the docker image we are using to build our site within.</p><p>The orbs: block defines some Orbs for the Amazon AWS CLI and for Bundler. Orbs are CircleCI&#39;s way of sharing code between projects and the open source community.</p><p>The jobs: block is where we define the steps that CircleCI will implement each time we push our a new Git commit. In our case, we define a job that builds the site, and another job that deploys the code to AWS.</p><p>Lastly, the workflows: block is where we define a workflow or a collection of jobs that run together. Here we&#39;ve said that <strong>build</strong> should run on every new git commit, and <strong>build</strong> and <strong>deploy</strong> will both be run when a new commit is pushed out to the <strong>master</strong> branch.</p><p>By configuring our project this way, we can push out new code on our feature branches without having to worry about it being prematurely deployed to our live site. Only code merged into the <strong>master</strong> branch will be deployed.</p><h3>Adding environment variables in CircleCI</h3><p>You’ll probably see that the first workflow in your CircleCI dashboard has failed. That’s fine. In this case, it’s because we haven’t included the config.yml file yet. But there&#39;s one more thing we have to do before we push our changes into <strong>master</strong> and run the <strong>deploy</strong> job. Without this next step, our CircleCI workflow will still fail.</p><p>Remember the Access Policy we created on our AWS account so we can restrict who may upload and edit files in our bucket? Well, we need to tell CircleCI about those credentials in order for AWS CLI tools to work on CircleCI. To do this, click on the “Project Settings” button, and then choose “Environment Variables”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SswIzXIHHcrSLVf2.png" /></figure><p>Define each of the following variables with the appropriate values as described below:</p><pre>JEKYLL_ENV = &#39;production&#39; AWS_ACCESS_KEY_ID = &quot;[Your Access Key ID]&quot; AWS_SECRET_ACCESS_KEY = &quot;[Your Secret Access Key]&quot; AWS_BUCKET_NAME = &quot;[Your bucket name]&quot; AWS_DEFAULT_REGION = &quot;[Your AWS region (e.g. &#39;eu-west-2&#39;)]&quot;</pre><p>Now, all we have to do to trigger a <strong>build</strong> and <strong>deploy</strong> is to commit our config.yml file to our <strong>master</strong> branch and push the changes out to GitHub.</p><pre>$ git add .circleci $ git commit -m &quot;Add CircleCI config.yml&quot; $ git push origin master</pre><p>In your CircleCI dashboard you should, after a few seconds, see a successful and build deploy workflow.</p><h3>Improve performance with CloudFlare</h3><p>So far, we have a static website that we can build and upload to AWS S3 with just a single git commit. I’m sure you’ll agree this is a very useful workflow for productivity. But there are a couple of things we need to complete before we can relax.</p><p>We still can’t view our website at our chosen domain yet, and we are still <em>downloading</em> content from AWS S3 each time we visit our site. This can be expensive over time, since AWS charges per download bandwidth.</p><p>To resolve both of these issues, we prefer to use a service called CloudFlare. CloudFlare is a CDN or <a href="https://www.cloudflare.com/learning/cdn/what-is-a-cdn/">Content Delivery Network</a> that acts as a cache layer between our visitors and our AWS Bucket. It’s also a great tool for adding some additional security, including SSL, and for managing DNS records. And it’s FREE!</p><p>If you don’t already have a CloudFlare account, <a href="https://dash.cloudflare.com/sign-up">head over to this page and sign up</a>. Once you’ve signed in, follow the online instructions for adding a new domain. This will be different depending on your own setup and domain provider, but CloudFlare should guide you through that process.</p><p>Once your domain is set up, click on the “DNS” tab, and create a new CNAME record for your domain. The name of the CNAME should be the domain we want people to visit in their browser, and the value should be our AWS URL we used earlier (the one that looks like this: [bucket name].s3-website.[aws region].amazonaws.com).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6j7CUxpKGjc63Hyh.png" /></figure><p>Now, when you visit your domain you should hopefully see your website loading as expected!</p><p><a href="http://jekyll-aws-demo.katanacode.com/">http://jekyll-aws-demo.katanacode.com</a>.</p><h3>Adding SSL</h3><p>Remember earlier, I said that AWS doesn’t support SSL (“https”) by default? Well, as an optional extra, we can also head over to the SSL/TLS tab and turn on “Flexible” encryption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9bfl3dZ-7hFInKK2.png" /></figure><p>This allows us to offer SSL secure browsing for our visitors, without the need for us to buy and configure our own SSL certificates</p><p>You can check that it’s working by visiting your domain, but this time using the “https” scheme at the start (e.g. <a href="https://your-domain.com)./">https://your-domain.com).</a></p><h3>Cache purging</h3><p>So let’s recap on what we’ve achieved so far. We’ve configured Amazon Web Services to host our static website in a super-low-cost S3 bucket. Next, we configured CircleCI to automatically update our site each time we push changes out to our <strong>master</strong> git branch. And then, we configured CloudFlare to cache our site’s HTML, Javascript, CSS, and image content, to reduce the time it takes for our website to load, and also keep the AWS costs low by reducing bandwidth.</p><p>There’s only one thing left for us to do. Can you guess what it is? If you said we need to automatically purge the CloudFlare cache with each deploy, you guessed right!</p><p>CloudFlare’s caching is an incredibly helpful service, but from time to time we’ll make changes to our website, and when we do, we don’t want the CloudFlare cache to continue to serve visitors the old, stale versions. There are a few different ways to avoid this problem, but the quickest and simplest solution is to simply purge everything in the CloudFlare cache and start afresh. Since we’re not going to be doing this too often, it shouldn’t have any negative impact on how quickly your website loads.</p><p>To automatically purge the CloudFlare cache, we first need a CloudFlare API token. You can get your token by clicking on the Overview tab in the CloudFlare Dashboard.</p><p>Down the right-side column, you should see a heading that reads “API”. Under there, copy your <strong>Zone ID</strong>, and keep a note of it-we’ll need that in a few minutes. Next, click the link that says “Get your API token” and proceed to the next page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/0*RbdNcibNfVc65ZqP.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hoIIR4RpuQG2AmKq.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d_kb1EyQIvjEbb7q.png" /></figure><p>Ignore the various API token templates, and instead click the option to “Create Custom Token”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ycfDcNyfqw5Ulcp9.png" /></figure><p>On this page, we’ll configure the API Token for our new website. Give it a name that makes sense to you (I chose “Purge Cache — Jekyll AWS Demo”), and choose “Zone” &gt; “Cache Purge” &gt; “Purge”.</p><p>This means we’re permitting requests with this API token to purge the cache on the specified <em>zone</em> (domain), and it’s the one and only permission we’re applying to this token.</p><p>Choose the “Zone” that matches your domain name, and click “Continue to summary”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YlMbtHTz_rBy-uAM.png" /></figure><p>Copy your API Token code that’s shown on screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BtlmGtZJrEovLq-_.png" /></figure><p>To test our API Token, CloudFlare gives us a bash snippet we can execute in our own Bash Terminal. Mine looks like this:</p><pre>curl -X GET &quot;https://api.cloudflare.com/client/v4/user/tokens/verify&quot; \ -H &quot;Authorization: Bearer ngVpOCV-dk4xSbQXZ3MQrhfmEUCzofJXXXXX&quot; \ -H &quot;Content-Type:application/json&quot;</pre><p>When I run that, CloudFlare responds with a JSON response that looks like this. I can see from the &quot;status&quot; that it worked!</p><h3>Purge CloudFlare cache from a CircleCI job</h3><p>To use this as part of our deploy workflow, we need to add an extra step to our CircleCI setup.</p><p>First, go back to the Project Settings in CircleCI and set a new environment variable, just like we did with JEKYLL_ENV and AWS_ACCESS_KEY_ID. This time, we want to name it CLOUDFLARE_API_TOKEN and set its value as the API Token we just created in CloudFlare.</p><p>Add another environment variable called CLOUDFLARE_ZONE and set this to the <strong>Zone ID</strong> we noted down earlier.</p><p>Finally, we need to add one last step to our deploy sequence in our config.yml file to tell CircleCI to purge the CloudFlare cache after we redeploy our site. Add the following step in the deploy: job block:</p><p>Your complete config.yml file should look <a href="https://raw.githubusercontent.com/KatanaCode/jekyll-aws-demo/master/.circleci/config.yml">like this</a>.</p><p>Stage and commit these changes on your <strong>master</strong> git branch, and push them out to GitHub.</p><pre>$ git add .circleci/config.yml $ git commit -m &quot;Add CloudFlare Cache Purge to CircleCI&quot; $ git push origin master</pre><p>Head back to the CircleCI dashboard and you should, after few seconds of nervous anticipation, see both jobs have passed successfully and your site was redeployed.</p><p>And that’s it! Now you can start making changes to your static Jekyll site and publishing them on AWS with a simple git push.</p><h3>Optional extra: Nightly builds</h3><p>If you blog a lot and would like to make the most of Jekyll’s ability to hide posts until a specified publish date, you might want to configure CircleCI to automatically build and redeploy your website once per day. With this configured, you can schedule blog posts for a specific date, and let CircleCI do the rest.</p><p>Getting this to work is easy, simply add one more block to the CircleCI config.yml workload:</p><p>With this in place, CircleCI should now automatically deploy your site each day at about midnight.</p><h3>Recap</h3><p>To recap, in this walkthrough we’ve covered how to host a Jekyll website in an AWS bucket. We walked through how to create the minimum permissions necessary to programmatically make changes to our bucket. Next, we configured CircleCI to go through the build and deploy steps for us automatically. Lastly, we configured CloudFlare to act as a CDN and improve the security and performance of our static website.</p><p>I hope you find this setup useful. If you have any ideas for how it can be improved, or for cool tasks we could add to our workflow, please feel free to get in touch or contact me <a href="https://twitter.com/morriceGavin">via Twitter</a>.</p><p><em>Originally published at </em><a href="https://katanacode.com/2020/07/24/how-to-host-jekyll-website-on-aws-s3-with-circle-ci.html"><em>https://katanacode.com</em></a><em> on July 24, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=771e7953a0a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A naming convention for ActionMailer emails]]></title>
            <link>https://medium.com/@gavinmorrice/a-naming-convention-for-actionmailer-emails-68612235ed8a?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/68612235ed8a</guid>
            <category><![CDATA[convention-over-config]]></category>
            <category><![CDATA[actionmailer]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[rails]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 00:00:00 GMT</pubDate>
            <atom:updated>2020-07-10T22:12:30.543Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the great things about Ruby on Rails is its philosophy of <em>Convention over Configuration</em>. Simply put, this means that developers are encouraged to name and organise their code in specific ways, and to stick to these naming principles wherever possible. The benefit of <em>CoC</em> is that developers spend less mental energy trying to make decisions about things like “where should I store my database schema?”, so they can spend more mental energy on building a great product.</p><p>In Rails apps, emails are typically sent by a library called ActionMailer. While ActionMailer is an excellent tool for the job, there’s no common convention in the Rails community on how to name mailer classes and methods. In this post, I’ll share with you the convention I’ve been using on projects I’ve worked on recently.</p><h3>The essence of a mailer</h3><p>To develop a naming convention, we must first reflect on how our code is being used, and look for some common patterns that apply to most use cases. In doing so, I’ve realised that most mailers can all be boiled down to these common features:</p><h4><strong>Recipient</strong></h4><p>The recipient is the entity that will receive the email. This will normally be a person, but it could also be another app or remote service that receives email inputs. Recipients might be your service users, customers, internal admins, or unauthenticated visitors to your website.</p><h4><strong>Resource</strong></h4><p>Emails are usually sent out to provide information about a specific resource or <em>domain concept</em> within the system. In other words, emails are usually sent about something relevant to some <em>thing</em>. An <em>article</em> has been published, a <em>subscription</em> has lapsed, a <em>purchase</em> has been completed. These will often be the models that you’ve defined in your code, but they might also be more abstract concepts (like a <em>password reminder</em>).</p><h4>Change</h4><p>Transactional emails don’t just spontaneously fire off. They’re usually sent in response to an event or some <em>change</em> that’s happened within the system. An example could be when some resource is created, updated, cancelled, or renewed.</p><h4>Intention</h4><p>Services email people for different reasons, which I’ve called the <em>intention</em>. In practice, using this convention, I’ve only ever come across four different intentions when sending emails: <em>notification</em>, <em>confirmation</em>, <em>warning</em>, and <em>provision</em>. You might discover more in your own use-cases, but I haven’t needed more than these in the past few months.</p><p>By identifying these common patterns within our mailers, we can now try to establish a naming convention that communicates what an email is being sent for within our code.</p><h3>What should you name your Mailer class?</h3><p>Web applications can sometimes become very large, and the number of different emails that are sent out can easily grow into the dozens, and this sometimes means large ActionMailer classes with dozens of methods defined in them. Most sensible developers will avoid having just one mailer class, and try to split the emails up into separate mailer classes for different purposes.</p><p>But this raises the question: what should I name my mailer class? Should I name it based on the recipient? Or the subject matter? Or the namespace within my app? A mixture of these?</p><p>My preference is to <strong>name ActionMailer classes based on the recipient</strong>. This should always be the term you use within your business domain. If you call your customers customers, call your mailer CustomerMailer. If you call your customers &quot;guests&quot;, call your mailer GuestMailer. This makes it very clear who is the intended recipient of a given email.</p><h3>What should you name your Mailer method?</h3><p>Just like our class name convention makes it easy to see, at a glance, who the recipient of an email is,<strong><em> </em>your mailer method name should make it clear what the email is about</strong>, and why it’s being sent. With this in mind, I use the following naming convention for mailer methods names: #&lt;resource&gt;_&lt;change&gt;_&lt;intention&gt;. To get a feel for this convention, let&#39;s look at a couple of examples:</p><h4>Example 1:</h4><p>A visitor to your blog receives an email to notify them that their comment has been approved. In this case, the recipient is a <em>visitor</em> (Recipient), and we’re <em>notifying</em> them (Intention) that their <em>comment</em> (Resource) has been <em>approved</em> (Change).</p><p>In this case, we’d define VisitorMailer#comment_approved_notification</p><h4>Example 2:</h4><p>An admin of your business receives an email to notify them that a customer payment has failed. In this case, the recipient is an <em>admin</em> (Recipient), and we’re <em>notifying</em> them (Intention) that their <em>payment</em> (Resource) has <em>failed</em> (Change).</p><p>In this case, we’d define AdminMailer#payment_failed_notification</p><h4>Example 3:</h4><p>A member of your organisation requests a link to reset their password. In this case, the recipient is a <em>member</em> (Recipient), and we’re <em>providing</em> them (Intention) with a <em>password</em> (Resource) <em>reset</em> (Change).</p><p>In this case, we’d define MemberMailer#password_reset_provision</p><h3>Conclusion</h3><p>So there’s my proposed naming convention for ActionMailer methods. I’ve been using this with great success, and it’s one less thing I have to think about when I’m coding apps.</p><p>If you have any thoughts, please feel free to share them with me on Twitter <a href="https://twitter.com/morricegavin">(@morricegavin)</a>.</p><p><em>Originally published at </em><a href="https://katanacode.com/2020/07/10/naming-convention-for-action-mailer-emails-in-rails.html"><em>https://katanacode.com</em></a><em> on July 10, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68612235ed8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trying to define the indefinable]]></title>
            <link>https://medium.com/@gavinmorrice/trying-to-define-the-indefinable-3b0b2b59361c?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b0b2b59361c</guid>
            <category><![CDATA[creationism]]></category>
            <category><![CDATA[evolution]]></category>
            <category><![CDATA[darwinism]]></category>
            <category><![CDATA[intelligent-design]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Sat, 13 Jun 2020 23:00:00 GMT</pubDate>
            <atom:updated>2020-06-14T20:02:36.159Z</atom:updated>
            <content:encoded><![CDATA[<h4>Species arguments can’t save Intelligent Design</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CT00Zr7cstf9s46R.jpg" /></figure><blockquote><em>It is really laughable to see what different ideas are prominent in various naturalists’ minds, when they speak of ‘species’… It all comes, I believe, from trying to define the indefinable.<br> -Charles Darwin, </em><a href="https://thinkersplayground.com/DarwinLetterHooker"><em>letter to Johnathan Hooker (December 24th, 1856)</em></a></blockquote><p>One of the ideas that still misdirects thinking in much of the world is <em>Creationism</em>: That life and all of its diversity came into being through a supernatural act of God, rather than through natural forces that are <em>gradually</em> developing over time.</p><p>Despite an ever-growing amount of evidence, creationists, under the guise of Intelligent Design, still try to oppose the natural origins of life with specious argumentation.</p><p>Although nearly every culture had creation myths, the Judaeo-Christian story that God created the Universe from nothing in seven days has thrived for almost three thousand years, and is undoubtably the most prevalent creation myth.</p><p>The concept of distinct groups of living things is tacitly baked into Judaeo-Christian belief. God created plants, animals, then humans on different days; The entire sub-order of snakes ( <em>Serpentes</em>) was cursed by God to crawl on its belly for deceiving Eve; Humans are considered to be distinctly different from animals, and were granted dominion over them; and didn’t God instruct Noah to take two of every <em>kind</em> on his Ark to preserve them (species is Latin for <em>kind</em>)?</p><p>Western philosophy has also played a key role in shaping how we perceive and organise various categories of living things, with Plato and Aristotle being the most notable contributors. Plato’s introduction of Universal Forms claimed that there are ideal <em>forms</em> of each thing, and that all of the <em>real</em> things we see on Earth are merely copies of them. In other words, somewhere outside of space and time, there exists a perfect blueprint for a dog, and your pet dog is an imperfect attempt to copy this <em>perfect dog</em>. This claim presupposes the existence of a particular thing called “dog” that has always existed and will always exist, cementing the species <em>dog</em> for eternity.</p><p>Aristotle was, in many ways, a more sensible philosopher than his predecessor Plato. He laid many of the foundations for modern science, including the idea that the Universe is made up of continual and causally connected series of events-everything is part of a single continuous process. However, in an effort to better understand the world, Aristotle named and categorised every living thing around him-which again, enforced the notion of separate and distinct categories of living things.</p><p>One doesn’t have to be an ancient philosopher to conclude that there are separate and distinct groups of living creatures. When we look around, we see that butterflies are different from bees, and dogs are different from dandelions. And for most of our history, people would only come into contact with a handful of the different forms of life that we now know of.</p><p>It should be fairly uncontroversial then to claim that our personal experience, our shared belief systems, and the roots of our cultural philosophies have all trained us to perceive the natural world as being separated into different and distinct groups of living things.</p><h3>On the Origin of Species</h3><p>Until about two hundred years ago, this way of thinking remained essentially unchallenged, but started to shift when a group of scientific thinkers known as “naturalists” began to recognise and speculate over the dynamics of evolution. Most notably, Charles Darwin’s <em>On the Origin of Species</em> presented an alternative view of life in which the diversity we see gradually emerged over an immense amount of time, and that all species probably shared a single common ancestor.</p><blockquote><em>I believe that animals have descended from at most only four or five progenitors, and plants from an equal or lesser number. Analogy would lead me one step further, namely to the belief that all animals and plants have descended from some one prototype… Therefore I should infer from analogy that probably all organic beings which have ever lived on this earth have descended from some one primordial form, into which life was first breathed.<br> -Charles Darwin, On the Origin of Species</em></blockquote><p>Around the same time, fossils of extinct species were beginning to be discovered and studied, and the science of palaeontology emerged. Our increasing understanding of the historical fossil record, and advancements in genetics and microbiology have all tested and validated the essence of Darwin’s ideas.</p><p>Still, <a href="https://news.gallup.com/poll/261680/americans-believe-creationism.aspx">recent Gallup Polls</a> show that over 40% of people in the USA believe that God created all life on Earth in its <em>present</em> form.</p><h3>“Intelligent Design”</h3><p>Religious opposition to the view that life’s origins are natural tends to centre around deliberately misleading claims. Proponents of <em>Intelligent Design</em> (creationism disguised as science) can often be caught misrepresenting scientific ideas in the hope that their audience is not able to recognise the flaws in their arguments.</p><p>One of the most common arguments against a natural origin of life, which <em>intelligent designists</em> like to make, is to claim that:</p><blockquote><em>Evolution can only occur within a species, and cannot create new species.</em></blockquote><p>In other words, dogs can be domesticated from wolves, but breeding dogs could never create something that is not a dog. New species, it is presumed, can only be created in the mind of the <em>Creator</em>.</p><p>The problem with this argument is that it’s based on a somewhat false premise, and one that was created by biologists.</p><h3>The problem with species</h3><p>Many biologists have struggled with providing clear rules for defining what makes a species distinct from another. When scrutinised, the rules for separating the different types of living things all seem to have problems with them. There isn’t even an agreement on the number of different <em>definitions</em> of species, let alone an agreement on a definition (some consider there to be seven main definitions, some twenty-six, some thirty, and so on). This is known as <em>The Species Problem</em>.</p><p>This is not a new problem. In <em>On the Origin of Species</em>, Darwin wrote:</p><blockquote><em>Nor shall I here discuss the various definitions which have been given of the term species. No one definition has as yet satisfied all naturalists; yet every naturalist knows vaguely what he means when he speaks of a species.</em></blockquote><p>To provide just a few examples of species definitions: The <em>Phenetic Species Concept</em> defines a species as a group of organisms that look significantly similar to each other. The <em>Recognition Species Concept</em> defines a species as a group of organisms that will recognise each other as mates. And the <em>Genealogical Species Concept</em> defines a species by how genetically similar organisms are to each other in comparison to other groups</p><p>But the definition that has become the most popular, is what is known as the <em>Biological Species Concept</em>, which describes a species as the largest group that can potentially breed with each other, and share in a gene pool.</p><blockquote><em>No term is more difficult to define than “species,” and on no point are zoologists more divided than as to what should be understood by this word.<br>-H. A. Nicholson</em></blockquote><p>Proponents of <em>Intelligent Design</em> use this <em>Biological Species Concept</em> to argue that natural selection can not explain the diversity of life, since a <em>species</em> can, by definition, only interbreed within its own gene pool, and will only produce offspring that are also of their same species. And so, organisms are perpetually trapped to share genetic information only within the confines of their species. In technical terms, they allow for <em>microevolution</em> (which is irrefutable), but not <em>macroevolution</em> (which may challenge deeply held religious beliefs).</p><p>But the slow and gradual process of small changes producing new species (which they deny) can be seen in action in a phenomenon known as <em>ring species</em> -populations that can share a gene pool with a more distant population through an intermediary population.</p><p>For example, suppose population <strong>A</strong> and population <strong>B</strong> are similar enough to interbreed. And population <strong>B</strong> and population <strong>C</strong> are similar enough to interbreed. But <strong>A</strong> and <strong>C</strong> are geographically separated, and <strong>A</strong> is too genetically distant from <strong>C</strong> to interbreed. In such cases, <strong>A</strong> and <strong>B</strong> could be considered the same species, and <strong>B</strong> and <strong>C</strong> the same species, but <strong>A</strong> and <strong>C</strong> are, by our definition, distinct species.</p><p>There are numerous examples of <a href="https://en.wikipedia.org/wiki/Ring_species#Examples"><em>ring species</em></a> in nature, sometimes with many compatible intermediary populations, but two or more that are incompatible at the extremes. This can be thought of as a geographical example of what has probably taken place over time, for hundreds of millions of years-just replace the <em>geographical</em> barrier separating the different populations for a <em>time</em> barrier instead.</p><p>It should be apparent then, that even our best working definition of species cannot accurately model the reality of the diversity found in nature.</p><p>The glaring problem with the creationist argument about species is that <em>species</em> don’t really exist-at least not in the way that they would like them to. Rather, they are a construct created by humans to help us make sense of the world.</p><blockquote><em>The concept of a species is a concession to our linguistic habits and neurological mechanisms.<br> -John B. S. Haldane</em></blockquote><p>Often in nature, things change gradually, slowly, and by degree. It’s only when enough changes in <em>degree</em> have taken place that we perceive them to be a difference in <em>kind</em>. <strong>In reality, the natural world is a near-infinite collection of <em>individual</em> entities that are more or less related to each other by degree</strong>.</p><p>To help us organise and make sense of the infinite diversity found in nature, we have to create some system that allows us to identify and describe the differences and similarities that exist at every level of life, and biologists have done well with this Herculean task. But these groupings will always be arbitrary, or suitable for certain purposes but not for others.</p><p>The claim that evolution can only occur within a species, and can never lead to new species, can be demonstrated false by recognising that there isn’t such a thing as species to begin with. The bold lines and distinct categories between different groups of species that creationists would like there to be, simply do not exist.</p><p><em>Originally published at </em><a href="https://thinkersplayground.com/trying-to-define-the-indefineable.html"><em>https://thinkersplayground.com</em></a><em> on June 13, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b0b2b59361c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grab data from your production database using ActionMailer]]></title>
            <link>https://medium.com/@gavinmorrice/grab-data-from-your-production-database-using-actionmailer-7aaf796a0875?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/7aaf796a0875</guid>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[rails]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Tue, 26 May 2020 23:00:00 GMT</pubDate>
            <atom:updated>2020-05-28T19:16:32.129Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a nasty and naughty hack, and one I’m abashedly apologetic for.</p><p>I was doing some work on a personal project recently, and wanted to pull some data out of the live database. I only needed a few columns from a couple of tables, so it didn’t seem to warrant a SQL dump.</p><p>I decided to format the records as JSON data, and copy/paste them from the console directly. But it turned out the length of the JSON data was longer than the backtrace in my terminal window, so I wasn’t able to see it all at once to select, copy, and paste.</p><p>Instead, I decided to save the data in a temporary file, and then email it to myself-using a temporary, memory-only mailer object.</p><p>Since this code was only held in memory during my rails console session, it disappeared as soon as I exited the session.</p><p>Here’s the code I used to do that:</p><pre>array <strong>=</strong> <em># ... get all of the data that I need from the DB</em><br><br><em># Create the Tempfile object</em><br>tempfile <strong>=</strong> Tempfile.<strong>new</strong>(&quot;my-data&quot;)<br><em># Write the data to the tempfile as JSON</em><br>tempfile.<strong>write</strong>(array.<strong>to_json</strong>)<br><br><em># Subclass ApplicationMailer in memory</em><br><strong>class</strong> <strong>DynamicMailer</strong> <strong>&lt;</strong> ApplicationMailer<br><br>  <em># Define a mailer method to send myself the file we just created</em><br>  <strong>def</strong> <strong>send_attachment</strong>(tempfile)<br>    attachments[&#39;attached.json&#39;] <strong>=</strong> File.<strong>read</strong>(tempfile.<strong>path</strong>)<br>    mail(to: &#39;me@handyrailstips.com&#39;,<br>         body: &quot;&quot;,<br>         subject: &quot;Incoming...&quot;)<br>  <strong>end</strong><br><br><strong>end</strong><br><em># Send the email</em><br>DynamicMailer.<strong>send_attachment</strong>(tempfile).<strong>deliver_now</strong><br><br><em># Clean up the Tempfile afterwards</em><br>tempfile.<strong>close</strong><br>tempfile.<strong>unlink</strong></pre><p>Hopefully the annotations in the code example make sense.</p><ol><li>I pulled the data that I wanted from the database in an Array of Hashes</li><li>I created a Tempfile to store the data in</li><li>I wrote the data to the Tempfile in the JSON format</li><li>I created a new Mailer class in memory with one method for sending attached files</li><li>I sent myself the Tempfile full of JSON</li><li>I cleaned up the Tempfile afterwards.</li></ol><p>I don’t advise that you ever use this horrible little hack for anything, on any project, ever.</p><p><em>Originally published at </em><a href="https://handyrailstips.com/hack/2020/05/27/grab-data-from-your-production-db-using-action-mailer.html"><em>https://handyrailstips.com</em></a><em> on May 26, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7aaf796a0875" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beware of descriptive words]]></title>
            <link>https://medium.com/thinkers-playground/beware-of-descriptive-words-e0b0629e4838?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0b0629e4838</guid>
            <category><![CDATA[misinformation]]></category>
            <category><![CDATA[protein]]></category>
            <category><![CDATA[plant-based]]></category>
            <category><![CDATA[veganism]]></category>
            <category><![CDATA[post-truth]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Thu, 13 Feb 2020 12:37:30 GMT</pubDate>
            <atom:updated>2020-02-14T10:43:36.601Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WgTYgA3J5osXWO9N.jpg" /></figure><p>Words that describe something, adjectives and adverbs, are often used to deliberately distort information and persuade you.</p><p>To give an example, those who are pushing the currently trendy <em>Plant-Based</em> diet (essentially veganism rebranded), will make claims like “green vegetables are rich in protein”, and then present the protein content data for a list of green vegetables.</p><p>While it’s probably correct to claim that, say, spinach has 2.9g of protein per 100g, calling this a “rich” source is completely misleading. For starters, compared to other foods such as meat, spinach has only about 1/10th the amount of protein per gram (chicken 31g/100g, turkey 27g/100g, beef 26g/100g, parmesan cheese 38g/100g, cheddar cheese 25g/100g, etc.). Even a Snickers bar has more protein than spinach, with 7g/100g.</p><p>Compared to most other foods, leafy greens are relatively low in protein, lower than white bread, and close to that of white rice.</p><p>Quantity of protein aside, <a href="https://www.mdpi.com/2072-6643/11/12/3016/htm">the <em>quality</em> of protein found in leaves is lower than that of meat sources</a> (as measured by DIAAS), so more vegetable protein has to be eaten to provide the same advantage relative to animal protein.</p><p>Finally, people seldom eat 100g of spinach in one sitting, whereas a meal consisting of meat may have much more than 100g of meat.</p><p>Words like “rich in” and “high in”, or “fast” and “rapidly” are just examples of the words constantly bounded around to skew the truth, and to persuade people to support a stance that the data itself does not justify.</p><p>When someone is trying to convince you that a given number is significant, a good tip is to ask yourself: compared to what? Always look for at least one other number for comparison.</p><p><em>Originally published at </em><a href="https://thinkersplayground.com/beware-of-descriptive-words-9b827b694d6b?post_slug=9b827b694d6b"><em>https://thinkersplayground.com</em></a><em> on February 13, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0b0629e4838" width="1" height="1" alt=""><hr><p><a href="https://medium.com/thinkers-playground/beware-of-descriptive-words-e0b0629e4838">Beware of descriptive words</a> was originally published in <a href="https://medium.com/thinkers-playground">Thinker’s Playground</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Book Review: The Pleasures of a Non-Conformist]]></title>
            <link>https://medium.com/thinkers-playground/book-review-the-pleasures-of-a-non-conformist-156ef9b07d1e?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/156ef9b07d1e</guid>
            <category><![CDATA[lin-yutang]]></category>
            <category><![CDATA[book-review]]></category>
            <category><![CDATA[nonconformist]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Wed, 05 Feb 2020 10:16:36 GMT</pubDate>
            <atom:updated>2020-02-05T10:16:36.635Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://amzn.to/33bodrB">The Pleasures of a Non-Conformist</a> is a collection of short essays by the Chinese thinker Lin Yutang.</p><p>I bought this book after reading and enjoying The Art of Living by Lin, and found it to be very much in the same style. Funny, interesting, extremely opinionated and candid.</p><p>Published in 1962, the book covers everything from the virtues of laziness, to women in hats; Chinese feminist satire, to buying a toothbrush.</p><p>Lin is the son of a Chinese christian minister, born in Victorian-era China, and highly educated in China, Europe, and the USA. Thus, he offers insights into the differences and the virtues from these disparate worlds. He’s half Taoist, half Confucianist; Half Christian, half Pagan; Half materialist, half humanist. A living contradiction.</p><p>His writing style is very plain and direct. Though clearly educated to the highest level, he doesn’t try to show this off with lofty language and long sentences. In the book, Lin writes about the importance of writing solely as a means of expression, and it would seem this book was written merely for him to express his own thoughts and feelings.</p><p>Modern readers may find some elements of his writing unacceptable. In a few instances, he uses terms for other cultures that are inappropriate by modern standards. These are not used as slurs to cause insult, but merely as labels when differentiating one group from another.</p><p>Being born in the 1800s, Lin also expresses very dated opinions about women. I believe it would be unfair, though, to mistake this as bigotry. Lin dedicates a whole chapter to praise Chinese feminism, and another to condemn the practice of foot binding. He was clearly a champion for improving the rights and living conditions of women.</p><p>As someone who witnessed the soul of his home country be destroyed by communism, Lin is not short of things to say about it. And I was delighted to find several tangents and comments throughout this book criticising that bloody ideology.</p><p>To give a full critique of Lin, I would have to complain that he often makes sweeping generalisations about cultures and other groups, and he never provides any evidence of any claim he makes (other than common sense or day to day experience). But in his defence, he is not writing a scientific text. He freely admits these are his own opinions, and the reader is welcome to take them or leave them.</p><p>The Pleasures of a Non-Conformist is a fun and educational read for anyone looking for an insight into the mind of someone born in a different world and era.</p><p>Originally published at <a href="http://thinkersplayground.com/book-review-the-pleasures-of-a-non-conformist-5a74e4fbcaf2">https://thinkersplayground.com</a> on November 23, 2019.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=156ef9b07d1e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/thinkers-playground/book-review-the-pleasures-of-a-non-conformist-156ef9b07d1e">Book Review: The Pleasures of a Non-Conformist</a> was originally published in <a href="https://medium.com/thinkers-playground">Thinker’s Playground</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding my bioenergetic self]]></title>
            <link>https://medium.com/thinkers-playground/understanding-my-bioenergetic-self-9223c00b9c84?source=rss-73d4a00c10dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9223c00b9c84</guid>
            <category><![CDATA[books]]></category>
            <category><![CDATA[bioenergetics]]></category>
            <category><![CDATA[metabolism]]></category>
            <category><![CDATA[ray-peat]]></category>
            <category><![CDATA[health]]></category>
            <dc:creator><![CDATA[Gavin Morrice]]></dc:creator>
            <pubDate>Wed, 05 Feb 2020 10:16:34 GMT</pubDate>
            <atom:updated>2020-02-05T10:16:34.444Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding my bioenergetic self</h3><p>A reading list for understanding the bioenergetic ideas of Dr. Raymond Peat and others</p><p>Over the past few years, an alternative movement has been gaining momentum in the world of health and nutrition. This movement has centred largely around the writings of Dr Raymond Peat PhD, a former professor of physiology and endocrinology.</p><p>I first stumbled across Ray Peat’s website in 2011 while researching caprylic acid (a fat found in coconut oil with some remarkable properties). But some of Peat’s main ideas ran contrary to other sources I was reading at the time (for example, <a href="http://raypeat.com/articles/articles/fishoil.shtml">that fish oil is not a sensible health supplement</a>) and it wasn’t enough to convince me away from my preexisting biases.</p><p>Over time, however, the name Ray Peat would crop up again and again in various places online, and I decided to go back and revisit Dr. Peat’s articles in 2014.Once I started reading Peat’s articles in greater depth, and began linking the common threads that are woven throughout them, my own understanding of human biology, health, and nutrition were changed substantively.</p><blockquote>“The key idea was that energy and structure are interdependent, at every level.”<br> — Ray Peat</blockquote><p>Peat presents a bioenergetic lens through which health, ageing, and disease can be understood as functions of energy metabolism — of the biological energy that’s available for use in our bodies. Energy isn’t just required for us to function, but for the organism as a whole to maintain its own integrity.</p><p>His articles describe how nutrition, hormones, and other environmental factors influence the availability of energy and impact health. These ideas are backed strongly by empirical research and validated by some of the most notable names in western science from the last century, including many Nobel Prize winners.</p><p>Despite this, many of the claims Peat makes run contrary to mainstream contemporary health advice and common cultural clichés.</p><p>This deviation from mainstream information, as well as the breadth and inherently technical nature of his articles, can mean Ray Peat’s work is often initially difficult to read.</p><p>I decided it would be helpful for me to dig deeper, and read the work of the authors Peat cites. This, I determined, would not only provide me with a deeper understanding of the information being discussed, but also help me to assess with greater confidence what are sometimes controversial ideas.</p><p>This educational journey has led me down a “rabbit hole”, creating a significant paradigm shift in how I understand and experience life.</p><p>Below is a list of books that I recommend to anyone who is interested in improving their own understanding of this broader way of thinking about life.</p><p>Note: This list is by no means exhaustive — these are only the books I’ve read so far — but I felt it would be inappropriate to recommend books that I haven’t read for myself. Readers will probably benefit from reading these books in the order presented, since they tend to progress in complexity.</p><p>If you would like to recommend a book be added, please feel free to contact me with your suggestion.</p><h3>How to Heal Your Metabolism</h3><p>Kate Deering</p><blockquote>“My recommendation is to eat to increase the metabolic rate (usually temperature and heart rate), rather than any particular foods.” <br> — Ray Peat</blockquote><p>Unlike many fad diet trends that focus on eliminating food groups or offer a strict protocol to follow, Ray Peat’s advice is simply to eat to increase the metabolic rate. Of course, this does include focussing more on certain food types and less on others — but you won’t find a specific “Peat diet” anywhere.</p><p>While there are many people who try to engineer their own diets based on Ray Peat’s recommendations (jokingly called “Peaters”), opinions are varied and can sometimes be conflicting. What’s more, scientific studies (which most Peaters try to look to) are often open to interpretation, and can be confusing to someone who doesn’t have a firm grounding in nutrition or biology.</p><p>To the newbie who is looking to start making positive health changes today, this book by fitness and nutrition coach Kate Deering is a great place to start. How to Heal Your Metabolism offers sensible, structured advice on the lifestyle, dietary, and training tips that are proven to increase metabolism and improve health.</p><p>How to Heal Your Metabolism on Amazon: <a href="http://amzn.to/2usBXkl">http://amzn.to/2usBXkl</a></p><h3>The Stress of Life</h3><p>Dr Hans Selye, MD</p><blockquote>“That mystique of diagnosing disease (specific, concrete, reified disease) was so strong that when Hans Selye noticed (in the 1930s) something that underlies all sickness (he first called it the “syndrome of being sick”), he was disregarded and disrespected, at least until his dangerous perceptions could be trimmed, distorted, and subsumed under some proper medical categories.”<br> — Ray Peat</blockquote><p>Central to understanding the impact of metabolism and energy on health, is to understand the effects of stress on the body. Hans Selye coined the term “stress” as we use it today. You could say he literally wrote the book on stress, and he called it The Stress of Life.</p><p>As a medical student, Selye observed that patients with a variety of conditions all displayed the same basic collection of symptoms. They just “looked sick”. While this observation seemed too obvious to be of any value to other medical academics, Selye made it his main area of focus.</p><p>Selye went on to define the General Adaptation Syndrome (GAS), which describes how an organism reacts to stress: Initial shock, resistance, and, if the stress persists, exhaustion. He explained how bad stress (or distress) quickly leads to decompensation, as the body is no longer able to meet the demands being placed on it.</p><p>In other words, Selye described how regular or chronic stress means using more energy than we can sustain; how this pushes the body beyond its ability to maintain or restore itself, and leads to ageing, disease and death.</p><p>Selye was awarded the highest decoration in Canada and was nominated for the Nobel Prize several times.</p><p>This book should be essential reading for anybody interested in understanding health, or who has a stressful lifestyle themselves.</p><p>The Stress of Life on Amazon: <a href="http://amzn.to/2uong1T">http://amzn.to/2uong1T</a></p><h3>The Authoritarians</h3><p>Prof. Bob Altemeyer</p><blockquote>“Altemeyer found that people who scored high on his scale of authoritarianism tended to have faulty reasoning, with compartmentalized thinking, making it possible to hold contradictory beliefs, and to be dogmatic, hypocritical, and hostile.Since he is looking at a spectrum, focusing on differences, I think he is likely to have underestimated the degree to which these traits exist in the mainstream, and in groups such as scientists, that have a professional commitment to clear reasoning and objectivity. With careful training, and in a culture that doesn’t value creative metaphorical thinking, authoritarianism might be a preferred trait.”<br> — Ray Peat</blockquote><p>In a perfect world, the dominant ideas in society would be determined by the best data available, and would be constantly updated as new information became available. However, that is usually not the case, and Peat’s articles often delve into the social and political factors that influence why good science is often ignored and bad science is heavily enforced.</p><p>This tendency to allow authority to govern the truth is probably best explained by Bob Altemeyer who spent most of his career researching authoritarianism. His book, the Authoritarians, is an amusing and eye-opening analysis of the authoritarian personality.</p><p>While this book does not cover health or nutrition directly, I found it useful for understanding many of the social factors that influence the decisions that do affect health, nutrition, and well-being at national and sometimes global scale.</p><p>The Authoritarians on Amazon: <a href="http://amzn.to/2usDS8c">http://amzn.to/2usDS8c</a></p><h3>Hypothyroidism: The unsuspected illness</h3><p>Dr Broda Barnes, MD, PhD</p><blockquote>“His work is clearly presented, not hard to understand, and it is scientifically so sound that no one challenges it, at least not on the scientific level. It is ignored, rejected by people who choose not to be bothered to read it. How many people have died from heart disease, since his work first became available? (And how many more from cancer, tuberculosis, and other diseases he showed occur mainly among hypothyroid people?)” — Ray Peat</blockquote><p>Those reading Ray Peat will very quickly realise that he has a lot to say about proper thyroid function and how it relates to health. This is probably one of the most controversial aspects of Ray Peat’s work.</p><p>Broda Barnes, M.D., studied hypothyroidism from the 1920s until his death in 1988.</p><p>In western medicine it’s well accepted that some individuals have problems with low thyroid function (hypothyroidism) — an insufficient amount of the active hormones that govern metabolic rate. But this is thought to affect only a small percentage of the population (about 5%), and is sufficiently treated with synthetic levothyroxine medication.</p><p>However, often with biological markers, the desired levels can be somewhat arbitrary, and the standards for what is considered a healthy range can vary from year to year, country to country, and even from doctor to doctor.</p><p>Barnes’s view was that hypothyroidism affects a very high portion of the American population, as much as 40%, and that hypothyroidism may lead directly to many degenerative or chronic health conditions (such as diabetes, hypertension, skin problems, etc). By treating what other doctors would regard as adequate thyroid function, Barnes was able to treat a variety of seemingly unrelated health issues.</p><p>In this book, Barnes describes his work in treating a variety of common diseases with thyroid medication, and his preference for using natural desiccated thyroid instead of the synthetic alternatives that are medically endorsed.</p><p>Despite the controversy, there are medical doctors today using Barnes’s approach today as and reporting the same positive results [<a href="http://www.telegraph.co.uk/news/health/alternative-medicine/10985192/Could-a-renegade-doctor-save-your-life.html">1</a>][<a href="http://www.altmedrev.com/publications/9/2/157.pdf">2</a>][<a href="http://journals.co.za/docserver/fulltext/m_samj/62/3/14786.pdf?expires=1500812163&amp;id=id&amp;accname=guest&amp;checksum=C7DABF09456F232A1E69EECB66A7160C">3</a>].</p><p>This book is very easy to follow, and helped me gain a deeper understanding of the role thyroid hormones can play in general health.</p><p>Hypothyroidism: The unsuspected illness on Amazon: <a href="http://amzn.to/2umy9Pw">http://amzn.to/2umy9Pw</a></p><h3>Nutrition for Women</h3><p>Dr Raymond Peat PhD</p><blockquote>“Americans have a lower metabolic rate than some other cultures, and the result is that obesity is a major problem in this county.”<br> — Ray Peat, Nutrition for Women</blockquote><p>This book, and others by Ray Peat, emerged as a collection of essays and lectures that he had prepared over the years. Because of this, his books often don’t follow a well structured narrative like many other books do that centre around developing a single proposition or idea.</p><p>Instead, Nutrition for Women is a collection of interesting notes that centre around the core theme of diet and lifestyle changes for better health. Since each section is very short, one can dip in and out, reading the book in many short intervals, or referring to the key topics as needed.</p><p>Despite the title, many of the ideas discussed here are, obviously, applicable to men too, although the book is composed with women in mind.</p><p>Nutrition for Women on Amazon: <a href="http://amzn.to/2um5Qk9">http://amzn.to/2um5Qk9</a></p><h3>Generative Energy: Restoring the wholeness of life</h3><p>Dr Raymond Peat PhD</p><blockquote>“The main features of ageing can be produced directly by administering excessive amounts of cortisol. These features include atrophy of skin, arteries, muscle, bone, immune system, and parts of the brain, loss of pigment (melanin), deposition of fat in certain areas, and slowed conduction velocity of nerves. The physiology of ageing (especially reproductive aging) overlaps the physiology of stress.” — Ray Peat, Generative Energy</blockquote><p>This book is probably the most “Peaty” of Ray Peat’s books, since it focusses on biological energy, and how this relates to hormones, health, ageing and stress — with some philosophy thrown in for good measure.</p><p>Areas covered include intelligence, brain development, hormones, stess, meaningful work, life as an emergent process, thyroid, metabolism, animal diets, coconut oil, etc.</p><p>Similar to Nutrition for Women, Generative Energy seems to be more of a collection of essays, rather than a book with a single focussed narrative. While I, personally, enjoy this meze of ideas, this style of writing be a little confusing for less technical readers.</p><p>Generative Energy on Amazon: <a href="http://amzn.to/2vvL2XJ">http://amzn.to/2vvL2XJ</a></p><p>Mind and Tissue: Russian Research Perspectives on the Human Brain</p><p>Dr Raymond Peat PhD</p><blockquote>“A common American idea of the brain is that it is ‘genetically’ ‘wired up,’ and some degree of rigidity is assumed even by the environmentalists, but beyond the rigid inherited system, there is assumed to be a passive ability to learn. In Russian brain physiology, increasing specialization of function in certain areas of the brain is considered to be a feature of evolution, but plasticity is believed to be the outstanding property of nervous tissue” — Ray Peat, Mind and Tissue</blockquote><p>Mind and Tissue is my favourite book by Ray Peat, and one of my all-time favourite books, mostly because it bridges my interests in biology, philosophy, and psychophysiology.</p><p>In the years since the Enlightenment, Western-European and Eastern-European philosophies diverged substantially. Emerging from the ideas of Darwin, and fuelled by inheritance of Roman dogma and Protestant individualism, Western-European thinking moved closer towards rigid genetic determinism, randomness, survival of the fittest, and individual competitiveness as its underlying ideologies.</p><p>In contrast, Eastern-European thinking, influenced by greek philosophers like Aristotle, and the neighbouring asian countries, led Eastern-European thinking closer towards ideas of holism (wholeness), environment, evolution as an internal drive, and collectivism.</p><p>This ideological divide, as well as the Cold War, which limited the sharing of scientific information between Russia and the rest of Europe and the United States, meant that there was a significant amount of Russian discoveries that were completely unknown outside of Russia.</p><p>As Peter Marin writes in the foreword: “The Russians peer into a mirror altogether different from ours, one obscured or distorted by different prejudices, and that often allows them to confront or admit what we miss. They are looking for different things, working out a different version of human nature and human possibility, a different kind of faith in reason, will and desire (forces we tend to ignore in our own psychology), and for that reason the complexity of the problems they have chosen to examine and the elegance of their tentative solutions often puts to shame our own approaches.”</p><p>Since Peat’s first degree was in linguistics — he speaks Russian. Mind and Tissue offers a window into the world of Russian psychology and brain physiology and compares it to the thinking of American science. In doing so, he also provides a justification of humanitarian sciences as hard sciences, since many of the Russian discoveries provide scientific bases for understanding phenomena such as creativity.</p><p>This book is less concerned with healthy eating, but provides fascinating and challenging alternative ideas on how we, as living organisms, are closely interrelated with each other, our own history, our environment, and our own view of ourselves.</p><p>Mind and Tissue on Amazon: <a href="http://amzn.to/2vvnUIH">http://amzn.to/2vvnUIH</a></p><h3>The Body Electric</h3><p>Prof Robert O Becker</p><blockquote>“Rather than studying the regeneration of organs and tissues, and recognizing its obvious importance in healing as well as in understanding the nature of life, much of the last century was devoted to the defamation of the researchers who were making real process in the field. Despite many demonstrations that regeneration can occur in adult mammals, students were taught that it happens only in lower vertebrates. I think it’s important to look closely at the ideology responsible for this great loss.”<br> — Ray Peat</blockquote><p>I thoroughly enjoyed this book! Robert O Becker was an orthopaedic surgeon and researcher in electrophysiology — he studied the electrical properties of the body.</p><p>In this biographical book he describes how his work progressed over the years, and how he discovered the electric potentials of the body.</p><p>Many people know that certain animals, salamanders especially, have the remarkable capacity to fully regenerate missing limbs — even a full leg. Becker’s early work began by exploring the question of why? … and why can’t other animals do the same.</p><p>In telling the story, Becker explains how he designed his experiments on regeneration, and the interesting electric properties of the body he discovered through his experimentation.</p><p>As his work progressed, Becker began to look into the biological effects of environmental EMF (electromagnetic fields), and the impacts that they have on the human body.</p><p>This book is essential reading for anybody who wants to better understand the interplay between metabolism, nerve tissue, and electrical fields in health, growth and development.</p><p>The Body Electric on Amazon: <a href="http://amzn.to/2vw0jYy">http://amzn.to/2vw0jYy</a></p><h3>Cells, Gels, and the Engines of Life</h3><p>Prof Gerald Pollock</p><blockquote>“Gilbert Ling’s revolution in cell biology remains outside the canon, despite the profound influence of MRI, which grew directly out of his view of the cell, because his work provided conclusive evidence that cells are not regulated by ‘semipermeable membranes and membrane pumps.’ Every field of science is ruled by a doctrinaire establishment.”<br> — Ray Peat</blockquote><p>This is, without a doubt, the most interesting biological textbook I’ve ever read. Had this been on the reading list when I started biology at University, I probably wouldn’t have dropped out.</p><p>Gerald Pollack is a professor of bioengineering at Washington University who is best known for his work with water that has earned him the nickname The Water Wizard. Building on the work of scientists like Gilbert Ling, Pollack has shown that water actually has 4 phases as opposed to the three (vapour, liquid, solid) that we are taught about in school.</p><p>This book describes how this 4th phase of water is probably the driver behind many functions within the cells of our bodies — in many cases, offering more elegant alternative theories to the current prevailing theories.</p><p>This is not a book for newbies. Many of the topics discussed here are at least high-school level biology, and require a bit of willingness on the part of the reader to fill in the knowledge gaps where necessary. However, the information presented is very clear, the layout is beautiful, and the theories developed in this book are so elegant that nerds will find themselves grinning uncontrollably while reading them.</p><p>Cells, Gels, and the Engines of Life on Amazon: <a href="http://amzn.to/2gSCXJ8">http://amzn.to/2gSCXJ8</a></p><p>— -</p><p>I’ll update this list as I finish any new books that are relevant.</p><p>Originally published at <a href="http://thinkersplayground.com/understanding-my-bioenergetic-self-39afdd06c987">https://thinkersplayground.com</a> on July 22, 2017.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9223c00b9c84" width="1" height="1" alt=""><hr><p><a href="https://medium.com/thinkers-playground/understanding-my-bioenergetic-self-9223c00b9c84">Understanding my bioenergetic self</a> was originally published in <a href="https://medium.com/thinkers-playground">Thinker’s Playground</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>