<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tomer Brisker on Medium]]></title>
        <description><![CDATA[Stories by Tomer Brisker on Medium]]></description>
        <link>https://medium.com/@tbrisker?source=rss-1417a75ba118------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RJri5CwHx7rz90nuyrpB-g.jpeg</url>
            <title>Stories by Tomer Brisker on Medium</title>
            <link>https://medium.com/@tbrisker?source=rss-1417a75ba118------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:19:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tbrisker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When an Array is not an Array]]></title>
            <link>https://medium.com/@tbrisker/when-an-array-is-not-an-array-f0e7499d5817?source=rss-1417a75ba118------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0e7499d5817</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[rails]]></category>
            <dc:creator><![CDATA[Tomer Brisker]]></dc:creator>
            <pubDate>Mon, 27 Aug 2018 10:32:44 GMT</pubDate>
            <atom:updated>2018-08-27T10:32:44.876Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<blockquote>The story of a simple array sort gone wrong</blockquote><p>Part of the release process of <a href="https://www.theforeman.org/">the Foreman</a> project includes extracting all strings from the source code for translation, and pulling updated translations from an online service we use called <a href="https://www.transifex.com/">Transifex</a>. As part of the release process for version 1.19.0, I ran the rake task which handles this step.</p><p>Unfortunately, it failed with a strange error message:</p><pre>ArgumentError: comparison of Array with Array failed<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/po.rb:270:in `sort_by’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/po.rb:270:in `sort_by_msgid’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/po.rb:227:in `sort’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/po.rb:208:in `to_s’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/tools/msgcat.rb:57:in `run’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/tools/msgcat.rb:32:in `run’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext-3.2.9/lib/gettext/tools/task.rb:391:in `block in define_po_file_task’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/gettext_i18n_rails-1.8.0/lib/gettext_i18n_rails/tasks.rb:65:in  `block (2 levels) in &lt;top (required)&gt;’<br>/home/tbrisker/.rvm/gems/ruby-2.5.1@foreman/gems/rake-12.3.1/exe/rake:27:in `&lt;top (required)&gt;’</pre><p>Well, let’s look at the failing function’s code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b6fcaad774529d26ebf35b7bc020bde/href">https://medium.com/media/2b6fcaad774529d26ebf35b7bc020bde/href</a></iframe><p>Hmm… That’s quite odd, looks like a fairly simple sorting function. Looking at the gem’s git <a href="https://github.com/ruby-gettext/gettext/blame/master/lib/gettext/po.rb#L268">history</a> indicates that this function hasn’t changed in ages — so why is it failing now? The comment seems pretty cryptic, and without digging too much into the gem’s internals it would be difficult to understand what it means. Time to pull out the ol’ debugger and stick a binding.pry breakpoint inside the function to try and understand what is going on here.</p><p>This is what one of the msgid_entry objects looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c9e70fbd2be24bc41e3069f0ca31d05/href">https://medium.com/media/7c9e70fbd2be24bc41e3069f0ca31d05/href</a></iframe><p>It is an array that is composed of two elements — the first, an array that has two element itself, and the second is a GetText::POEntry object, that is some sort of internal gettext object representing the translation. Now the comment in the code makes a bit more sense — the array contains a message context (in this case, nil) and a message ID, in this case, the string &quot;Failed to fetch:”. The sorting function sorts the entries by the values of this array ( msgid_entry[0]). So for some reason, one (or more) of the entries contains a problematic value in this array.</p><p>Stepping a few iterations in the loop didn’t cause the exception to be raised, and considering there are over 3,000 entries in the list (presumably, one for every string that needs translation), manually finding the problematic entry can be painstaking.</p><p>After a while of playing around with conditional breakpoints, the culprit was found:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abc452a67bd43a78356b01cac95c7bca/href">https://medium.com/media/abc452a67bd43a78356b01cac95c7bca/href</a></iframe><p>This was the only entry that had something other than nil for the first element of the first array — the message context. To make my work easier, the second element included the reference to the exact line of code where the string came from.</p><p>Looking at the code, the <a href="https://github.com/theforeman/foreman/pull/5860/files#diff-e44063a1ea27ac907ae659670db4eb54R86">fix</a> was clear — someone had a typo and used the incorrect method for marking the string for translation. So instead of using the method needed to mark the string as one that requires both singular and plural translation, the regular translation method was called, which apparently accepts an additional parameter for the context.</p><p>With the issue found and fixed, one question remained — why did this fail with the cryptic “comparison of Array with Array failed” error message?</p><p>To understand this, we need to dive a bit into how Ruby’s sort_by function works. Simply put, it accepts a block that gets as its argument the elements in the sorted object, and sorts them by the return value of the block. In our case, the entries are sorted by the value of the first element in each entry — the array containing the message context and ID.</p><p>But how does Ruby sort arrays? Turns out it sorts them by comparing one element at a time. To compare the elements, it calls the function &lt;=&gt;, which returns -1, 0, or 1 that indicate if the first element is smaller, equal or larger than the other. However, it can also return nil if the two elements can’t be compared, which is exactly what is happening here.</p><p>To illustrate, let’s look at what happens when we compare two strings:</p><pre>[1] pry(main)&gt; &#39;a&#39; &lt;=&gt; &#39;foo&#39;<br>=&gt; -1</pre><p>Since a is lexically smaller than foo, the function returns -1. But what happens when we try to compare a string with nil?</p><pre>[2] pry(main)&gt; nil &lt;=&gt; &#39;foo&#39;<br>=&gt; nil</pre><p>As expected, since Ruby doesn’t know how to compare the two different object types, it returns nil. But what does that have to do with our error? Well, if you recall, Ruby <a href="https://docs.ruby-lang.org/en/2.5.0/Array.html#method-i-3C-3D-3E">compares arrays</a> by iterating over the array elements. Now lets see what happens if it tries to compare arrays containing elements that can’t be compared:</p><pre>[3] pry(main)&gt; [[nil, &#39;foo&#39;], [&#39;bar&#39;, &#39;baz&#39;]].sort<br>ArgumentError: comparison of Array with Array failed<br>from (pry):4:in `sort&#39;</pre><p>One of the arrays being sorted by has a string for its first element, while the other has nil. This means Ruby doesn’t know how to sort these arrays, leading to an error. Unsurprisingly, since this is exactly the case as we were hitting with our translations, with one array containing a string where all others had nil, the same error was raised as when we tried the sorting in this example.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0e7499d5817" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA["It works on my machine"]]></title>
            <link>https://medium.com/@tbrisker/it-works-on-my-machine-b712fd532670?source=rss-1417a75ba118------2</link>
            <guid isPermaLink="false">https://medium.com/p/b712fd532670</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Tomer Brisker]]></dc:creator>
            <pubDate>Sun, 08 Jul 2018 12:45:45 GMT</pubDate>
            <atom:updated>2018-07-25T08:50:45.366Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>“It works on my machine”</h3><blockquote>When Webpack is broken, but only in the packaged build — or how I created my first Webpack plugin</blockquote><p>First, this story requires some background and a basic understanding of the architecture of the main project I’m working on, <a href="https://theforeman.org/">Foreman</a>. Foreman is a Ruby on Rails application, used for infrastructure management. It has many plugins that add various functionality, which are written as Rails Engines. Since this application is used to manage critical infrastructure, it is installed on-premise and is shipped as packages — both .deb and .rpm. Many of the various plugins are also packaged and shipped as .rpm and .deb files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*3Q7A1CdCA2Pcb7zhSP3Pfg.png" /><figcaption>The Foreman Project Logo</figcaption></figure><p>About 2 years ago we started modernizing our application’s front-end. To achieve this, it meant integrating Webpack and npm into our assets build process. This allowed us to use cool new technologies such as React and ES6 in our client side code, as well as start to solve long standing issues we had with our legacy JS code — such as lack of linting and unit-tests. The initial step of creating a Webpack build process took a while. One of the reasons is that RPM package builders must run in a disconnected environment, so we had to figure a way of providing all the node modules to the builder as RPMs at build time, but that is a story for an entire post on its own. Once that was solved, we started gradually migrating or replacing the legacy code-base with new, Webpack-managed code.</p><p>A major challenge we had recently was providing plugins with a way to leverage the existing npm and Webpack stack in Foreman core for writing their own client-side code. We didn’t want to force every plugin to create its own build pipeline, and we didn’t want to have multiple copies of every library included in each plugin’s JS bundle. This was solved with some clever engineering work by some of my colleagues. The solution involved compiling the Webpack assets for both plugins and the core application from the main application’s code base, while providing plugins with a vendor.js file that contains multiple shared libraries — such as React, Redux, Lodash and more. This worked perfectly fine during development, and even when compiling assets for production locally.</p><p>However, when we tried installing the nightly packages with plugins, something strange started happening. The core pages which included Webpack-managed assets worked fine, but the plugin pages requiring use of the plugin’s Webpack assets didn’t load. Looking in the console, it didn’t look promising:</p><pre>TypeError: e[t] is undefined</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*RHn006Wsn8p6W1oVCCf73A.jpeg" /><figcaption>e[t]</figcaption></figure><p>Now, part of the Webpack build process includes minifying the js code, making it smaller to download but much more difficult to debug. Luckily the browser developer tools knows how to prettify the code, so it is at least semi-readable. This pointed to the error being in the following function, on line 9:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06b43a40ee37aef944eaf123ed474508/href">https://medium.com/media/06b43a40ee37aef944eaf123ed474508/href</a></iframe><p>Well, that’s not much better… but at least .exports hints at what this does. Comparing to the unminified and looking at the code around this function led me to understand that this is part of the Webpack code that handles module exports and makes them available to other modules, such as the plugin. But what is e[t]? And why is it failing here?</p><p>Since I know the failure occurs when e[t]===undefined, I set a conditional breakpoint on this line in my browser’s developer tools and reloaded the page. Now I could finally see what is happening here. e seems to be an object mapping hashes to functions:</p><pre>» e<br>{0: ƒ, 00bb67fca9d5ae35c4aa: ƒ, 0139edb80f5e8e5773b7: ƒ, 01637e90596e3c444fa8: ƒ, 01d23de99989e6fe314f: ƒ, 01f59de879b1bcfbb8e7: ƒ, …}</pre><p>And t is a hash — 2278734bfcaa57409dda in this case. But for some reason, this specific hash isn’t included in the e object. Digging inside the plugin’s minified code, I searched for the hash to try and figure out what it meant. This allowed me to make a guess at which module this hash should map to, but surprisingly, it was a module that should have been included inside vendor.js! So what happened here? Where did this hash come from? why was this working fine in the development environment? Looking in our Webpack config file, I found the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5f765814fe8388a0fe442b632344923/href">https://medium.com/media/c5f765814fe8388a0fe442b632344923/href</a></iframe><p>Git history indicated that HashedModuleIdsPlugin was introduced <a href="https://github.com/theforeman/foreman/pull/5579">recently</a>, in an attempt to allow plugins to use modules by creating a stable identifier for each module. Apparently, the default Webpack configuration generates a sequential ID for each module, making it difficult to use from inside plugins as the module IDs would change every time a module was added or removed. This explained why the hashes were only present in production, but not why this worked when compiling the Webpack assets for production locally and failed when using the bundles produced by our RPM builders. I needed to understand how this hash was generated.</p><p>Reading the Webpack documentation for this plugin, I found out that :</p><blockquote>This plugin will cause hashes to be <strong>based on the relative path of the module</strong>, generating a four character string as the module id.</blockquote><p>This led me to suspect that for some reason or other, when generating the Webpack bundles on the builders, the relative paths to the node modules were different for the plugins to those that were used when building Foreman core. However, from hashes this theory was very difficult to confirm. I needed some way to find the relative path that Webpack was using to generate the hash.</p><p>After some research, I found there is a plugin that does just that — NamedModulesPlugin. For some reason, the documentation (that has been <a href="https://github.com/webpack/webpack.js.org/pull/2287">deleted</a> recently) claimed it should be used only in development and that it will show the relative path to the modules only when Hot Module Replacement is active. Looking at the code and trying it out, though, seemed to indicate that this is exactly what I needed to verify my hypothesis. When I replaced HashedModuleIdsPlugin with NamedModulesPlugin, the hashes were replaced with relative paths, both in the object mapping module identifiers to functions and in the code calling them. Locally, this continued working just as before with plugins, but the real test was when running with plugins built as RPMs.</p><p>Once the change to NamedModulesPlugin was <a href="https://github.com/theforeman/foreman/pull/5723">merged</a> into the project, it was time to rebuild the packages with the change and see if and what failed now. It took a couple of days to get the packages built again, since in the meantime some unrelated changes were merged that led to broken builds. But once they finally were, the moment of truth arrived — was my hypothesis correct?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fVK7xWRGTW8dlrv9" /><figcaption>“A bright neon on a brick wall in a store” by <a href="https://unsplash.com/@austinchan?utm_source=medium&amp;utm_medium=referral">Austin Chan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Unsurprisingly, the same e[t] is undefined error showed up again. Only difference is that this time t should point to the actual module that was failing to load, so we can finally figure out why this was failing. Returning the conditional breakpoint to the vendor.js code allowed me to find what it was:</p><pre>» t<br>&quot;../../../../../../../usr/lib/node_modules/react/index.js&quot;</pre><p>React? That’s odd! We use React in multiple pages in the core application as well, and they work just fine, so it must be available in vendor.js! Quick CTRL+f for react in vendor.js led to identifying the culprit. This is the identifier that was present in vendor.js for the React module:</p><pre>&quot;../../../../usr/lib/node_modules/react/index.js&quot;</pre><p>Bingo! Notice the different number of leading ../’s compared to the identifier the plugin was looking for. This indicated that indeed, during the build process, Webpack was being run from different paths when building the plugin, compared to when building Foreman core.</p><p>Now that we finally got to the root of the issue, it was time to fix it. There were two different ways we could go about this:</p><ol><li>Make sure that the build always occurs from the same root directory, leading to consistent module IDs.</li><li>Create a unique identifier for the modules that doesn’t care about where the build was started from, only about the module itself.</li></ol><p>Option 1 could possibly work, but it required some deeper digging in our fairly complex build pipeline, followed by changes that may lead to other unexpected issues further down the process. It would also tie the whole build process to the Webpack configuration and reduce its flexibility in the future.</p><p>Option 2 required finding a way to make sure that module IDs are consistent across builds, regardless of what folder the build process was started from. One idea that was brought up was to use a hash of the module’s entire source code, but that would mean that every time any module is changed, all plugins would have to be rebuilt — since the hash would change as well. While for some changes it makes sense to also have plugins updated, usually module APIs don’t change much between versions, so that a minor change shouldn’t matter to the consuming plugins.</p><p>The solution that was finally decided on was to strip everything up to node_modules from the path for the module — so that React, for example, would get an ID of &quot;node_modules/react/index.js&quot; instead of something like &quot;../../../../usr/lib/node_modules/react/index.js&quot;. We don’t really care where the node_modules folder is, and it shouldn’t matter to plugins either. This ID will remain consistent across builds, and for as long as the module doesn’t change its internal structure. But how can we do this?</p><p>To figure this out, I looked at the <a href="https://github.com/webpack/webpack/blob/webpack-3/lib/NamedModulesPlugin.js">source code</a> of the NamedModulesPlugin that is shipped with Webpack 3 (the version we are currently using). The code was surprisingly short— a loop over all modules, setting the ID using libIdent(). Some searching inside webpack code allowed me to understand the this function call will return the path to the module, without trying to dig to much into its internals.</p><p>Since the plugin doesn’t provide any means of modifying the ID, what was needed now was to make a new Webpack plugin that strips everything up to node_modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5b438a12b7de8c5f742d33d96a16098/href">https://medium.com/media/c5b438a12b7de8c5f742d33d96a16098/href</a></iframe><p>This code is almost the same as the original NamedModulesPlugin. The only change I added are lines 14–16, which strip out the extra part of the path from the module ID where needed.</p><p>As I assume others may run into this or similar issues in the future, I have extracted this plugin from our project to its own <a href="https://github.com/tbrisker/simple-named-modules-plugin-webpack">repository</a>, and released it as a <a href="https://www.npmjs.com/package/simple-named-modules-plugin-webpack">node module</a>.</p><p>While this story was mostly written in the first person, as it depicts my perspective of the events, it is important to note that this solution, like everything I work on, is the result of a team effort.</p><p>Special thanks go to all those who helped in the process of fixing this issue, in no particular order: Daniel Lobato Garcia, Ewoud Kohl van Wijngaarden, Eric Helms, Avi Sharvit, Ohad Levy, Walden Raines, and anyone else who lent a hand and I forgot to mention.</p><p>This solution would also not have been possible if Webpack hadn’t been an open-source project — as I would not have been able to dig into its internals, extract and modify code from it otherwise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b712fd532670" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bug report to pull request in under 1 hour]]></title>
            <link>https://medium.com/@tbrisker/bug-report-to-pull-request-in-under-1-hour-f845de6f554c?source=rss-1417a75ba118------2</link>
            <guid isPermaLink="false">https://medium.com/p/f845de6f554c</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Tomer Brisker]]></dc:creator>
            <pubDate>Mon, 12 Feb 2018 21:53:14 GMT</pubDate>
            <atom:updated>2018-02-12T21:53:14.145Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>A user came in to our IRC channel today complaining about an issue with searching:</p><pre>[16:31] &lt;user&gt; Why is it that when I search parent_hostgroup = “somerandom” in the “Hosts” searchbox, where “somerandom” doesn’t match anything, it gives me a list of all hosts instead of none? I have some issues with this behavior when performing searches using the API<br>[16:40] &lt;tbrisker&gt; user: sounds like a bug, can you open an issue on projects.theforeman.org?<br>[16:49] &lt;user&gt; tbrisker, <a href="https://projects.theforeman.org/issues/22556">https://projects.theforeman.org/issues/22556</a></pre><p>Luckily, this one was a quick fix. But let’s dive in together and understand why it failed in the first place.</p><p>We use a gem called <a href="https://github.com/wvanbergen/scoped_search/">Scoped Search</a> to handle searches in our application. It allows us to provide our users with the ability to search the various resources in the application using a powerful query language and auto-completer. While for most cases scoped search Just Works™, sometimes more complex logic is needed for certain queries. This is possible by telling scoped search to use an external method to generate the query by passing an :ext_method parameter to the search definition.</p><p>The case of parent_hostgroup is one such case. A host in Foreman can belong to a host group, which in turn might be a child of one or more host groups. To allow searching by parent host group, i.e. any hosts that belong to a certain host group or any of its children, an external method is needed:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7a5c38012eaf7716f9a6d3a35ff4518/href">https://medium.com/media/d7a5c38012eaf7716f9a6d3a35ff4518/href</a></iframe><p>Can you spot the bug?</p><p>It’s on line 5. If no host groups match the search on line 4, hostgroup will be nil. That means that hostgroup.subtree_ids should raise an exception:</p><pre>NoMethodError: undefined method `subtree_ids&#39; for nil:NilClass</pre><p>That will be easy to fix, we just need to make sure that the hostgroup is present before collecting its sub-tree ids:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a9d2a9d817a9b2d6b2fa27347b4c6df/href">https://medium.com/media/9a9d2a9d817a9b2d6b2fa27347b4c6df/href</a></iframe><p>And sure enough, less then an hour after the initial report, I had a pull request ready, which also included a unit test to ensure this function won’t break again in the future:</p><pre>[17:23] &lt;tbrisker&gt; user: <a href="https://github.com/theforeman/foreman/pull/5249">https://github.com/theforeman/foreman/pull/5249</a><br>[17:24] &lt;user&gt; tbrisker, thanks! :)</pre><p>34 minutes of work and one happy user!</p><p>Since I try to follow the <a href="http://programmer.97things.oreilly.com/wiki/index.php/The_Boy_Scout_Rule">Boy Scout Rule</a>, I also fixed a minor issue on line 9: the idea here is to return a condition that is always evaluated to false, thereby returning no results for the search. However, when the DB sees a condition like host.id &lt; 0, it checks if any of the IDs are in fact smaller then 0. When you have tens of thousands of objects, this condition combined with other conditions (like default scope, default order, STI type etc.), even when executed against the index, can still take some time:</p><pre>foreman=# explain analyze SELECT “hosts”.* FROM “hosts” WHERE “hosts”.”type” IN (‘Host::Managed’) AND ((hosts.id &lt; 0)) ORDER BY “hosts”.”name” ASC;<br>                      QUERY PLAN <br>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — <br> Sort (cost=62.19..62.20 rows=4 width=6134) (actual time=17.702..17.702 rows=0 loops=1)<br> Sort Key: name<br> Sort Method: quicksort Memory: 25kB<br> -&gt; Bitmap Heap Scan on hosts (cost=16.50..62.15 rows=4 width=6134) (actual time=17.692..17.692 rows=0 loops=1)<br> Recheck Cond: ((type)::text = ‘Host::Managed’::text)<br> Filter: (id &lt; 0)<br> Rows Removed by Filter: 43835<br> Heap Blocks: exact=1429<br> -&gt; Bitmap Index Scan on index_hosts_on_type (cost=0.00..16.50 rows=12 width=0) (actual time=9.268..9.268 rows=43835 loops=1)<br> Index Cond: ((type)::text = ‘Host::Managed’::text)<br> Planning time: 0.359 ms<br> Execution time: 17.808 ms<br>(12 rows)</pre><p>Ok, that’s not too bad 😄</p><p>But on the other hand, when the RDBMS sees a condition like 1 &lt; 0 it immediately evaluates it to false and returns much faster (by about 3 orders of magnitude):</p><pre>foreman=# explain analyze SELECT &quot;hosts&quot;.* FROM &quot;hosts&quot; WHERE &quot;hosts&quot;.&quot;type&quot; IN (&#39;Host::Managed&#39;) AND ((1 &lt; 0)) ORDER BY &quot;hosts&quot;.&quot;name&quot; ASC;<br>                      QUERY PLAN                                          <br>------------------------------------------------------------------<br> Sort  (cost=0.01..0.02 rows=0 width=6134) (actual time=0.011..0.011 rows=0 loops=1)<br>   Sort Key: name<br>   Sort Method: quicksort  Memory: 25kB<br>   -&gt;  Result  (cost=0.00..0.00 rows=0 width=6134) (actual time=0.001..0.001 rows=0 loops=1)<br>         One-Time Filter: false<br> Planning time: 0.241 ms<br> Execution time: 0.077 ms<br>(7 rows)</pre><p>Not a huge gain, but doesn’t hurt either. Perhaps learning this trick may come in handy for other cases, where it may lead to a more significant gain.</p><p>But wait, the user wasn’t actually complaining about an error. They complained that when the parent host group doesn’t exist, all hosts are returned instead of none of them. Why didn’t they get the NoMethodError? That’s peculiar!</p><p>Good thing the Scoped Search gem is Open Source, I can take a look inside and try to figure out what happened.</p><p>Here is the code that handles the external method definitions inside Scoped Search:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75e8485a554027fe50932dc1d0224d61/href">https://medium.com/media/75e8485a554027fe50932dc1d0224d61/href</a></iframe><p>Line 3 is the line that actually calls the external method when needed.</p><p>But what is that rescue {} doing there? That means that any error that occurs in the external method is swallowed silently, and the search will execute as if there was no condition at all! This is effectively suppressing any errors inside the external method, which is considered <a href="https://github.com/bbatsov/ruby-style-guide#dont-hide-exceptions">bad practice</a> in Ruby and in programming in general. There are two reasons this is considered bad practice:</p><ol><li>A user hitting an error (for example, by sending an invalid search query) will receive no indication that something went wrong. They may incorrectly assume the results they received were correct, which could lead them to take further actions that are mistaken. In our example, say a user wants to create a script that deletes all hosts in a certain host group and its children. If they mistype the host group name, instead of receiving an error or no results, they may end up deleting all of their hosts!</li><li>The developer creating the method who’s exceptions are being supressed may think it is functioning correctly, and receive no indication that there is any issue in it. In this case, looking at the git history, this bug has existed for 3.5 years without being noticed!</li></ol><p>So, thanks to the power of Open Source, I fixed the issue in Scoped Search and sent the maintainers a <a href="https://github.com/wvanbergen/scoped_search/pull/173">Pull Request</a> as well:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/724dc4f0ee1cc2c6bb7b8793c3f6935d/href">https://medium.com/media/724dc4f0ee1cc2c6bb7b8793c3f6935d/href</a></iframe><p>Now let’s just hope they like it enough to merge it into their project! 😃</p><p>Thank you for reading, I would be happy to hear feedback in the comments below or <a href="https://twitter.com/tbrisker_pro">on twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f845de6f554c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strong Parameters in Rails —down the rabbit hole]]></title>
            <link>https://medium.com/@tbrisker/strong-parameters-in-rails-down-the-rabbit-hole-2426d331625?source=rss-1417a75ba118------2</link>
            <guid isPermaLink="false">https://medium.com/p/2426d331625</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Tomer Brisker]]></dc:creator>
            <pubDate>Wed, 17 Jan 2018 22:40:45 GMT</pubDate>
            <atom:updated>2018-01-17T22:40:45.866Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>It all started with a bug report (doesn’t it always?). A user was complaining that when they passed an incorrect argument to our API by mistake, they didn’t get any error message. Sounds like a quick fix, right?</p><p>Wrong.</p><p>Some background first, simplified greatly for the purpose of this post. Our application, like many others, has users. A user can belong to multiple organizations. Our REST API allows updating a user by PUTting a JSON object to the relevant endpoint, for example:</p><pre>{ &quot;user&quot;: { &quot;organization_ids&quot;: [1,2,3]}}</pre><p>But our user had a typo. This is the JSON they sent:</p><pre>{ &quot;user&quot;: { &quot;organization_ids&quot;: 1}}</pre><p>It seemed to work fine — they got no error message. Perhaps it worked even too well: When they tried to pass a non-existing organization ID, it still returned a successful response! In fact, whenever they forgot to wrap the ID in an array, that parameter was simply ignored. No error response, no mention in the logs of something being wrong, nothing.</p><p>After I reproduced the bug in my development environment, I started digging in. With some help from pry inside the controller, I saw that while the organization_ids parameter was present in the request, it wasn’t there in the user_params object that contains the allowed parameters for assignment. This pointed me in the direction of Strong Parameters.</p><p>Strong Parameters is a feature of Rails that prevents assigning request parameters to objects unless they have been explicitly permitted. It has its own DSL (Domain Specific Language, or in other words, a predefined syntax it understands), that allows you to indicate what parameters should be allowed. It also lets you indicate if each parameter should be a hash, array or scalar (i.e. integer, string, etc.), as well as some other functionality that is not relevant for this post.</p><p>Here is an excerpt from our definition of permitted parameters in the users controller:</p><p>params.permit :name, :login, :mail, :organization_ids =&gt; [], ...</p><p>Ok, so Strong Parameters expects an array for the :organization_ids parameter, but it gets an integer instead. So why am I not seeing any indication that it failed in the logs?</p><p>Time to dig into the Rails code. Good thing it’s Open Source and I can just read it myself! Here is the <a href="https://github.com/rails/rails/blob/v5.1.4/actionpack/lib/action_controller/metal/strong_parameters.rb#L522">function</a> that does the actual filtering:</p><pre>def permit(*filters)<br>  params = self.class.new<br><br>  filters.flatten.each do |filter|<br>    case filter<br>    when Symbol, String<br>      permitted_scalar_filter(params, filter)<br>    when Hash<br>      hash_filter(params, filter)<br>    end<br>  end<br><br>  unpermitted_parameters!(params) if self.class.action_on_unpermitted_parameters<br><br>  params.permit!<br>end</pre><p>So looks like it does have some sort of handling for the unpermitted parameters… Lets take another <a href="https://github.com/rails/rails/blob/v5.1.4/actionpack/lib/action_controller/metal/strong_parameters.rb#L855">step</a> inside the code:</p><pre>def unpermitted_parameters!(params)<br>  unpermitted_keys = unpermitted_keys(params)<br>  if unpermitted_keys.any?<br>    case self.class.action_on_unpermitted_parameters<br>    when :log<br>      name = &quot;unpermitted_parameters.action_controller&quot;<br>      ActiveSupport::Notifications.instrument(name, keys: unpermitted_keys)<br>    when :raise<br>      raise ActionController::UnpermittedParameters.new(unpermitted_keys)<br>    end<br>  end<br>end</pre><p>But wait, what is this action_on_unpermitted_parameters thing? Turns out this is a configuration option you can define in your application initialization. So I looked in our application.rb and this is what we had defined:</p><p>config.action_controller.action_on_unpermitted_parameters = :strict</p><p>Well, this explains why we saw nothing! This isn’t actually a value this option expects!</p><p>Digging in git blame, turns out the :strict option is actually a left-over from Rails 3 days — when it was passed to config.active_record.mass_assignment_sanitizer, which was a previous method of sanitizing input that accepted the :strict option. When we implemented Strong Parameters in preparation for the Rails 5 upgrade, this option was renamed correctly, but its value remained the same. Which means that for about a year and half, we were silently dropping all of the filtered parameters instead of raising an exception.</p><p>Great! Now all I have to do is replace :strict with :raise, write a little bit of error handling code to give the user a nice friendly message informing them which of the parameters they passed was wrong and why and I’m done! Shouldn’t take too long…</p><p>After a couple of hours of work, I realized that due to the way that Strong Parameters is currently implemented, there is no easy way of telling whether the parameter was filtered out because the name wasn’t permitted or because the value wasn’t the correct type (which was the original issue — passing an integer when the filter only accepted an array). OK, no worries, for now let’s just display a general message informing which parameter was the problematic one, that should be enough to help the user figure out what’s wrong.</p><p>So I added something like this in the base API controller:</p><pre>rescue_from ActionController::UnpermittedParameters do |error|<br>  message = &quot;Invalid parameter: %s. &quot; % error.params.to_sentence<br>  message &lt;&lt; &#39;Please verify that the parameter name is valid and the values are the correct type.&#39;<br>  render_error &#39;param_error&#39;, :status =&gt; :bad_request, :locals =&gt; { :exception =&gt; error, :message =&gt; message }<br>end</pre><p>Excellent! Good day’s work, let’s open a PR and see what the CI says!</p><blockquote>About an hour later…</blockquote><blockquote>(we have an extensive test suite that is run against different Ruby versions and different databases)</blockquote><p>Jenkins reports that tests have failed. 636 failed tests to be exact. Yikes! I guess a lot of code was changed in the year-and-half since we made the change to Strong Parameters.</p><p>Digging one more level down the Rails <a href="https://github.com/rails/rails/blob/v5.1.4/actionpack/lib/action_controller/metal/strong_parameters.rb#L868">code</a>, the unpermitted_keys that raise an exception is calculated this way:</p><pre>def unpermitted_keys(params)<br>  keys - params.keys - always_permitted_parameters<br>end</pre><p>Luckily, turns out always_permitted_parameters is actually another config option. Despite what its name suggests, it is actually a list of parameters that are filtered out but don’t raise an error when they are. By default, it is set to %w( controller action ), since these two parameters are always sent by Rails, but it can easily be overridden in the application config. This is useful for parameters that you want to pass but don’t need to be assigned to the models — for example, number of entries to display per page in index views. I added a few common parameters to this list, like so:</p><pre>config.action_controller.always_permitted_parameters = %w( controller action format page per_page search locale utf8 _method authenticity_token locale _ie_support apiv id)</pre><p>This got me down to 237 failing tests. Not bad for a quick, one line change.</p><p>However, now the failing tests will need individual handling. Some may be easily fixed by adding some more common parameters that I didn’t think about to the list, but some of them are actually broken and have been for a while — we just didn’t realize it because we were silently filtering out many parameters which may have caused the tests to fail. In the following clean-up, we may even find some new bugs that were missed because of this.</p><p>On the positive side, getting this fixed properly will give us several benefits:</p><ol><li>Our users will receive informative feedback when they send an invalid parameter to the API. Even though at this point we still won’t be able to tell them what’s wrong, they will at least know which parameter(s) are causing issues, and won’t think an action succeeded when in fact some of the parameters were silently filtered out.</li><li>Developers will have an easier time when debugging. Instead of wasting hours trying to figure out why the parameter they expected wasn’t available in the controller, they will get a clear exception that they can handle.</li><li>Our tests will be better. Silently failing during tests leads us to an false feeling that everything is fine. Since we have a large test suite with good coverage (sadly not 100% yet), we often rely on tests when doing code review as an indication that new code isn’t introducing a regression.</li></ol><p>I think any one of these is in itself worth my effort, not to mention getting all three at once.</p><p>My key takeaways from all of this:</p><ul><li>Failing silently is a Bad Idea™. In this case, we missed the fact that after the change to Strong Parameters, the :strict option no longer produces the desired outcome — raising an error when filtering out parameters.</li><li>Open Source is awesome. Thanks to the fact that the Rails code is open, I was able to figure out the cause of this bug in just a few hours.</li><li>Don’t be afraid to dig in deep. Even if that means going into the code of some of your dependencies or their dependencies. I learned a whole lot in my career from digging deep — both in terms of understanding our stack better and of reading some great code written by very talented developers. If you find a bug in some dependency’s code — fix it and send a PR! That way everyone will benefit from the fix, and you won’t have to create an ugly workaround in your code.</li><li>Tests are important. Making sure they work the way you expect them to work is even more important. Don’t just check the request completed successfully, check that it did what it was supposed to do.</li><li>Code review is important. If this issue was caught in code review 1.5 years ago, we wouldn’t have to fix hundreds of tests now. Even though the PR did undergo review, it was a huge PR — 257 files changed, which makes sense when changing such a fundamental aspect that touches practically every part of the application. However, such a large amount of changes makes it easy to miss something when reviewing. Whenever possible, small PRs are always better, as the chance that something will be missed is much smaller.</li><li>Good documentation is important, especially if you are writing a library others depend on. Thanks to the Rails docs, I was quickly able to understand what each function does and identify which configuration options I need to set.</li></ul><p>Thank you for reading this! I would be happy for any feedback in the comments, or by DM on <a href="https://twitter.com/tbrisker_pro">Twitter</a>.</p><p><em>P.S. Thanks </em><a href="https://medium.com/u/68a23010fc3d"><em>Dafna Rosenblum</em></a><em> for writing the post that led me to write this one, about 2 years after I told myself I should start a blog.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2426d331625" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>