<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gaurav Rakheja on Medium]]></title>
        <description><![CDATA[Stories by Gaurav Rakheja on Medium]]></description>
        <link>https://medium.com/@gauravrakheja002?source=rss-5d8948887ef6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0ztlmGA0i5YDJMls.</url>
            <title>Stories by Gaurav Rakheja on Medium</title>
            <link>https://medium.com/@gauravrakheja002?source=rss-5d8948887ef6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:41:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gauravrakheja002/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Ticket Has Been Resolved]]></title>
            <link>https://medium.com/gojekengineering/your-ticket-has-been-resolved-8da4d85e35e8?source=rss-5d8948887ef6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8da4d85e35e8</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[refactoring]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[functional-programming]]></category>
            <dc:creator><![CDATA[Gaurav Rakheja]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 03:56:22 GMT</pubDate>
            <atom:updated>2020-04-28T03:56:22.749Z</atom:updated>
            <content:encoded><![CDATA[<h4>A deep dive into how we used Ruby’s metaprogramming abilities to remodel our ticket arbitration system as a functional pipeline.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3fAsC58pv8XaqNB8OnjvWQ.jpeg" /></figure><p>At Gojek, we’re a bunch of customer-obsessed folks. So one of our main goals is to provide our users exceptional customer support.</p><p>To allow product teams to focus on feature development and iteration, we have a centralized ticket creation/arbitration process. This story is about what it looked like, what it has become, and what it could be.</p><h3>The Past</h3><blockquote>First, let me explain our original system.</blockquote><p>We have two micro-services written in Ruby on Rails. Any Gojek product that wants to create a customer support ticket to be handled by our agents calls the Ticketing Service which — based on a given set of rules and ticket properties — determines if we can automate this ticket or not. If we can, it raises a Kafka message with the details of the ticket. The other service we listens to this Kafka message and tries to automate the ticket. When it’s done, it calls the ticketing service back to update the ticket details.</p><p>We are only going to talk about the Ticket Arbitration Service. For every ticket that we can automate, Ticketing Service assigns aissue_idto the ticket which acts as an identifier for Ticket Arbitration Service to figure out what kind of automation it needs to apply to this ticket.</p><p>Historically, this system was modeled as a state machine where each state for a ticket could be represented by a state object. Each state had a name and some metadata required to process it.</p><p>Based on this issue_id we had some plain Ruby hash maps that basically drafted the flow to be followed for automation and the state column in the ticket to keep track of what happens to a ticket. A simplified version of the hash looked something like:</p><pre>SOME_ORDER_ISSUE = {<br>UNASSIGNED: Transitions::GetDriverDetails<br>GET_DRIVER_DETAILS: Transitions::GetOrderDetails<br>GET_ORDER_DETAILS: Transitions::TakeSomeAction<br>...}</pre><p>The transitions looked like this:</p><pre><em># frozen_string_literal: true<br><br>module </em>Transitions<br>  <em>class </em>GetOrderDetails &lt; Base<br>    steps :return_next_state<br><br>    <em>private<br>    def </em>return_next_state(context)<br>      continue(State::Base.new(Issues::Generic::GET_ORDER_DETAILS, context[:ticket]))<br>    <em>end<br>  end<br>end</em></pre><p>The contract of a transition was to take the ticket and the output of the last state and return a new state object no matter what. So if something unexpected happens, the transition would return the manual state which will tell the system to assign this ticket to an agent for manual intervention.</p><p>We did the automation in a sidekiq job which took the current ticket state and the hash applicable for the ticket. Then after getting the state object for the next state, we would trigger another job from the parent job, which would look up another class from a factory method that checked the state object’s name column and provided a relevant processor for the same.</p><p>Each processor implemented the perform method to use the configuration/data provided in the state object and performed some logic which could range from getting some information from another service to sending an email which was then called by the job.</p><p>Post completion, the job would update the ticket with the name of the state just processed. The job would then just queue itself recursively until there is no next state returned for the current state. This way, whenever the automation fails in say, the second step, the state reflects that the first step was completed — which then results in the right processor being called again.</p><p>This started off very clean and easy to understand. However, when we increased the number of automation flows, the system started getting very complex. As the number of states and processors grew, the hash maps simply did not tell us the exact flow of the automation because we soon ended up with the possibility of the ticket going to two possible states from a given state.</p><p>To see what actually happens in that state, we would have to go to the transition and read every branch of it to understand what the next state could be. With the number of growing, and increasingly complicated, use cases (eg: automation for an order cancellation issue would be very different than someone saying the driver was rude) we saw a state explosion. 💥</p><p>There were now too many states and even when you want to do 90% of what another state does, you end up repeating a lot of code. Also, every transition knew about the output that came out of the previous state, and hence an automation was not order-independent.</p><p>We noticed that we had to come up with a better architecture to model this system and started thinking about what the ideal one could be. We realised that maybe there is a problem in how we defined the domain and maybe an automation is not a state machine at all.</p><p>This resulted in us performing an exercise where we thought about what the system does — taking away how it does it — to see if there is a pattern.</p><blockquote>And we found one.</blockquote><h3>The Present</h3><p>Although most of the flows can take multiple paths, they would always end up in two possible situations — ticket is solved and no intervention required or the ticket has to be moved to an agent because either something unexpected happens or the automation logic includes a manual step.</p><p>Also, in a typical state machine there are events that trigger state transitions. We realised that we only had a single event i.e. the creation of a ticket and every subsequent transition solely depended on the successful/failed completion of the last state. This told us that our flow could, in fact, be linear rather than cyclic.</p><p>For this we devised a mental model but were not sure how it would be implemented. We came up with four terms that would define the new automation domain:</p><ul><li>Query — these are steps that usually get a value from an external system(mostly another Gojek product) and add it to the context</li><li>Actions — these are steps that have external side effects, these must be handled in an idempotent manner in the automation lifecycle</li><li>Condition — these are special kinds of steps that have the ability to branch into one of the two possible flows</li><li>Fragment — an internal step that is required for the flow but has no external dependency, for example: updating a database record with some information</li></ul><p>These entities could represent even the most complex automation flows we had and were easy to talk about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKvVQVJCOHnhA69nmuJOYw.png" /></figure><p>You can see that it practically represents any flow that ever happens in a software system. We also wanted a clean way to make a lot of workflows while sharing the maximum code possible without the loss of readability. This was when we came across the <a href="https://github.com/gilbert/solid_use_case">solid_use_case</a> gem. It leverages a technique called <a href="https://www.slideshare.net/ScottWlaschin/railway-oriented-programming">Railway Oriented Programming</a>, which turned out to be exactly what we were looking for. There is a lot of content on the Internet about what this technique is, so we’re going to focus on how our team used this library to develop a framework to create automation workflows.</p><p>From the gem’s official documentation you can see an example:</p><pre>class UserSignup<br>  include SolidUseCase<br><br>  steps :validate, :save_user, :email_user<br><br>  def validate(params)<br>    user = User.new(params[:user])<br>    if !user.valid?<br>      fail :invalid_user, :user =&gt; user<br>    else<br>      params[:user] = user<br>      continue(params)<br>    end<br>  end<br><br>  def save_user(params)<br>    user = params[:user]<br>    if !user.save<br>      fail :user_save_failed, :user =&gt; user<br>    else<br>      continue(params)<br>    end<br>  end<br><br>  def email_user(params)<br>    UserMailer.async.deliver(:welcome, params[:user].id)<br>    # Because this is the last step, we want to end with the created user<br>    continue(params[:user])<br>  end<br>end</pre><p>As you can see, it essentially allows you to chain a set of operations without doing a lot of nested conditionals. Although this is nice, solid_use_case by itself does not have an opinion of the type of value returned from an operation i.e. a step can take a user and return an email.</p><p>Although this is valuable, it limits what can and cannot be chained with each other. One of the goals of the rewrite we were doing was to take code that was being run in context of a completely different automation and to use it in a new one, for that we would never know in advance what we will and will not have across different automations.</p><p>To make the set up a little more robust, we needed a few things:</p><ul><li>Ever use case should take and return a hash. This would allow us to not limit the use cases to a given set of inputs. We call this hash the ‘automation context’.</li><li>No use case should delete any key from the context</li><li>If you need to read the key from the context you need to validate it. If you want to add a key to the context you need to call it out in advance. This is so that a person does not have to read the logic to figure out what is needed to run a step and what will be added to context after it runs.</li></ul><p>We were already using a gem <a href="https://github.com/deseretbook/classy_hash">classy_hash</a> to do some schema validations in a completely different context. It allows for a really expressive API to perform validations on a given ruby hash. From the documentation, the usage looks something like:</p><pre>schema = {<br>  key1: String,<br>  key2: Integer,<br>  key3: TrueClass<br>}</pre><pre>hash = {<br>  key1: &#39;A Hash with class&#39;,<br>  key2: 0,<br>  key3: false<br>}<br><br>ClassyHash.validate(hash, schema)</pre><p>So we combined the two gems and added a sprinkle of metaprogramming to create what would form the base of our framework. This is the actual code:</p><pre><em># frozen_string_literal: true<br><br>class </em>SuperUseCase<br>  <em>include </em>SolidUseCase<br><br>  <em>class </em>&lt;&lt; <em>self<br>    def </em>steps(*args)<br>      args.unshift(:validate_context)<br><br>      <em>super</em>(*args)<br>    <em>end<br><br>    def </em>contract(**context_validations)<br>      @__validations ||= {}<br>      @__validations.deep_merge!(context_validations)<br>    <em>end<br><br>    def </em>adder_for(*keys)<br>      @__adders ||= []<br>      keys.each <em>do </em>|key|<br>        <em>define_method </em>&quot;add_#{key}&quot; <em>do </em>|value, context|<br>          context[key.to_sym] = value<br>        <em>end<br>        </em>@__adders &lt;&lt; key<br>      <em>end<br>    end</em></pre><pre><em>   def </em>can_branch?<br>     <em>true<br>   end</em></pre><pre><em>  end<br><br>  private<br><br>  def </em>validate_context(context)<br>    errors = []<br>    <em>if </em>validate_contract(context, errors)<br>      continue(context)<br>    <em>else<br>      </em>context[:errors] = errors<br>      <em>fail </em>:contract_unfulfilled,<br>           context: context<br>    <em>end<br>  end<br><br>  def </em>validate_contract(context, errors)<br>    validations = <em>self</em>.class.keys_required || {}<br>    ClassyHash.<em>validate</em>(<br>      context,<br>      validations,<br>      raise_errors: <em>false</em>,<br>      errors: errors,<br>      verbose: <em>true<br>    </em>)<br>  <em>end<br>end</em></pre><p>Before I explain what’s going on, here’s how you use this base class:</p><pre>class MyUseCase &lt; SuperUseCase<br>  contract some_key: String,<br>           another_key: Integer<br>  <br>  adder_for :key1, <br>            :key2<br><br>  steps :test_step,<br>        :next_step<br>  <br>  <em>private<br>  <br></em>  def test_step(context)<br>    result = context[:some_key].something<br>    add_key1(result, context)<br>    continue(context)<br>  end<br>  <br>  def next_step(context)<br>    another_result = context[:another_key].another_thing<br>    add_key2(another_result, context)<br>    continue(context)<br>  end<br>end</pre><p>You can see there just a subtle difference between how the library presents a use case and how we do it.</p><ul><li>You have to define a contract on each use case that should contain the key and type of what this use case will read from the context</li><li>You have to give a list of keys this use case will add to the context</li></ul><p>This makes the framework a little more expressive and also order independent while still telling you how the different use cases are dependent on each other.</p><p>Now that we had the ability to build a pipeline, it was time to create a framework that will handle the actual ticket automation. The reason this was not enough was that it only handles linear flows. We had use cases where an automation could branch out at multiple points and then some would even converge back (as pointed out by the diagram). Until this point, we’ve not built that ability.</p><p>It was now time to define the actual entities in our domain. The steps actions, queries, fragments were all quite similar and only differed in the kind of code they had. The most interesting bit was to model the conditional/branching steps.</p><p>For a use case that branches into multiple flows, the library had no support and railway oriented programming also did not give a solid way to solve our problem. So we leveraged Ruby’s metaprogramming capabilities to dynamically create use cases that could branch. This is the code for the condition base class:</p><pre><em># frozen_string_literal: true<br><br>module </em>Steps<br>  <em>module </em>Conditions<br>    <em>class </em>ConditionBase &lt; StepBase<br>      <em>class </em>&lt;&lt; <em>self<br>        def </em>inject(success:, failure:)<br>          Class.<em>new</em>(<em>self</em>) <em>do<br>            </em>on_success success<br>            on_failure failure<br><br>            contract <em>self</em>.superclass.keys_required</pre><pre>            adder_for self.superclass.keys_added      </pre><pre>            .....<em><br>          end<br>        end<br><br>        def </em>on_success(use_case)<br>          @__on_success = use_case<br>          ensure_step_added<br>        <em>end<br><br>        def </em>on_failure(use_case)<br>          @__on_failure = use_case<br>          ensure_step_added<br>        <em>end<br><br>        def </em>ensure_step_added<br>          <em>return if </em>@__steps.present?<br><br>          steps :decide_action<br>        <em>end<br><br><br>        def </em>success_case<br>          generated_case(@on_success)<br>        <em>end<br><br>        def </em>failure_case<br>          generated_case(@on_failure)<br>        <em>end<br><br>        def </em>generated_case(some_steps)<br>          Class.<em>new</em>(SuperUseCase) <em>do<br>            </em>steps *some_steps<em><br>          end<br>        end<br><br><br>        def </em>can_branch?<br>          <em>true<br>        end<br>      end<br><br>      private<br><br>      def </em>decide_action(context)<br>        <em>if </em>check(context)<br>          <em>self</em>.class.success_case.run(context)<br>        <em>else<br>          self</em>.class.failure_case.run(context)<br>        <em>end<br>      end<br><br>      def </em>check(_context)<br>        <em>raise </em>&quot;implement check in #{<em>self</em>.class}. Must return true/false&quot;<br>      <em>end<br>    end<br>  end<br>end</em></pre><p>There’s a lot going on here so let’s break it down:</p><ul><li>The class has an inject method that returns a new use case that copies over the contract and adders from the base class, and sets the instance variables @on_success and @on_failure which are just arrays of use cases on the metaclass of the generated class.</li><li>You can see that the base class exposes two methods called success_case and failure_case which are generated use cases which have either the @on_success or the @on_failure as the steps.</li><li>The base condition also declares that there is only one step in this use case i.e. decide_action. In the definition, you can see that it calls a method called check which has to be implemented by the class inheriting from the condition base.</li></ul><p>The concept will become a lot more clear once you see what a real condition step looks like:</p><pre><em># frozen_string_literal: true<br></em><br><em>class </em>SomeCondition &lt; ConditionBase<br><br>  contract some_key: String<br><br>  <em>private<br><br>  def </em>check(context)<br>   context[:some_key] == EXPECTED_VALUE<br>  <em>end<br>end<br></em></pre><p>A condition in the definition does not declare what happens when it is met or not met, it only defines the keys it needs to check the condition and the condition to be checked. The metaprogramming kicks in when another use case wants to have a condition as a part of its steps.</p><p>This is what the end results look like:</p><pre><em># frozen_string_literal: true<br><br>module </em>Strategies<br>  <em>class </em>SomeAutomation &lt; SuperUseCase<br><br>    steps Steps::Queries::GetOrderDetails,<br>          Steps::Conditions::SomeCondition.inject(<br>            success: [<br>              Steps::Fragments::UpdateTicket,<br>              Steps::Queries::GetDriverDetails,<br>              Steps::Actions::DoSomeAction,<br>              Steps::Conditions::NestedCondition.inject(<br>                success: [Steps::Actions::SwitchToAgent], <br>                failure: [Steps::Actions::SolveTicket]<br>              ),<br>            ],<br>            failure: [Steps::Actions::SwitchToAgent]<br>          )<br>  <em>end<br>end</em></pre><p>As you can see we’ve achieved the following things</p><ul><li>Order independence</li><li>Re-usability/composition as a default</li><li>Increased verbosity and readability of a given workflow</li></ul><h3><strong>The Future</strong></h3><ul><li>A condition is used very similarly to a <a href="https://stackoverflow.com/questions/36636/what-is-a-closure">closure</a> in functionality as the inject method returns a new class that could run either of the two dynamically created flows injected by its caller. This idea can be extended to abstract away steps that share most of the code but differ by some parameter.</li><li>Moving the order of the steps to a database table. Since each step already exposes the keys it adds and the keys it requires.</li><li>Eventually, build a <a href="https://zapier.com/home">Zapier</a>-like tool that works with all Gojek products and allows creation of custom workflows as long as the logic for what happens in a step is coded.</li></ul><p>That’s the journey of how we rewrote our ticket arbitration system to be cleaner and more intuitive. If you liked what you read, consider <a href="https://mailchi.mp/go-jek/gojek-tech-newsletter">signing up for our newsletter</a> to have our stories delivered straight to your inbox. 🙌</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPWwoozDxeKoXMtXEyrtLw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8da4d85e35e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gojekengineering/your-ticket-has-been-resolved-8da4d85e35e8">Your Ticket Has Been Resolved</a> was originally published in <a href="https://medium.com/gojekengineering">Gojek Product + Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use ant design + dark mode in rails + webpacker + react]]></title>
            <link>https://medium.com/@gauravrakheja002/how-to-use-ant-design-dark-mode-in-rails-webpacker-react-8bcae73e9fe?source=rss-5d8948887ef6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bcae73e9fe</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[webpacker]]></category>
            <category><![CDATA[webpack]]></category>
            <category><![CDATA[ant-design]]></category>
            <dc:creator><![CDATA[Gaurav Rakheja]]></dc:creator>
            <pubDate>Sun, 19 Apr 2020 10:45:32 GMT</pubDate>
            <atom:updated>2020-04-19T10:45:32.546Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to use ant design with dark mode in rails + webpacker + react</h3><p>We have a react-rails application and we use ant design already, there is nothing special required to set it up but when we wanted to modify the theme and enable dark mode it was quite a challenge.</p><p>Since I did not find a good blog telling this in one place I decided to write one. This post assumes you have a rails application already setup with webpacker.</p><p>Webpacker makes working with javascript in rails very similar to how to it feels while working on a regular javascript application, but there can be a few differences on how to do things.</p><p>Installing ant design(this guide is for v4.1.4) is as straight forward as</p><pre>yarn add antd</pre><p>Now when it comes to importing styles from ant design there are multiple approaches.</p><ul><li>Import the ant design styles in your application.scss/css.</li><li>Import the ant design styles in your application.js javascript pack.</li><li>Use babel style import to get the styles</li></ul><p>We’re going to go with the third approach as it lets us inject less variables in ant design styles without having to write both less and scsss working in our application.</p><p>What we’re basically doing is teaching webpacker to understand and compile less. You can do that by</p><pre>yarn add less<br>yarn add less-loader</pre><p>The less package is self explanatory, less loader is like any other webpack <a href="https://webpack.js.org/loaders/">loader</a> and we’re going to see how to make use of it when working with webpacker.</p><p>First, we will add the code to load the ant design styles using babel when our application boots by adding this line of code to babel.config.js in the application root</p><pre>plugins:[<br>  ...otherPlugins, <br>  [&quot;import&quot;, {&quot;libraryName&quot;: &quot;antd&quot;, &quot;style&quot;: <em>true</em>} ] // add this<br>].filter(Boolean)</pre><p>If you don’t want to load the less files you can remove the style true call and it will load the css styles from the library</p><p>To make webpacker compile less files we need to do a few things</p><p><strong>Edit webpacker.yml</strong></p><pre>default: &amp;default<br>  ... <br>  extensions: <br>    - ... <br>    - .less              //add this<br>    - .module.less       //add this</pre><p><strong>Create a new file config/webpack/lessLoader.js</strong></p><pre><em>const </em>getStyleRule = require(&#39;@rails/webpacker/package/utils/get_style_rule&#39;)<br><em>const </em>{ getThemeVariables } = require(&#39;antd/dist/theme&#39;)<br><br>module.exports = getStyleRule(/\.less$/i, <em>false</em>, [<br>  {<br>    loader: &#39;less-loader&#39;,<br>    options: {<br>      modifyVars: {<br>        ...getThemeVariables({<br>          dark: <em>true</em>,<br>          compact: <em>true</em>,<br>        }),<br>        &#39;primary-color&#39;: &quot;#39c486&quot;, <br>      },<br>      javascriptEnabled: <em>true</em>,<br>    },<br>  }<br>])</pre><p>Here we’re using the wepacker getStyleRulefunction to inject the less loader, and we’re using the getThemeVariables function from ant design to add the variables required for the dark and compact themes. We’re also overriding the primary color used in all the components, you can refer to <a href="https://github.com/ant-design/ant-design/blob/master/components/style/themes/default.less">this</a> link to see all the variables you can override.</p><p><strong>Adds the new loader to config/webpack/environment.js</strong></p><pre><em>const </em>lessLoader = require(&#39;./lessLoader&#39;)<br>environment.loaders.prepend(&#39;style&#39;, lessLoader)</pre><p>And that’s it, you should have the ability to toggle dark/compact theme and also override other variables without having to write less code in your application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bcae73e9fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So Rails is slow?]]></title>
            <link>https://medium.com/@gauravrakheja002/so-rails-is-slow-71de1b07636a?source=rss-5d8948887ef6------2</link>
            <guid isPermaLink="false">https://medium.com/p/71de1b07636a</guid>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[activerecord]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[monolith]]></category>
            <dc:creator><![CDATA[Gaurav Rakheja]]></dc:creator>
            <pubDate>Wed, 17 Apr 2019 15:01:09 GMT</pubDate>
            <atom:updated>2020-04-19T10:49:26.187Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to correctly scale with Active Record</h3><p>I would just start with some background on where I come and why I say what I say.</p><p>I was working at a startup called Vinterior for about 8 months. When I joined we were about 4 full stack developers. When I left we were a team of 6. Somewhere midway my journey the marketing team launched a few paid campaigns which scaled the traffic up by 600%.</p><p>We were a monolith rails application and we were not ready for such an increase in the traffic as well as the data. We had no SRE(Site Reliability Engineer). I spent time reading up and learning about rails performance and also took a short course on the same. A few key points to note from this are:</p><ul><li>Optimization of an end-point was a story for me.</li><li>We had more than 100 endpoints and most of them were allocating more than a few million objects(more on how I got this later).</li><li>I spent about 4 months where I would spend half a day building features, and the other half working on site performance.</li><li>When I started the average response time was 500ms and the average 95th percentile response time was above 1000ms. We were also running 8–10 medium-sized performance dynos on Heroku on average even with low traffic. We were also facing a lot of timeout errors(as many as 1000+ a day) where the average traffic volume was not too huge.</li><li>After those 4 months, we the average response time was 90–110ms and the average 95th percentile response time was around 500ms. We were now running about 2–5 medium-sized performance dynos on average and about 8–10 during traffic spikes. The number of timeout errors went to 1–2 a day to none.</li></ul><p>During my time doing these optimizations, I started to notice a few patterns and made some observations about some of the things I think require a lot of attention.</p><p><strong>Firstly, Your Database is not your application</strong></p><ul><li>The purpose of a database is to provide centralized access to the same data to multiple users. This by design stops us from doing things like deploying a 100 servers just as databases at least without a lot of complication(it’s comparatively way easier to manage multiple app servers)</li><li>The database is usually another machine, which means there is a network latency involved when you do anything with the database.</li><li>If you do efficient DB calls and think about it while coding you can delay having to scale up your DB or even worse be bottlenecked by it.</li></ul><p><strong>Also, Ruby is a little notorious and so is ActiveRecord</strong></p><ul><li>One of the strongest features of the language is the flexibility it offers, it allows you to fall into pitfalls and tries to hold up as long as possible.</li><li>When you create an AR model, you add all of the code that is in ActiveRecord::Base in your class i.e. you include 40 different modules with different functionality all of which you will never use.</li><li>Does this make AR bad? No. It means it has great power. And with great power comes great responsibility.</li><li>Most of the performance issues are caused by how ruby memory management works. It causes memory bloat with large allocations.</li></ul><p><strong>A little about memory bloat</strong></p><ul><li>Ruby memory is weird.</li><li>Let’s say you had a program that was using 100Mb of memory to run, in that program you allocated a huge array, that required 400Mb of additional memory, the ruby process will get the additional 400Mb from the operating system immediately but due to how the ruby GC works, it’s not going to free up the extra memory after you’re done with the array. This acts as an optimization as the process doesn’t have to keep allocating and freeing memory.</li><li>But this also means that one large allocation will have a long-lasting effect on the whole program. When you put this in context of a multi-threaded complex rails application things get even trickier.</li></ul><p><strong>So what are the pain points then?</strong></p><ul><li>If you allocate a huge collection and then only use part of that data, that allocation was useless in terms of memory. Eg, loading 1000 users into memory by creating an array and counting the users with a certain attribute by doing ruby operations as opposed to DB operations.</li><li>Missing pagination is a time bomb.</li><li>N+1 queries i.e. you wanted n users which you looped through and then inside the loop added code that does a DB query so n users lead to n additional queries.</li></ul><p><strong>How do you solve them:</strong></p><p>It’s simple. Read code, understand the responsibility that comes by using a framework thousands of people contribute every year to.</p><p>When done right, rails scales just as smoothly as any other language/framework.</p><p>Over repeated work on performance, I have gathered a few tools that I think can avoid a lot of the problems mentioned before by design without you having to think a lot about the underlying mechanics.</p><p><strong>Your friend Active Record</strong></p><p>Below is a snapshot from the official rails documentation on Active Record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tu-7z3DP4kwEC0jn" /></figure><p>These are not so many methods.</p><ul><li>These are all the methods in ActiveRecord that are capable of generating SQL queries.</li><li>Learn to identify these, tell yourself that it is going to go to another computer to do this operation if you use any of these methods.</li><li>Do not use them without care it will cost you.</li></ul><p><strong>To query or not to query</strong></p><p>It is perfectly fine to put any of the methods mentioned before in these places.</p><ul><li>Controllers</li><li>Service Objects</li><li>Scopes</li><li>Relationships</li></ul><p>It is almost never a good idea to use them in instance methods of models as models often tend to be looped through which causes n+1’s if you’ve used these methods.</p><p><strong>Tool #1: Active Record Query Cache and Includes</strong></p><p>Read below, it’s a small section but take a second and absorb what it says.</p><p><a href="https://edgeguides.rubyonrails.org/caching_with_rails.html#sql-caching">Caching with Rails: An Overview - Ruby on Rails Guides</a></p><pre>Query caching is a Rails feature that caches the result set returned by each query. If Rails encounters the same query again for that request, it will use the cached result set as opposed to running the query against the database again.</pre><ul><li>This is your best friend as a rails developer.</li></ul><p>Another powerful tool is the active record .includes method.</p><ul><li>This function, available on any <a href="https://apidock.com/rails/ActiveRecord/Base">Base</a> derived class, allows eager loading. So when iterating over a result, all the data can be pre-cached, reducing the number of queries <a href="https://apidock.com/rails/ActiveRecord/QueryMethods/from">from</a> n+1 to 2.</li><li>Several ways to use this include:</li></ul><pre>Post.includes(:author).where(..)<br>Post.includes([:author, :comments]).<a href="https://apidock.com/rails/ActiveRecord/QueryMethods/where">where</a>(..)<br>Post.includes( :comments =&gt; :replies ).<a href="https://apidock.com/rails/ActiveRecord/QueryMethods/where">where</a>(..)</pre><p><strong>Tool #2: Conditional Relationships</strong></p><p>We’ve all seen relationships declared in models. Like user has_many posts and post belongs to user etc.</p><p>What we don’t see often is that the relationship methods given by rails accept a lambda with active record code in it. It allows you to build powerfull relationship that can help you avoid putting the same code in an instance method.</p><p>In the example above, you can also declare a relationship like</p><pre>Class User &lt; ActiveRecord::Base</pre><pre><br>has_many :active_posts, <br>         class_name: “Post”, <br>         -&gt; { where(post: { status: “active” } }</pre><pre>end</pre><p><strong>Tool #3: Activerecord merge</strong></p><p><a href="https://apidock.com/rails/ActiveRecord/SpawnMethods/merge">merge (ActiveRecord::SpawnMethods) - APIdock</a></p><p>For those of you who did not open the above link, this is what it says</p><p><strong>merge</strong>(other) <em>public</em></p><p>Merges in the conditions from other, if other is an <a href="https://apidock.com/rails/ActiveRecord/Relation">ActiveRecord::Relation</a>. Returns an array representing the intersection of the resulting records with other, if other is an array.</p><pre><a href="https://apidock.com/rails/Post">Post</a><strong>.</strong>where<strong>(</strong>published<strong>:</strong> true<strong>).</strong>joins<strong>(</strong>:comments<strong>).</strong><a href="https://apidock.com/rails/ActiveRecord/SpawnMethods/merge">merge</a><strong>(</strong> <a href="https://apidock.com/rails/Comment">Comment</a><strong>.</strong>where<strong>(</strong>spam<strong>:</strong> false<strong>)</strong> <strong>)</strong><br><em># Performs a single join query with both where conditions.</em></pre><pre>recent_posts <strong>=</strong> <a href="https://apidock.com/rails/Post">Post</a><strong>.</strong>order<strong>(&#39;</strong>created_at DESC<strong>&#39;).</strong>first<strong>(</strong>5<strong>)</strong><br><a href="https://apidock.com/rails/Post">Post</a><strong>.</strong>where<strong>(</strong>published<strong>:</strong> true<strong>).</strong><a href="https://apidock.com/rails/ActiveRecord/SpawnMethods/merge">merge</a><strong>(</strong>recent_posts<strong>)</strong><br><em># Returns the intersection of all published posts with the 5 most recently created posts.</em><br><em># (This is just an example. You&#39;d probably want to do this with a single query!)</em></pre><p>Procs will be evaluated by merge:</p><pre><a href="https://apidock.com/rails/Post">Post</a><strong>.</strong>where<strong>(</strong>published<strong>:</strong> true<strong>).</strong><a href="https://apidock.com/rails/ActiveRecord/SpawnMethods/merge">merge</a><strong>(-&gt;</strong> <strong>{</strong> joins<strong>(</strong>:comments<strong>)</strong> <strong>})</strong><br><em># =&gt; Post.where(published: true).joins(:comments)</em></pre><p>This is mainly intended for sharing common conditions between multiple associations.</p><ul><li>The above words are gold. Because merge allows you to create some really powerful queries without having to repeat the logic.</li></ul><p><strong>Enough Talk, time for some action</strong></p><ul><li>Let me explain the above tools with an example problem statement.</li><li>I need all the users who have active posts. And then show the content of those posts. In a regular set up you’d expect something like this. Where active is just a scope and is also a method on the post model.</li></ul><pre>users_with_active_posts = User.all.reject { |user| user.posts.none?(&amp;:active) }</pre><pre>users_with_active_posts.each do |user|</pre><pre>  user.posts.active.each do |post|</pre><pre>    puts post.content<br>  end<br>end</pre><ul><li>Although the above code looks innocent, there are several problems with this:</li><li>Loads all the users in memory.</li><li>Loads all the posts for each user in memory to get the users where all the posts are active also leading to an n+1 to get them.</li><li>It then does n+1 queries to get all the active posts for each of these users.</li></ul><p>It works perfectly fine when the tables are small and seems very innocent. However, if deployed this will soon become code that keeps timing out because of how long it takes.</p><p><strong>The alternative.</strong></p><pre>class User &lt; ActiveRecord::Base</pre><pre>  has_many :posts</pre><pre>  has_many :active_posts,<br>    -&gt; { merge(Post.active) },<br>    class_name: &quot;Post&quot;</pre><pre>  scope :with_active_posts, -&gt; { joins(:active_posts) } </pre><pre>end</pre><pre>User.with_active_posts.includes(:active_posts).find_each do |user|</pre><pre>  user.active_posts.each do |post|</pre><pre>    puts post.content</pre><pre>  end</pre><pre>end</pre><p><strong>Wins:</strong></p><ul><li>The logic of how a post is active is still in the post model</li><li>We’re only doing 2 database queries ever.</li><li>Not loading all users or all of the posts into memory unnecessarily.</li><li>Use of find_each that allows batch access to loops which will ensure we never load more than a specific amount of records into memory at once(default is 1000). There is also a similar .find_in_batches method that returns you a collection instead of a record at once which allows you to do more complex queries in batches(eg bulk updates and deletes).</li><li>The above pattern sets a DSL which when followed by even junior engineers and can lead to everyone writing scalable code without even completely understanding the underlying concepts completely i.e. the leveraging the rails ideology of convention over configuration.</li><li><strong>The resultant code is more readable, more performant and even more object-oriented than the code before.</strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71de1b07636a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[how to get autocomplete working with sunspot solr and typeahead js]]></title>
            <link>https://medium.com/@gauravrakheja002/how-to-get-autocomplete-working-with-sunspot-solr-and-typeahead-js-9d77daf3faa6?source=rss-5d8948887ef6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d77daf3faa6</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[solr]]></category>
            <category><![CDATA[rails]]></category>
            <dc:creator><![CDATA[Gaurav Rakheja]]></dc:creator>
            <pubDate>Thu, 15 Feb 2018 06:40:56 GMT</pubDate>
            <atom:updated>2018-02-15T06:40:56.561Z</atom:updated>
            <content:encoded><![CDATA[<p>I have recently implemented autocomplete for the searchbar in the website I work for and after struggling so hard with what i found on the internet, i decided that i will a blog post combining the things i found at different places so that people don’t struggle as much as i did.</p><p>I’ve tested this scenerio with rails 4.0.6 and with gems sunspot_solr, rsolr and sunspot_rails installed. We use the new way to configure solr(using configsets instead of {app-root}/solr/conf).</p><p>To get autocomplete to work with solr, you need to first enable partial keyword matching if you haven’t already. This can be achieved by digging into the schema.xml file under in your config(in configset/{set-name}/conf directory). You need to stop solr by executing bundle exec rake sunspot:solr:stop before/after changing this file. Then replace the text field type definition by the following:</p><pre>&lt;fieldType name=&quot;text&quot; class=&quot;solr.TextField&quot; omitNorms=&quot;false&quot;&gt;<br>  &lt;analyzer type=&quot;index&quot;&gt;<br>    &lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/&gt;<br>    &lt;filter class=&quot;solr.StandardFilterFactory&quot;/&gt;<br>    &lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/&gt;<br>    &lt;filter class=&quot;solr.NGramFilterFactory&quot; minGramSize=&quot;2&quot; maxGramSize=&quot;15&quot;/&gt;<br>  &lt;/analyzer&gt;<br>  &lt;analyzer type=&quot;query&quot;&gt;<br>    &lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/&gt;<br>    &lt;filter class=&quot;solr.StandardFilterFactory&quot;/&gt;<br>    &lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/&gt;<br>  &lt;/analyzer&gt;<br>&lt;/fieldType&gt;</pre><p>(Replace Ngram with EdgeNgram if you want to break down words sarting from one end rather than on just size, there’s plenty of documentation about this on the internet)</p><p>After this you need to confirm if the schema file got updated. Do this by going to the solr admin panel, select a core, then open the files tab and click on schema.xml and ensure the file contains the changes you made.</p><p>For those who do not see the changes loaded, check if you started your solr server earlier with sunspot-solr start command as that loads the schema from the gem directory, to solve this restart your system, then run solr with bundle exec rake sunspot:solr:start command and you should see the changes being reflected in the solr panel.</p><p>After you’ve ensured the schema is updated run bundle exec rake sunspot:reindex to reindex solr with the newly added libraries(ngram in this case). Now if you perform a solr search(functionally or through the solr search method for a model), you should get results for partial words(‘coff’ will return results with objects containing ‘coffee’, ‘coffeee’, ‘coffin’, etc).</p><p>This takes care of half of the stuff. Now make an api end point(if you don’t have one already) to perform the search that you need to do(in my case i had to perform search in 3 different models and return the results.</p><p>add the following to your routes</p><pre><em>get </em>&#39;search/:keyword&#39;, to: &#39;search#new&#39;, as: &#39;search&#39;</pre><p>Now you can call this end point from anywhere.</p><p>Now let’s set up the controller.</p><pre><em>module </em>Api<br> <em>class </em>SearchController &lt; ActionController::Base<br>   <em>before_filter </em>:perform_search, only: [:new]</pre><pre><em>   def </em>new<br>     respond_with(@results, default_template: &#39;api/search/new&#39;,status: 200)<br>   <em>end</em></pre><pre><em>  private</em></pre><pre><em>  def perform_search<br>    search = {your-model}.search do </em></pre><pre>     fulltext params[:keyword]<br>    end<br>   @results = search.results</pre><pre><em>  end <br>end<br>end</em></pre><p>I used rabl rails to format the data going in the api response, that can be done by adding the rabl gem and making a new new.rabl file under api/search/ in your views. Inside the file you can format the results any way you want.</p><pre>collection @results<br>attributes :name</pre><pre>#search keyword is implemented in the model to give a string such as &#39;Products, Users, etc&#39;<br>node(:source) { |result| result.search_keyword } </pre><p>This will give the following result when the api is called:</p><pre>[{&quot;name&quot;:&quot;Table Lamps&quot;,&quot;source&quot;:&quot;Products&quot;},{&quot;name&quot;:&quot;Sam&quot;,&quot;source&quot;:&quot;Users&quot;}]</pre><p>Next step is to install typeahead-js library. There are two versions of the library one of which is no longer maintained.</p><ul><li><a href="https://github.com/twitter/typeahead.js/">https://github.com/twitter/typeahead.js/</a> is no longer maintained.</li><li><a href="https://github.com/corejavascript/typeahead.js">https://github.com/corejavascript/typeahead.js</a> is.</li></ul><p>The typeahead bundle consists of two parts: a token engine(bloodhound) and a UI library(typeahead). We use both of these to display add autocomplete in our application.</p><p>Add the class typeahead to the search bar in your view.</p><p>Now add the following javascript in your application:</p><pre>var autoComplete = function(){<br>  var {your-engine-name} = new Bloodhound({<br>      datumTokenizer: Bloodhound.tokenizers.ngram,<br>      queryTokenizer: Bloodhound.tokenizers.ngram,<br>      remote: {<br>          url: &#39;/api/search/%QUERY&#39;, #your api endpoint<br>          wildcard: &#39;%QUERY&#39; #this is what gets replaced by keyword typed and used in the api call<br>      }<br>  });</pre><pre>  {your-engine-name}.initialize();</pre><pre>  $(&#39;.typeahead&#39;).typeahead({<br>      hint: true,<br>      highlight: true,<br>      minLength: 2 #the number of letters after which the engine makes the call<br>  }, {<br>      name: &#39;products&#39;,<br>      displayKey: &#39;name&#39;,<br>      source: {your-engine-name}.ttAdapter(), <br>      limit: 100, #max number of suggestions to display<br>      templates: {<br>          notFound: function(data){<br>              return &#39;{the html you want when no results found}&#39;;<br>          },<br>          pending: function(data){<br>              return &#39;{the html you want for when the user is typing and search is being performed}&#39;;<br>          },<br>          suggestion: function(data) {<br>              return &#39;{the html you want for each of the suggestions}&#39;;<br>          }<br>      }<br>  });<br>};</pre><pre>#the data above responds to the fields sent in the api response, for example for the api we created here, we can get the name of the member entity by calling data.name, and source by calling data.source</pre><p>It is also possible to customize both of these libraries further. The above implementation is simple for the sake of understanding the mechanics.</p><p>And that’s it, you’ve successfully added autocomplete with sunspot_solr and typeahead-js.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d77daf3faa6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>