<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Guilherme Plichoski on Medium]]></title>
        <description><![CDATA[Stories by Guilherme Plichoski on Medium]]></description>
        <link>https://medium.com/@gplichoski?source=rss-d3b692940f5e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*OdId5-pdHO-zLa50wzNbGw.jpeg</url>
            <title>Stories by Guilherme Plichoski on Medium</title>
            <link>https://medium.com/@gplichoski?source=rss-d3b692940f5e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:58:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gplichoski/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Priority Queues and Sidekiq::Batches]]></title>
            <link>https://medium.com/magrathealabs/priority-queues-and-sidekiq-batches-ccc788634f72?source=rss-d3b692940f5e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccc788634f72</guid>
            <category><![CDATA[sidekiq]]></category>
            <category><![CDATA[batch]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[priority-queue]]></category>
            <category><![CDATA[rails]]></category>
            <dc:creator><![CDATA[Guilherme Plichoski]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 18:56:07 GMT</pubDate>
            <atom:updated>2019-08-05T18:56:07.424Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to implement Priority Queues using Sidekiq::Batches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IM6vK6yN8izE7v3RjSOBhg.jpeg" /></figure><p>Once upon a project, the team I work with and me faced the challenge of implementing a priority queue using <em>Sidekiq::Batches</em>. In our application, we had batches with varying sizes (most of them with about 10 jobs, while some batches had about thousands), so we had to come up with a way to prioritize the small ones first.</p><p>We found two ways to implement this. In the first one (which was our first thought), the application enqueues the jobs with a calculated priority and then they are sorted accordingly, as shown in the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*1x5UH8icw4P4GHWM2qFzRg.png" /></figure><p>However, due to <em>Sidekiq::Batch</em>’s logic of enqueueing being private, it’s not possible to change it without unzipping the gem, which is undesirable.</p><p>The second alternative seemed more viable — <strong><em>a multiple pool approach</em></strong></p><p>In this solution, we maintain different queues for each priority and the application is responsible for setting the priority of each job and sending it to the appropriate queue, as shown in the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*O2LHqY6r0AZCU401t1f6dA.png" /></figure><p>Aside from being a simpler approach, it solved the problem! Additionally, in the single pool approach, messages that have a very low priority could be continually superseded and might never be processed. In the multiple pool, lower priority messages will always be processed. Also, it is possible to configure <em>Sidekiq</em> to execute only queues being used, this way lefting empty queues deactivated. On the other hand, we had to implement the logic to score the jobs, thus increasing the runtime and resource usage in our application.</p><h3>The implemented solution</h3><p>Basically, we’ve created a method to return the queue based on job priority. Then, we push the jobs directly to the queue as follows:</p><pre><strong>queue_name</strong> = queue_by_size(size_of_job) <strong># method to return the queue</strong></pre><pre>batches.each do |batch_args|<br>  Sidekiq::Client.push(<br>    &#39;class&#39; =&gt; &#39;ClassJob&#39;,<br>    &#39;queue&#39; =&gt; <strong>queue_name</strong>,<br>    &#39;retry&#39; =&gt; 3,<br>    &#39;args&#39; =&gt; [...]<br>  )<br>end</pre><p>Since we cannot define a custom queue in runtime using #perform, we used Sidekiq::Client#pushdirectly. This way, we could set the parameter queue passing the custom queue name.</p><p>Ta-da! 🎉<br>Thats it, as simple as it seems!<br>Hope this helps someone 😜</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccc788634f72" width="1" height="1" alt=""><hr><p><a href="https://medium.com/magrathealabs/priority-queues-and-sidekiq-batches-ccc788634f72">Priority Queues and Sidekiq::Batches</a> was originally published in <a href="https://medium.com/magrathealabs">Magrathea</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>