<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aman Garg on Medium]]></title>
        <description><![CDATA[Stories by Aman Garg on Medium]]></description>
        <link>https://medium.com/@aman-garg?source=rss-1f6d3dd56261------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Lbi7yWpFHDIx84qDzFr7XA.jpeg</url>
            <title>Stories by Aman Garg on Medium</title>
            <link>https://medium.com/@aman-garg?source=rss-1f6d3dd56261------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:54:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aman-garg/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The “Nullbuster ” workaround for the unique index in SQL]]></title>
            <link>https://aman-garg.medium.com/the-nullbuster-workaround-for-the-unique-index-in-sql-9423c315cd2e?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/9423c315cd2e</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[nullbuster]]></category>
            <category><![CDATA[gorm]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Sun, 17 Oct 2021 20:03:39 GMT</pubDate>
            <atom:updated>2022-02-01T17:41:09.805Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*vj9fJABgUbD89fMfJ21-LA.png" /></figure><p>The SQL community is fairly <a href="https://stackoverflow.com/questions/4081783/unique-key-with-nulls">divided</a> on what a “null” means for a column that is a part of a unique index key. MySQL allows it, while some such as SQL Server outright reject it. It’s no surprise that there are no SQL standards governing whether nulls are even comparable or not.</p><p>This misalignment, however, does not preclude the situations in which a null unique index is actually useful. We’ll take a real life use case to demonstrate why one might need to find a workaround for it.</p><p>Consider yourself to be a developer in ad-tech for an e-commerce firm building a system for users to create ad <a href="https://developers.google.com/admob/android/banner">banners</a> that can go live. Once live, banners are clickable and target a “content”, which is what people who click on it will be redirected to.</p><p>The primary users of your system are<em> internal teams who </em>create multiple banners for contents day in and day out. As a rule of thumb, there may be multiple banners getting created that target the same content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bKG6iv0C48XVfqCxqJAUIA.png" /><figcaption>Entity diagram (left) shows the schema. B3 can be added as it B1 is not yet live.</figcaption></figure><p><strong>Business ask</strong></p><blockquote>There can be only one <strong>LIVE</strong> banner for a given content at any given moment. A live banner is one where both its “banner” and “ingestion” status are set to “LIVE”.<br>This restriction is to avoid multiple banners for the same content that may trigger an opportunity loss. However, there’s none on draft banners or other statuses.</blockquote><p>So, in the figure above, B4 can be added but B5 can’t be added lest there’d be two banners (B1 and B5) for the same content C1 which is a problem.</p><h4>How would you solve this?</h4><p>As always, there are multiple approaches with trade-offs here. You could:</p><ul><li>Option I Do a <strong>“check before set”</strong> in your API i.e before writing a new banner, check if there exists any live banner for the same content. If it does, discard it, else allow it.</li></ul><p>Pros:<strong> </strong>Easy to implement, leaning towards optimistic locking. No schema changes or new dependencies.</p><p>Cons: Functionally <em>incorrect</em>. A race condition exists where two banners might be simultaneously written to at the same instant. This might be tolerable for few systems, but for us, it is a no go.</p><ul><li>Option II <strong>Query a central coordination service</strong> for a lock against a banner on a content. If a new banner does not find any “locked” banners for its content, we give the lock to the new banner, else we reject it.</li></ul><p>Pros:<strong> </strong>Functionally correct as long as the coordinator is up. The coordinator can scale independently and can also serve other lock requests.</p><p>Cons: State leaks outside of central source of truth; service changes to talk to coordinator; new dependency and infra maintenance of coordinator setup (if from scratch). This feels like an overkill for our problem.</p><p>It seems that the most non invasive and intuituve solution is to somehow, <strong>rely on database unique index</strong>. We could create a vanilla unique index on the “banner” for the the three columns meaning that no duplicate record can be inserted if all its three columns have duplicate values in an another record.</p><pre>alter table banner add constraint <strong>unique_live_banner_idx</strong> UNIQUE      <br>          (content_id, banner_status, ingestion_status);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8hU5vEMHxwUes35ccZxVUA.png" /><figcaption>Existing index covering the three main columns</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mA9yGwLVclqareqDM2NZ4g.png" /><figcaption>Assuming an ORM like Gorm (Go) or Hibernate (Java), we could define the index mapping as above</figcaption></figure><p><strong>However, this is not correct.</strong> Recall, the business requirements. Apart from LIVE, all other combinations of banners maybe allowed for a banner.</p><p>Let’s write a simple <a href="https://gist.github.com/isopropylcyanide/039d9ab4a746672d6fef0424448c7768">Go program</a> (could be anything, really) that uses <a href="https://github.com/go-gorm/gorm">Gorm</a> (could be anything, again) to talk to MySQL to demonstrate this behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2nFj3yTUz-44XuU85SaMjg.png" /><figcaption>Sample test scenarios I-V. Note that scenario IV should succeed as it is in DRAFT, but V should fail.</figcaption></figure><p>If the queries to our system were issued in the above order, our current index will fail query V (as expected), but also query IV, which is <strong>incorrect</strong>. Two draft banners may exist for the same content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L8wYVbWBxYdQDYsjg_9laA.png" /><figcaption>Scenario IV is also failing as the constraint C2-DRAFT-DRAFT appears violated. Not good.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h7DstPhzCZ9Dy6zXdHz90A.png" /><figcaption>State of the system. 3/4 records inserted. Record for Scenario IV should have been here but isn’t.</figcaption></figure><h3>So, what’s the trick ?</h3><p>It appears that we are close and heading in the right direction <em>(database unique integrity is what we are after)</em>. However, we need a way to filter the unique index when the value is LIVE.</p><p>There’s <strong>no</strong> standard way of doing this. So, we introduce a new <strong>nullable</strong> bitfield / bool in the entity schema and set its <strong>default value as null</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fuGwqcjEe9wlI8Bg1KWB6A.png" /><figcaption>Notice the <a href="https://gist.github.com/isopropylcyanide/bf56c4bd583412b599d63c9e2bb4043a">patch</a>. We added a new field (and to the existing index) and set it default value as null</figcaption></figure><p>We then add it to the existing index so it now covers the new column.</p><pre>alter table banner add constraint unique_live_banner_idx UNIQUE      <br>    (content_id, banner_status, ingestion_status, <strong>protected_status</strong>);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qkLQwIADMjj7VS2S9Qc1_A.png" /></figure><p>On its own, MySQL interprets null as <em>incomparable</em> which means the below combination of values is now <strong>“non unique” </strong>and can be inserted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IG-71NeiwFacphjJT6tnMQ.png" /><figcaption>Scenario IV can now pass. Notice even though everything is identical null is incomparable and hence unique</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8udD6UjzROjtVkEYxI7_Mw.png" /><figcaption>New state of the database on running through Scenarios I — V.</figcaption></figure><p>However, there’s a problem. Scenario V also passes for free now while it shouldn’t. We never told the system to deal with a LIVE banner specially.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FP-ne8hjzX7s4lT2uysMUg.png" /><figcaption>Like key &quot;C2_DRAFT_DRAFT_NULL” (scenario IV), key “C1_LIVE_LIVE_NULL” (scenario V) is non unique too</figcaption></figure><p>This is where, the benefit of using the protected_status comes in. Whenever, our business logic warrants us to be unique, we could simply set its value to <strong>1 (or any non null value).</strong></p><p>Doing this guarantees that no two duplicate banners can be inserted because they’ll now be <strong>“non unique” </strong>and fail the constraint</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5kN3GZsWyzkBVmhqoHywQ.png" /><figcaption>Since LIVE banners now carry same non null protected status, only one of them can be inserted.</figcaption></figure><h4>How and where do I set the non null value prior to create / update ?</h4><p>This depends on the methodology used to communicate with the database. ORM frameworks such as the like of Gorm for Go, Hibernate for Java make it extremely easy to create <a href="https://gorm.io/docs/hooks.html">hooks</a> / <a href="https://docs.jboss.org/hibernate/orm/3.5/api/org/hibernate/Interceptor.html">interceptors</a> that can modify the entity prior to <a href="https://www.yourdictionary.com/upsert"><em>upsert</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8I-qyO-7EwH02PTEZMEfDQ.png" /><figcaption>The two-line “BeforeCreate” hook that fixes our use case. When we detect a live banner, we mark it protected</figcaption></figure><p>If you are not using an ORM (<a href="https://martinfowler.com/bliki/OrmHate.html"><em>and I don’t blame you</em></a><em>)</em>, then you could plug the logic in manually. The only thing you’d need to guard is that the event of multiple paths that may write to “banner” entity, all of them should go through the hook.<strong> Even a single path that doesn’t follow the logic will breach the invariant</strong> that would need to be reconciled separately.</p><p>The recommended way, therefore, is to create a dedicated well documented hook. Using this, let’s modify our <a href="https://gist.github.com/isopropylcyanide/421d0e89aec91e16647646d1c7e59532">program</a> to use the above hooks.</p><p>The final patch to “fix” the unique constraint , now looks like the following.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52ec4b47257c6688ca54b77a21b51fa1/href">https://medium.com/media/52ec4b47257c6688ca54b77a21b51fa1/href</a></iframe><h4>But, I’m not using MySQL. Nulls are not unique in my database</h4><p>Fair point. That includes most databases. The fix is pretty easy. Instead of null, can you find any other “now unique” identifier for your record ?</p><p>Yes, that’s right. <strong><em>Use the primary key</em></strong>. In the default block, set the protected_status: primary_key and only modify in the hook.</p><h3>That’s a wrap.</h3><p>So, that was the trick. Let’s summarise what we did.</p><ul><li>We identified that a unique constraint was the preferred approach here but had to be tweaked slightly.</li><li>We introduced a new field protected_status in the schema that can be nullable. To avoid wastage, we chose a nullable tiny bit / boolean field.</li><li>We then modified our index to cover the new column. In theory, if the original index exists, this “update” should not be costly.</li><li>We then set the right default values for the protected_status and also create a hook to undo the “null uniqueness” if our logic needs it to be.</li></ul><p>Let’s now think of the pros and cons of this approach. This neat trick has its place but cannot be termed as a panacea.</p><p>Pros:</p><ul><li>Field protected_status can scale to multiple indexes and columns even when the business logic is updated. One source of truth, in the hook.</li><li><strong>Zero race conditions</strong> and works without the need of any coordinator. If no duplicates can be tolerated, this might be an excellent option.</li><li>If using an ORM, the implementation is trivial.</li></ul><p>Cons:</p><ul><li>Entity schema edits may not always be possible.</li><li>If not using an ORM, there’s a chance of going around the hook.</li><li>New index is not for free and occupies some space, although paltry.</li></ul><p>Thank you for reading. Would appreciate any feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9423c315cd2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elegant Patterns to execute work concurrently using Completion Service [Java]]]></title>
            <link>https://aman-garg.medium.com/elegant-patterns-to-execute-work-concurrently-using-completion-service-java-1bee26f186b9?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bee26f186b9</guid>
            <category><![CDATA[concurrent-programming]]></category>
            <category><![CDATA[thread-pool]]></category>
            <category><![CDATA[java-thread]]></category>
            <category><![CDATA[threadpoolexecutor]]></category>
            <category><![CDATA[executorservice]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Sun, 14 Mar 2021 12:16:25 GMT</pubDate>
            <atom:updated>2021-04-12T21:15:01.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d0D3kMPG-TOHr2g-.png" /><figcaption>Courtesy: <a href="https://www.baeldung.com/">https://www.baeldung.com/</a></figcaption></figure><h3>Use Case Analysis</h3><p>Assume you have a use case, where there’s an input of N numbers from the user and you need to fetch the results of an<em> expensive mathematical function</em><em> f(x) </em>computed on those numbers and return any first K that finish first.</p><p>Further, the logic of computing the result of the expensive function is obtained through HTTP interactions with a remote service, say Math Service</p><p>The <strong>objective</strong> is to solve this with a minimal latency overhead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-8TmOjmih3cYBPR__yYOg.png" /><figcaption>High Level Use Case</figcaption></figure><p>How do we go on about solving this? Let’s start by asking questions.</p><h4>Characterising our work load</h4><p>It’s important to analyse the work load we are dealing with it before we come up with any set of solutions.</p><ul><li>Work is I/O Bound and not compute bound.</li><li>Input is going to be a list of numbers, can be modelled as Iterable&lt;Work&gt;</li><li>We’re most likely going to use HTTP for remote service communication.</li><li>Out of order execution / completion of results is perfectly acceptable.</li><li>A limited set of results K ≤ N is required.</li></ul><h4>Exploiting the domain service.</h4><p>Now is a good time to know more about the domain. We need to have an engaging discussion with the maintainers of Math Service and stakeholders.</p><ul><li>Is there any kind of input that is slow to compute on?</li><li>Can we pre filter the slower inputs such that the input size N = K</li><li>Is the Math Service reliable? If <em>yes</em>, how so? If <em>no</em>, why not.</li><li>Does Math Service degrade / rate limit requests after a certain input?</li><li>Do we know what kind of inputs the user is going to give so that either the Math Service or our service can precompute / cache the results?</li></ul><p>Let’s assume the answer is No to all of the above. Any Yes on the above is bound to make the problem easier and add scope for more optimisations.</p><h4>Building Blocks</h4><ul><li>We know the workload can be parallelised. Hence, multiple threads.</li><li>We know we need some implementation of a <strong>Thread Pool Executor</strong> to be able to manage the threads effectively. Let’s start with a fixed thread pool.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*504Nk8R2qOCiJSyiO-zjGA.png" /><figcaption>Pseudocode. Submit work in a fixed executor. Not the most efficient way.</figcaption></figure><p>This works. However, there are some obvious <strong>problems</strong>:</p><ul><li>We need to block for the results of each call in an order. This means that if the user wanted any 10 results on a batch of 20 and if the first request was the slowest, the time to compute the entire batch is affected.</li><li>We’re not leveraging the fact that we do not need to get the results in an order. Any order will do.</li></ul><p>Is there an implementation of an Executor that, to which, when the tasks are supplied, the results are placed on a queue in the <strong>order they complete </strong>and <strong>not</strong> in the <strong>order they were submitted</strong>?</p><p>Is there an implementation that places results on a queue as and when the tasks are completed from which we can take the first K and be done with?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BjqWLnd6L24N5YSL-6XsFQ.png" /><figcaption>Ideal Executor Service queuing “completed” tasks in “completion” order and not “submit” order</figcaption></figure><h4>Executor Completion Service</h4><p><a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorCompletionService.html">Executor Completion Service</a> is exactly that.</p><p>Instead of trying to find out which task has completed (to get the results), it just asks the CompletionService instance to return the results as they become available.</p><p>This is what we want. Let’s code out the building blocks.</p><h4>Math Service:</h4><p>We don’t have to worry too much about the underlying details here. To simulate the calls on a remote interaction, let’s add a random delay on inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BXa3kWoCb4fFxviC1vag3w.png" /><figcaption>Our interface for the Math Service. Simple. Beautiful.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7MMyCCrtknDK6oKKM0Nxg.png" /><figcaption>We just return the same number after a random delay. Simulates network delay in a real call.</figcaption></figure><h4>Work Executor</h4><p>This is the workhorse where the actual work is done. Here, we have a dependency injection of the Math Service and expose an API to the method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/079ed08c220277f7b5a17f7b83fe3a0f/href">https://medium.com/media/079ed08c220277f7b5a17f7b83fe3a0f/href</a></iframe><h4>Driver Program</h4><ul><li>Initialise the dependency and the dependent in the driver.</li><li>Output the results on the given input.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNucyMtjg_-5rj9zCBdcnQ.png" /><figcaption>Our driver. We want first 3 in the given list.</figcaption></figure><p>Let’s look at the <strong>outputs</strong>. Notice the first tasks that are submitted <strong>in order</strong> are <strong>not</strong> the first ones to complete. This is because there’s a random delay to each and we process the first available results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_-vr9jTVt8B3FhuVuGTYg.png" /><figcaption>What we wanted to achieve. Return top K that finish first.</figcaption></figure><p>If we were to run this again, we might notice different results as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFPlTlJu2nW9tjlSaN7pSA.png" /></figure><h4>Generalising</h4><p>This is good and all. What if now there’s a similar requirement, but to User Service instead of Math Service. Say, we want first K payment methodsof the user and display results as and whey it arrives <strong>on the UI<em> </em></strong><em>(Checkout page)</em></p><p>Do you notice a <strong>pattern</strong> here? We’re breaking work into chunks and submitting it to an executor and looping to find the first <strong>K</strong>. We should not be worried about the low level take / submit APIs of the executor service.</p><blockquote>Indeed, there is.</blockquote><ul><li>Define an iterable / stream of input on which the work would be split and submitted into the executor.</li><li>Define a mapper function that creates a task for each element in the iterable. This specifies the actual work.</li><li>Specifying the count of results that are expected. Not all methods require completion of all sub events. This would be the parameter <strong>K</strong></li><li>Implement a collector for collecting the intermediate results and returning the final response. It could be a <em>list, set </em>or what have you.</li></ul><p>Let’s define an <strong>interface</strong> that the whole codebase can benefit from.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4591b7fae0441770ae19e91512373f19/href">https://medium.com/media/4591b7fae0441770ae19e91512373f19/href</a></iframe><p>With the above interface, we can model our previous solution to the Math Service as elegantly as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4d169278aec80fcac0a1eca2340f6f3/href">https://medium.com/media/b4d169278aec80fcac0a1eca2340f6f3/href</a></iframe><p>And that’s it. Using the splitJoin abstraction powered by the Concurrent Work Executor our application programs can benefit from not knowing the low level details. Here’s the output again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LG4tPqyC3Awfu5V_Qv5vsg.png" /><figcaption>We get a similar output. But now we have a very good abstraction.</figcaption></figure><p>Fetching top K Payment methods now becomes easy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ae7e73471df4b7a58a3ff303f92c29c/href">https://medium.com/media/6ae7e73471df4b7a58a3ff303f92c29c/href</a></iframe><h4>Interface implementation: Out of Order Concurrent Work Executor</h4><p>The last piece of the puzzle is to implement the concurrent work executor interface. We can have multiple implementations. One that can process results <em>out of order</em>, one that processes in order etc.</p><ul><li>This executor submits its tasks into a newCachedThreadPool This can be parameterised further. Some might require a fixed thread pool.</li><li>Shuts down the pool immediately to avoid more work.</li><li>Processes non null events of completion results as they arrive using a Completion service (a blocking completion queue)</li><li>It waits for as many tasks that need to be completed.</li><li>Throws checked exceptions when either a future#get fails</li></ul><blockquote>What if you don’t want the first K but the whole results? <br>You simply set K to be the size of the entire input.</blockquote><p>This should be straightforward to implement. I’ve added relevant comments wherever necessary. Feel free to skip this or implement on your own.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e745ec2779fecf86b80d5d30eba72d2d/href">https://medium.com/media/e745ec2779fecf86b80d5d30eba72d2d/href</a></iframe><h3>Conclusion</h3><ul><li>Identify the work load</li><li>Try to exploit the domain wherever possible.</li><li>Choose the right executor service implementation and the right executor.</li><li>Build a meaningful abstraction around it.</li><li>Document the interfaces and how they provide value.</li></ul><blockquote>The full gist along with unit tests can be seen <a href="https://gist.github.com/isopropylcyanide/b345af36179ee522d4b2152a502e174c">here</a>. <br>That’s a wrap. Hope this piece added value. Open to constructive feedback :)</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bee26f186b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decluttering the Uber Tech Blog]]></title>
            <link>https://aman-garg.medium.com/decluttering-the-uber-tech-blog-15de68004162?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/15de68004162</guid>
            <category><![CDATA[decluttering]]></category>
            <category><![CDATA[enhanced-user-interface]]></category>
            <category><![CDATA[uber-engineering-blog]]></category>
            <category><![CDATA[uber]]></category>
            <category><![CDATA[tech-blog]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Fri, 12 Mar 2021 18:20:52 GMT</pubDate>
            <atom:updated>2021-05-03T09:03:24.594Z</atom:updated>
            <content:encoded><![CDATA[<p>Uber engineering blogs are a goldmine that provide solid insights into their architectures. There’s no doubt. Like all big tech firms, they take great pride in the quality of their content.</p><p>However, <strong>there’s something off</strong> about how the page is structured. Too much <em>focus on sidebars</em>, <em>big sticky headers</em> and <em>misaligned content.</em></p><p>I find myself spending so much time perusing these blogs that I have gotten insanely triggered by the small nits that violate my user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2jtKHjGHUflZEb9npHbBw.png" /><figcaption>Default Mode</figcaption></figure><p>I developed an extension to help declutter this. It essentially addresses these problems in order. The results are self explanatory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wEDAy2Jyhw8aRbaR-s57Q.png" /><figcaption>Focussed Mode. On the content. Nowhere else. Neat margins.</figcaption></figure><p>The code for this is <a href="https://github.com/isopropylcyanide/declutter-uber-tech-blog">open source</a>. In fact it’s one of the smallest codebases I’ve ever pushed. Now there’s plenty of ways in which this could be made more useful and configurable, but for now, I’m KISS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c7b1104b00ec4bc1518d4a158f3e30b/href">https://medium.com/media/2c7b1104b00ec4bc1518d4a158f3e30b/href</a></iframe><p>If you, like me are annoyed by this, simply install the extension and forget about it. This extension requires basically zero permissions.</p><p>This is available on the <a href="https://chrome.google.com/webstore/detail/declutter-uber-tech-blog/pdpkcnndnkfadmfohggcloaihobejlci">Chrome Web Store</a> 😄</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqZoRog6vodMa0ai8ZwIlQ.png" /></figure><blockquote>Whenever the page loads, you are presented with a clean, decluttered view of the blog post.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15de68004162" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dissecting why some Pull Request merges get delayed]]></title>
            <link>https://levelup.gitconnected.com/dissecting-why-some-pull-request-merges-get-delayed-739432b7ebba?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/739432b7ebba</guid>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[developer-productivity]]></category>
            <category><![CDATA[master-branch]]></category>
            <category><![CDATA[pull-request]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Fri, 12 Mar 2021 17:40:13 GMT</pubDate>
            <atom:updated>2021-03-15T00:10:51.503Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dissecting why Pull Request merges get delayed</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jPteh6suNurcBP14.png" /><figcaption>Image Courtesy: Atlassian Blogs</figcaption></figure><p><strong>Pull request</strong> creation, review, comment redressal, re-review and <a href="https://www.urbandictionary.com/define.php?term=LGTM">LGTM</a> process carve out a significant chunk of a developer’s bandwidth. There’s no denying the importance of this process. However, this can take time.<br>Let’s discuss how we can reduce the turnaround for the entire process.</p><p>For this article, let <strong><em>developer</em></strong> be the a person requesting a review and let <strong><em>peer / reviewer</em></strong> role correspond to the peer reviewing the developers’ pull request (although their roles may interchange on a different PR)</p><h3>Pull Request Lifecycle</h3><p>Broadly, a PR approval lifecycle can be summarised as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qfzZONl8oGlYf-DeQ3ZPkQ.png" /></figure><p>Through experience, we gather that the rate determining step of a PR getting approved is <strong><em>not determined by the time to get it reviewed</em></strong>, but the <strong><em>time spent in crafting the PR.</em></strong></p><blockquote>Onus is on the <strong>developer</strong> and not the reviewer to speed the entire process.</blockquote><p>And that’s the TL;DR version of the article.</p><h3>Developer Onus</h3><p>Presented below are the guidelines that devs may follow to in order to speed up the approval process.</p><h4>Specify the ETR: Estimated Time to Review .</h4><p>ETR or the <strong>estimated time to review </strong>needs to be called out when requesting a review on the channel (Yes, I just made the acronym up)</p><ul><li>This helps the reviewer allocate their time better. If <em>ETR is 2.5 hours</em> and the peers have no bandwidth, they can call out the delay.</li><li>It shows thought and compassion towards and a step towards respecting the other person’s time. If ETR turns out to be huge, dev’s know they have to break the PRs for it is not fair to assume unlimited peer bandwidth.</li></ul><p>Here’s a sample template.</p><blockquote>Hey Peer P, requesting review on PR &lt;link&gt; (brief description). <strong>ETR: 40 mins</strong></blockquote><p>We’ve followed this at work and this helps reviewers plan better.</p><h4>Respecting the codebase.</h4><p>Always respect the codebase you’re working on. Try to spend some time aligning your changes with its true philosophy.</p><ul><li>It should look like a single coherent codebase. Model classes and interfaces to follow existing conventions.</li><li>Pay particular attention to <a href="https://www.smashingmagazine.com/2012/10/why-coding-style-matters/">formatting</a>. This should be consistent across the codebase. <a href="https://aman-garg.medium.com/nitpicks-from-a-triggered-senior-developer-in-a-code-review-8b914211a9d0">Small nit picks</a> crop up into unmanageable, inconsistent bases.</li><li>Consider importing a team-wide formatter in your IDE or review the code checklist. If you are using IntelliJ, <a href="https://plugins.jetbrains.com/plugin/7642-save-actions">Save Actions</a> plugin is your friend.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FhxVgwN5HlQ9AGHAhXf1fw.png" /><figcaption>This plugin automatically formats the current working file when you save</figcaption></figure><blockquote>If you respect the codebase in all forms, it leads to a view where it feels as if a single person wrote the entire thing. Such a codebase will be easy to review.</blockquote><h4>Keep it small.</h4><p>As the saying goes,</p><blockquote>100 <strong>lines</strong> changed = <strong>2</strong> issues, <strong>5</strong> review comments and open discussions<br>100 <strong>files</strong> changed = LGTM</blockquote><p>Huge PR’s are a pain to review and manage. <strong>They never help anyone.</strong> Always split a PR into smaller PR’s if you feel it’s getting longer. How do we define a huge PR?</p><ul><li><strong>Number of files changed</strong><br>Usually, it’s a good metric to gauge how big a PR would be, however it doesn’t hold true for refactor PR’s. At worst, keep the number of files changed under <strong>50</strong> (hard limit). Anything over than that and be prepared to split.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyFubOZIkhg36swy8TD78A.png" /></figure><ul><li><strong>Number of additions / deletions<br></strong>This is a better metric when it comes to gauging a PR. Usually this is the number of lines a reviewer’s eyes will span across the course of a review.<br>It takes into account <em>refactors</em> (usually, equal deletions and additions), <em>features</em> (more additions) and <em>clean up</em> (more deletions).</li></ul><blockquote>At worst, keep the number of additions and deletions under 1500. Anything above and you’d need to split. <strong>Or your reviewer will ask you to do anyway.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Q4bdDMwk4byxqVQXKka5w.png" /><figcaption>Last one is a BIG <a href="https://www.youtube.com/watch?v=31g0YE61PLQ">NO</a></figcaption></figure><h4>Due Diligence — Self Review.</h4><p>A self-review on your PR is very important. Please, <strong>pretty please</strong>, go through your own PR before raising it. I can’t stress this enough. A lot of turnaround can be reduced right here. If you look at your own PR in the eyes of a reviewer, you’ll see better.</p><ul><li>You’ll see <strong>commented blocks </strong>which need not be checked in</li><li>You’ll see <strong>errors</strong> induced due to copy-paste like incorrectly named method</li><li>You’ll see <strong>todos</strong> that have been long done and are now irrelevant.</li><li>You’ll see that you ended up adding<strong> redundant information / tests.</strong></li><li>You’ll see <strong>local dev configs </strong>being incorrectly checked in to production.</li><li>You’ll see visible<strong> formatting inconsistencies.</strong></li><li>You’ll see <strong>redundant logs, system outs, debug info </strong>being checked in.</li></ul><p>As a developer, It’s also important to add self review comments where applicable and providing justification to the not so obvious changes. These include (but not limited to)</p><ul><li>Changing the value of configs / magic numbers / literals.</li><li>Changing application structure blocks or new dependencies.</li><li>Changing method signatures of unrelated classes.</li><li>Tweaking things outside of the scope of the PR within the same PR.</li><li>Stuff that is bound to get your reviewer dumfounded.</li></ul><blockquote>Once you get a PR review, you can compare it with your own assessment in the self review. With multiple iterations, you’ll close on the expected review comments and raise optimal PRs which will lead to faster merges.</blockquote><h4>Test Cases that improve quality and not coverage.</h4><p>Test cases help garner confidence. When you write unit tests that deal with the following, pay close attention to their innards.</p><ul><li>Explicit concurrent features (using <em>latch, barriers, phasers for units</em>)</li><li>Deal with non thread safe classes (like <em>Calendar, SimpleDateFormat, Date</em>)</li><li>Dealing with code that mutates objects.</li><li>Dealing with static resources (like Embedded db, containers)</li><li>Dealing with unit tests that rely on an implicit order (<em>unfortunate</em>)</li></ul><p>It is quintessential to<strong> stress test your unit tests until failure.</strong> What happens otherwise is that the test outputs remain indeterminate. Your test might run on local, but fail on your CI pipeline, or vice-versa.<em> Many aren’t aware that we can run a unit test until failure.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wm1vplWMj4WE0-jQcsLUdQ.png" /><figcaption>Run tests until failure to catch hard to review errors.</figcaption></figure><p>As a rule of thumb, for each non trivial test case you write, ensure that the output is consistent across multiple invocations.</p><blockquote>Since these are extremely hard to detect in code reviews, the idea is to adopt a proactive approach and fix them beforehand. <strong>Do not wait for the reviewer</strong> to point this out, which will ultimately delay the PR merge.</blockquote><h4>Request Reviews on team channel.</h4><p>All PR reviews should be requested on the common team channel (slack/teams what have you) and not through DMs. <strong>Why, you ask?</strong></p><ul><li>Team leads are kept abreast of the changes being requested for review.</li><li>Other peers can pitch in when they see that they can contribute to the review besides the original requested peer.</li><li>Bumping on the channel is different from bumping on DM. Devs can call out if they are blocked on the review.</li><li>If a particular peer is clogged with multiple PRs to review, the leads can take note and assign it to others (incl. themselves) instead.</li></ul><h4>Label the PR.</h4><p>A label helps to group and tag related PRs. We can create a wealth of appropriate tags to better communicate the scope of the PR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOgBnESjQVLUiAjLuivNTw.png" /></figure><p><strong>How do labels help?</strong></p><ul><li>Labels like “dont merge”<em> </em>communicate intent. This means a less stringent review.</li><li>Labels like<em> </em><em>“bug fix”</em> take higher priority than “tooling”.</li><li>They are multi coloured and cool.</li><li>Labels can be used to group related changes &amp; track lineage.</li></ul><h4>Writing the PR Description.</h4><p>This is one place that can be optimized to a large extent.</p><ul><li>This is where a developer communicates their intent, their design and the decisions made to the reviewer. Refer a <a href="https://github.com/flipkart-incubator/zjsonpatch/pull/120">sample</a>.</li><li>This is the first place a reviewer looks at before starting the review. The more engaging, clear and concise the description, the better it is.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cFU0cJN5p1DJM0kluucBeA.png" /></figure><p>Capturing a<strong> general summary</strong> prepares the reviewer accordingly.</p><ul><li>If it’s a work in progress PR, the reviewer can get back to her tasks until the PR moves back to Ready state.</li><li>If it’s a feature PR, extensive review needs to be carried out.</li><li>If it’s a bug fix PR, the new test cases need to be run again on the reviewer’s machine</li><li>If the impact is high, the review priority goes <strong>high</strong>.</li></ul><p><strong>Description<br></strong>Describe the changes that are present in the PR. Just the what’s. Not the why’s. Keep it short and sweet.</p><blockquote>E.g This PR introduces new methods in class &lt;class&gt; for &lt;reason&gt;<br>E.g This PR fixes bug in class &lt;class&gt; where &lt;reason&gt;</blockquote><p><strong>Reviewer Notes<br></strong>Talk to your reviewer here. This helps is establishing the <strong>context</strong> before the review process starts.</p><ul><li>Does the build break because the PR is split into small PR’s but the overall build succeeds?</li><li>Is the design document not required because it’s a trivial change?</li></ul><blockquote>E.g The coverage dipped because of &lt;reason&gt;<br>E.g More than 60 files have changed because of &lt;reason&gt;</blockquote><p><strong>Motivation and Context<br></strong>This is where we spit out the why’s. PR’s solve a particular problem. Before expending energy on a PR, <strong>we should be clear why we’re doing this</strong>. A lot of times, while writing this block, the developer will realize that a subset of changes are really not required and are an overkill, or that they belong in a separate PR.</p><ul><li>Why was this change needed? What was the problem with the old one?</li><li>How does this change solve the problem?</li><li>Was the design change (if any) reviewed internally or externally?</li></ul><blockquote>Help your reviewer with these and you are set for a faster review</blockquote><p><strong>How was this tested</strong><br>Arguably, the most important section. It helps reviewer garner confidence that the developer has done the due diligence. And that the code compiles none the less.</p><ul><li>Specify if<strong> unit tests</strong> were added. <br>If not, there should be a very good reason for the same.</li><li>Specify if <strong>&gt; 90% coverage</strong> was achieved on new code <br>This should be the expectation anyway. If no, specify why.</li><li>Is the code<strong> running in an integration environment</strong> already?<br>If yes, share stack details. If no, specify how has the integration been done.</li><li>Have the<strong> integration tests been written </strong>and run against existing suite?<br>If yes, share success screenshots / build link to eliminate regressions.</li></ul><p><strong>Related Links / Notes<br></strong>Add anything that the reviewer can use to gain extra context about the PR</p><ul><li>Add design documents, contracts, solution docs, meeting notes etc here.</li><li>Have any NFR runs been executed on this PR? Add them.</li><li>Are you adding any new APIs? Share the curls / swagger specs.</li></ul><h3>What happens post the first round of review?</h3><p>The reviewer provided a feedback. What do you do now?</p><h4>Ack, ack and ack.</h4><ul><li>Every feedback you receive is a result of the <strong>effort</strong> that has been put by the reviewer to understand your code better (taking out valuable time). As such it becomes very important to acknowledge that their comment has been received and where it will/will not be worked upon.</li><li>This gives the reviewer the notion that the feedback has been taken positively. Otherwise, it carries the feeling of a task that has been assigned by the reviewer. Also, this is not required for nit comments.</li><li>A lot of times a developer’s views may not be aligned with the reviewer. It’s better to be courteous and support your views using facts rather than holding it in or dissing the comment altogether. Start a healthy discussion.</li></ul><p>Not doing this also increases merge TAT (the elephant in the room). How?</p><blockquote>Let’s say a reviewer adds few critical comments, say 10. The dev has worked on 6 and asked for a re-review. The remaining points are either not worth fixing in the eyes of the developer or out of scope. But the same needs to be called out.</blockquote><blockquote>We cannot expect the reviewer to ignore the remaining 4 comments just because it’s the second round of review already. They are still significant in the eyes of the reviewer. If the dev disagrees, start a discussion. But always ack. <br>Also, a comment 📝 is more thoughtful than a bland thumbs up 👍</blockquote><p>That’s it for the <strong>developer</strong> onus. We covered a lot of points. To summarise</p><pre>::: Specify ETR<br>::: Respect the codebase<br>::: Keep it small<br>::: Due Diligence: Self Review<br>::: Test Cases that improve quality and not coverage<br>::: Requesting Reviews on the channel<br>::: Labelling PRs<br>::: Writing a good PR Description<br>::: Ack after review on the comments.</pre><p>But hold on, we can’t drop off without talking about the reviewer onus. Surely they’d be having some responsibilities, right?</p><h3>Reviewer Onus</h3><p>A reviewer has an important role to play as well. Particularly because peers are both reviewers and developers at the same time.</p><h4>Pre review guidelines</h4><ul><li>Call out if you don’t have sufficient bandwidth to review after seeing the ETR. Dev can choose to wait it out or seek another peer.</li><li>Share rough estimates with the dev for the review completion. This should be on the channel, again. This helps the developer plan better.</li><li>If PR exceeds the thresholds specified above, feel free to reject the PR or request the dev to split into multiple PR’s</li><li>If PR code coverage drops below the threshold, it should ideally be rejected unless the developer provides a valid reasoning</li></ul><h4>Review Guidelines</h4><ul><li>Spend some time going through the PR description and the context that has been established by the developer. Value the effort.</li><li>For feature, design or enhancement PR’s, it’s better to check out the branch in your local IDE and review from there onwards.</li><li>Provide constructive feedback and avoid critical assessment of your own peers. Always supplement your arguments with <strong>facts</strong>.</li><li>Try to pitch in and share your own wisdom as to how you could have solved the problem in a different, seemingly better way.</li><li>Appreciate wherever applicable. If you see neat PR’s or PR’s solving a hard problem, do make a note of it.</li></ul><p>That’s a wrap for today. I didn’t intend to cover points in such depth, especially when the article focusses on reducing time.</p><p>But I’m glad I could touch upon these nuances.<em> Hope this piece added value.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=739432b7ebba" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/dissecting-why-some-pull-request-merges-get-delayed-739432b7ebba">Dissecting why some Pull Request merges get delayed</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nitpicks from a triggered senior developer in a code review]]></title>
            <link>https://levelup.gitconnected.com/nitpicks-from-a-triggered-senior-developer-in-a-code-review-8b914211a9d0?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b914211a9d0</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[pull-request]]></category>
            <category><![CDATA[nitpicking]]></category>
            <category><![CDATA[senior-developer]]></category>
            <category><![CDATA[code-review]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Fri, 05 Mar 2021 21:01:26 GMT</pubDate>
            <atom:updated>2021-03-07T11:57:51.613Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SbkgG7gIerSGVC5a.jpg" /><figcaption>Courtesy: <a href="https://bit.ly/38cWpbZ">https://bit.ly/38cWpbZ</a></figcaption></figure><p>We’ve all been there. For our pull requests (PRs), there’s this.</p><p><strong>Hoping</strong> they safely make it through the barbaric eyes of a senior developer. <br><strong>Hoping</strong> that we do not get nit picked on some “code” aspects by a pedant.<br><strong>Hoping</strong> that we are only judged by the functional completeness of the task. <strong>Hoping</strong> that our 100 line diff will not garner more than a 1–2 comments.</p><p>The wishful thinking finally ends with this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*fC_ESj1WC9MvN7vjt-jSJg.png" /><figcaption>What?</figcaption></figure><h3>What gives?</h3><p>In this article, we try to focus on small things that trigger a senior developer during a Code review. While the examples are more geared towards Java, I try to be language agnostic wherever possible. We’ll not focus on the language semantics or <a href="https://rules.sonarsource.com/java/type/Code%20Smell">sonar smells</a>, rather the soft aspects of readability and maintainability.</p><h3>#1:Documenting the public API’s</h3><p>Small interface. Big interface. Interface with only one implementation. Abstract class used only in one flow, two flows. etc. <strong>It doesn’t matter.</strong></p><p><em>Always, document public APIs.</em> This means, documenting the behaviours you expose in the interface. Documenting the methods designed for @Override in abstract classes.</p><p><strong>Why?</strong></p><p>Without this, there’s no way of knowing what the implementation is supposed to do and why it was designed that way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*BqnuXW1gjF30IX_VUub08w.png" /><figcaption>Take this interface for example</figcaption></figure><ul><li>Just by reading this interface, I assume it’s a pure function since there’s no declared exception in the throws clause. Does it mean that the clients should handle / swallow exception if it arises? What kind of exceptions.</li><li>What should the implementations look like? What all can they assume / not assume? What all is undefined behaviour?</li><li>enrich() could definitely be a function that has a side effect that blows. Clients that solely rely on the method signature will be going places.</li></ul><blockquote>Instead, disambiguate this like below.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*yqfkPS9PE1MFQsf1psHYGw.png" /><figcaption>Clear, concise, documented behaviour.</figcaption></figure><blockquote>Public APIs are used more often than not. Interfaces will be extended by your peers in the future. Don’t do this to them. Don’t let the behaviour be a mystery.</blockquote><h3>#2: Ordering arguments in method signature</h3><p>As long as my method receives all the arguments it needs to do its work, I’m good right? <em>Not always.</em></p><p>We’re not talking about passing a small number of arguments in a method invocation. That’s not news. We’re talking about the <strong>order</strong>.</p><p>Take the below method for example. This tries to create a map on a collection using a key obtained from the keyFunction after running it through a filterFunction</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*0lJ1uEXoCpOrkE1sLORZHQ.png" /><figcaption>Look at the order of the arguments. Do you notice anything?</figcaption></figure><p>The method arguments seem totally random. To obtain a map, a filter function has to be passed first. Why is that?</p><p>A client using this API will have the following workflow <strong>for a null filter.</strong></p><pre>collection c;<br>keyFunction k;</pre><pre>class.identityMap(..? should I pass null? I don&#39;t have a filter...)</pre><p>By now, you would have realised that the method is most intuitively called by passing a collection (source of data), a grouping function (keyFunction) and then the filter(filterFunction)</p><blockquote>So, choose this.</blockquote><p>You first pass the collection and then the <strong>necessary arguments</strong> that operate on your collection. Small things like this improve readability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*PLYotGf3BbcZALnc3qQVIA.png" /><figcaption>The readability stems from the fact that the key function and filter “work” on a collection that is passed first</figcaption></figure><p><strong>Why?</strong></p><ul><li>Following this makes up for a more readable API that is a joy to work with. It’s also easy for the consumers of the API where the call parameters proceed in an intuitive order.</li><li>Violate this and there’s a small cache miss (on the order) that every developer has to bear. Instead the arguments should flow like a stream merges into a river.</li></ul><blockquote>Detecting this subtlety requires experience. To an untrained eye, there’s no difference in the order. To embody this, think about the API usage at the clients.</blockquote><h3>#3 Ordering Annotations By Rarity For Decreased Cognitive Load</h3><p>This by far, is one of my favourite nit picks. I also am amazed how many developers simply ignore this or not realize the gravity of it.</p><p><em>Rarer annotations on a member should be closer than the common ones. </em><br>An annotation is <strong>rare </strong>if it conveys a special meaning to the member it is attached to. If the same annotation is attached to multiple members, it is by definition, less rare.</p><p>Consider an API that requires a <strong>non null</strong> list of <strong>integers</strong> but they have to be <strong>within a range</strong> and the numbers have to be <strong>prime</strong>. In a JSR-303 specification, we would annotate the member like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*8Dl5Bv0l_dp6OqxlzoezVg.png" /><figcaption>This works. But ..?</figcaption></figure><p><strong>Do you see a slight problem?</strong> If yes, congratulations.</p><p>NotNull and Getter are common. But there’s something off about the PrimeNumbersOnly and Range validation. Keeping a number in a range is an easier check than ensuring it is a prime (custom code).</p><p>Hence, PrimeNumbersOnly by definition is a rarer annotation than Range and needs to be closer to the member. NotNull is rarer than a Getter .</p><blockquote>Instead, Look at below,</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*UhE_01fabmgzlmFXTE7p3g.png" /><figcaption>Proper ordering of annotations.</figcaption></figure><p>When I read the above code, my mind automatically switches gears and focusses more on the important annotations that serve the business use case.</p><p><strong>Why?</strong></p><ul><li>It’s a convention you can follow to make sure your codebase reads better to the devs. With practice, you’ll automatically start focussing on the most important annotations.</li><li>Easier to read codebases and detect diffs in a pull request. Your eye automatically prioritises changes in the rarer annotations first.</li></ul><h3>#4 Log exceptions properly. Log assumptions. Log context</h3><p>The violation of this is again commonplace. <em>This is around logging.</em></p><ul><li>Consider logging extra helpful information around the original context that helps pinpoint the issue. This helps the on-call at 3 AM. Things like, what input was passed in the first place (if it was null). This also helps in easier grep-ing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*DGjU_xUMI-tNzLGoF7xWnQ.png" /><figcaption>How am I supposed to know what the “entity type” was for which no resolver was registered?</figcaption></figure><ul><li>Another is around the unintentional use of log.error in which developers fail to pass the stack trace information as a method parameter and pass it in the log string. This blows up later when a log is fired but there’s no stack trace to debug off of.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*tDRbIP6SsBCPws__9K4bBA.png" /><figcaption>This was unintentional. However, a senior developer will catch it.</figcaption></figure><ul><li><strong>Logging assumptions undertaken by a developer.</strong> This helps in debugging misbehaving flows where a certain context that was assumed to be true doesn’t stand anymore. Consider a sessionFactory that was silently initialised to a default value without any loggings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*VJj2L8GUuYNx9qR1KyMtpw.png" /><figcaption>If the factory starts misbehaving on production, would we even know we are using the “default”</figcaption></figure><h3>#5 Adding redundant dependencies</h3><p>I admit, I’m guilty of doing this in the past. Depending on javafx libraries just to be able to have a Pair&lt;K, V&gt; or a Tuple&lt;A, B, C&gt; only to find out later that the javafx classes are not included in OpenJDK and the build fails on CI.</p><p>Don’t do this. <strong>Roll out your own</strong>. How hard is it to write an immutable class that doesn’t add any unnecessary dependencies.</p><pre>@AllArgsConstructor<br>@Getter<br>public static class Pair&lt;K, V&gt;{<br>        private final K key;<br>        private final V value;<br>}</pre><p>A NodeJS equivalent would be to have a dependency on entire <strong>Lodash</strong> library just to be able to do a _.sum(array)</p><p>Unless you are convinced that the dependency offers a significant advantage, don’t bother adding it in. This is a red flag during the review.</p><p>That’s a wrap for this article. The points covered above, though nit picks, are important for a codebase’s health. With experience, you automatically start to favour readability. Hope this piece added value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b914211a9d0" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/nitpicks-from-a-triggered-senior-developer-in-a-code-review-8b914211a9d0">Nitpicks from a triggered senior developer in a code review</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fixing Debezium Connectors when they break on production.]]></title>
            <link>https://levelup.gitconnected.com/fixing-debezium-connectors-when-they-break-on-production-49fb52d6ac4e?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/49fb52d6ac4e</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[mysql-binlog]]></category>
            <category><![CDATA[binlog]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Fri, 05 Feb 2021 20:58:28 GMT</pubDate>
            <atom:updated>2022-05-14T16:48:58.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*fBdWf_7MoEQHxMfU" /></figure><p><a href="http://debezium.io/">Debezium</a> is a wonderful piece of technology. You can easily start a Debezium connector having followed the <a href="https://debezium.io/documentation/reference/install.html">basic guide</a> and lo and behold, <strong>you have events!</strong></p><p>However, problems starts to appear when the environment around the initial happy path configuration changes which then breaks your connectors and the downstream consumers.</p><p>Here we’ll look at the following problems.</p><ul><li>The database binlog rolls over and we were out of sync.</li><li>The entire database host changes along with its binlog,</li><li>You lose events for some reason.</li><li>You want to capture a new huge table in an existing connector</li><li>You introduce sudden schema changes but they are not synced.</li></ul><p>Of course, creating a new connector altogether works, but the idea is to<strong> issue a minimally invasive update to the connector configuration</strong> in order to bring it back to life. A connector may be monitored by its name, in scripts, metric registries etc. <em>You may not want to use a sword when a knife can do the job.</em></p><p>In this article, I try to debunk a few. We’ll also see the interaction of Debezium with the internal Kafka Connect topics.</p><p>As always, refer the official <a href="https://debezium.io/documentation/">Debezium documentation </a>for more information.</p><h3>Problem I: Rolled Over Bin-logs</h3><p>If the connector task is dead for a duration long enough such that the offset (binlog position) stored by Debezium in the internal Kafka Connect’s <a href="https://docs.confluent.io/home/connect/userguide.html#kconnect-internal-topics"><em>offset.storage.topic</em></a> doesn’t exist anymore, the connector <strong>cannot</strong> resume.</p><p>You may try recreating the connector with a different name, restarting tasks, changing server ids but if the offset is gone, it cannot resume.</p><h4><a href="#"><strong>Fix:</strong></a></h4><ul><li>You need to tell the connector to recreate a snapshot.</li><li>Use <strong><em>snapshot.mode=initial.</em></strong></li><li>If you know how long your connector was down, you can issue <strong><em>snapshot.select.overrides</em></strong> on heavy tables so that you do not snapshot all rows. <em>Do this only if your tables have a timestamp column (see problem IV)</em></li><li>Start tailing <em>offset.storage.topic</em><strong> </strong>as a Kafka console consumer to view in on the action.</li><li>Issue the connector update.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c6db08a897f262ecab51e2052aaec55/href">https://medium.com/media/5c6db08a897f262ecab51e2052aaec55/href</a></iframe><ul><li>In the <em>offset.storage.topic</em> you’ll see that the connector tries to obtain an initial snapshot and then commits regular updates.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5f13ff0c749d95064a480e53f089059/href">https://medium.com/media/f5f13ff0c749d95064a480e53f089059/href</a></iframe><h3>Problem II: Database Host changes</h3><p>If your database is migrated to a different host, you’d need to update the connector. However, the binlog file will naturally change.</p><p>As such, any existing connector, which was reading from the old bin-log will suddenly not find the file on the new server. It will <strong>die</strong> a horrible death.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc757734d8734c3a6c2c711cd871c242/href">https://medium.com/media/cc757734d8734c3a6c2c711cd871c242/href</a></iframe><h4><a href="#"><strong>Fix:</strong></a></h4><ul><li>Do what the error message says. Update the connector config by pointing to the new host. Set a new <em>server-id</em> as well.</li><li>Set <strong><em>snapshot.mode=when_needed</em></strong>. Ensure that the replication user exists on the new host and has the necessary permissions.</li><li>A new snapshot would be created. Use the <strong>snapshot.select.overrides</strong> to avoid reprocessing a lot of events.</li><li>Tail the <em>status.storage.topic</em>. From being <em>UNASSIGNED</em>, the connector should move to <em>RUNNING</em>.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8eb5562b123133f93a88e3cd5be82f7/href">https://medium.com/media/e8eb5562b123133f93a88e3cd5be82f7/href</a></iframe><ul><li>If you tail the <em>offsets.storage.topic</em>, you’d see that the binlog is switched to the new host against this connector.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48023a18f054bd08c59c82aa44315b09/href">https://medium.com/media/48023a18f054bd08c59c82aa44315b09/href</a></iframe><h3>Problem III: Lost Events</h3><p>This is less of a Debezium problem and more of an application problem. Although Debezium is battle tested and rarely misses an event, events can sometimes be skipped for a few reasons<em> (code bug in applications, edge cases in Debezium itself).</em></p><p>Depending on the criticality, a lost event could be catastrophic and unacceptable. However, if your consumers are idempotent, we can fix this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3kh1XkKKQyK-Cz_K4fxI6A.png" /><figcaption>A happy state for a Debezium connector. Metrics are a must.</figcaption></figure><h4><a href="#"><strong>Fix</strong></a></h4><ul><li>If you lose events, check if the offset has rolled over in the bin log as well. In such cases, you usually need to know the time at which you started losing out events or you risk taking a bigger less granular snapshot.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9567e93dfe3ff74f10efd3f828e07cf/href">https://medium.com/media/c9567e93dfe3ff74f10efd3f828e07cf/href</a></iframe><ul><li>If <strong>NO</strong>, then, congrats, we’re in a good state as we can replay events. We need to pause the relevant Debezium Connector, reset its offsets and restart it. If <strong>YES</strong>, then we are back to problem 1. And we solved it.</li><li>Stop the connector. Note it’s configuration so that we create the exact same connector again.</li><li>Manually reset Kafka Offset. <strong>Do this only if you know what you are doing. </strong>Below command writes a <strong>NULL</strong> message for the given key which is logically translated to removing stored offsets for the given connector.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f29622e16c7ea78e4da8e714ebf01448/href">https://medium.com/media/f29622e16c7ea78e4da8e714ebf01448/href</a></iframe><ul><li>That’s it. <strong>Offsets have been reset</strong>. To avoid risking a full snapshot, recreate the connector with a <em>snapshot.override </em>with a time that is just before the time where you lost out on events.</li><li>Once connector is recreated, watch the <em>offsets.storage.topic</em> for the new offset written by Debezium.</li><li>All events including the lost events would be replayed. Your consumers should reprocess them now.</li></ul><h3>Problem IV: No Index on Override Table</h3><p>All along we’ve been talking about taking snapshots using <em>snapshot.override, </em>but why?</p><p>If you do not, the <a href="https://debezium.io/documentation/reference/connectors/mysql.html#mysql-property-snapshot-mode">default behaviour </a>is to <strong>select *</strong>. This can be too costly for huge tables, which is when you’d like to restrict the number of rows in the initial snapshot with a <strong>where</strong> clause. Most likely, on a timestamp column.</p><p>However, it’s important to note how Debezium performs an initial snapshot in consideration with larger tables. It acquires a <strong>read lock </strong>and <strong>blocks writes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g9Jcl9Hyi0_43gihmW6oTQ.png" /><figcaption>A non optimized query would stall on a bigger dataset</figcaption></figure><p><em>The lock is unavoidable.</em> After all Debezium has to ensure a consistent state. But what does it mean for writes?</p><blockquote>It means that your APIs, scheduled jobs that write will fail for the duration of the snapshot. This, may again be unacceptable.</blockquote><h4><a href="#">Fix</a></h4><ul><li>It is therefore quintessential to have a short snapshotting period or do it at favourable times during the day (lesser load)</li><li>If the where clause in the override doesn’t have an index, it is in fact worse as it will lead to full scans. <strong>Ensure an index</strong> is created on the snapshot override column. <a href="https://levelup.gitconnected.com/advanced-profiling-and-tracing-mysql-queries-c0c95dc7db24">Profile the query</a> and ensure it is fast enough.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a979b9a280f62b3988772e6b903fd639/href">https://medium.com/media/a979b9a280f62b3988772e6b903fd639/href</a></iframe><p>If by mistake, an initial snapshot starts on a huge table (&gt; 1M rows) and stalls writes, quickly perform the following.</p><ul><li>Delete the connector.</li><li>Deleting connector won’t kill the running database query. <strong>Kill the query.</strong></li></ul><pre>mysql $ show processlist → <br>mysql $ kill &lt;connection-doing-snapshotting&gt; →</pre><h3>Problem V: Encountered Change Event Whose Schema Isn’t Known</h3><p>This is a pretty unfortunate error to have. It happens when there’s a mismatch between the <em>internal meta model</em> and the <em>database schema history</em>. Most likely a DDL didn’t make it to the database history topic and now<strong> any new event will fail.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gtJmNFxsVS-NGZoun1s0A.png" /><figcaption>A failing connector</figcaption></figure><p>When Debezium tries to restart the task, it reads the latest offset from the <em>connect.offsets.topics</em> with a key of the <em>connector name</em>.</p><p>It then tries to compare the schema of later (not latest) events with its internal model (which isn’t updated).</p><p>This blows up depending upon the <em>“inconsistent.schema.handling.mode” </em>property (default behaviour is to fail, others are skip, warn)</p><h4><a href="#"><em>Fix</em></a></h4><ul><li>On checking the <a href="https://github.com/debezium/debezium/blob/237b4ac0f1773e7cfd103bfc216afc44bdbe2635/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/BinlogReader.java#L863">relevant Debezium source code</a>, it was clear that this was a dead end. A simple fix was to recreate the connector with a different name because the offset partition messages are keyed by the connector name. This’d take a new snapshot.</li><li>However, renaming the connector is not always feasible. In such cases, one can directly <strong>specify a tombstone message</strong> that basically tells Debezium that no offset has been recorded so far and a snapshot needs to be performed</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57cff4ec7cab708b1464853e16a868e9/href">https://medium.com/media/57cff4ec7cab708b1464853e16a868e9/href</a></iframe><ul><li>The connector can now be recreated with the same name.</li></ul><blockquote>You might ask, <br>Why would a DDL not make it to the database.schema.history topic. After all Debezium is streaming all events, right?</blockquote><p>It’s a valid question. To answer this, think of what happens when the retention of the history topic is set to be a fixed number of days or bytes, either of which exceeds?</p><p>Debezium needs to maintain all history under one topic.<br><strong> Schema events cannot roll over.</strong></p><p>That’s a wrap for this article. I hope this helps the readers to be able to tackle the issues with Debezium in their production configurations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49fb52d6ac4e" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/fixing-debezium-connectors-when-they-break-on-production-49fb52d6ac4e">Fixing Debezium Connectors when they break on production.</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Hacking” a Hackerrank problem using Python Basics.]]></title>
            <link>https://aman-garg.medium.com/hacking-a-hackerrank-problem-using-python-basics-d61549e37ee6?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/d61549e37ee6</guid>
            <category><![CDATA[hackerrank]]></category>
            <category><![CDATA[hackerrank-solution]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[python-hacks]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Sun, 31 Jan 2021 11:14:50 GMT</pubDate>
            <atom:updated>2021-03-16T01:49:43.319Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*cG7GTePQs2zSEfdBecWt4w.png" /></figure><p>This is a story about how I leveraged some of the Python’s lesser known capabilities to my advantage in solving a regular expression problem in a <em>rather esoteric way</em> (as we’ll see) on a popular coding platform called <a href="https://www.hackerrank.com/"><strong>Hackerrank</strong></a>.</p><p>It’s been 5 years since and I think I can finally come out of the closet and reveal to the world what I did that day.</p><p>Year 2016, January. The 7th.</p><p>I was in the third year of my Computer Science undergrad when I came across the <a href="https://www.hackerrank.com/regular-expresso">Regular Expresso</a> challenge on <strong>Hackerrank</strong>. I found it intriguing that I had to solve a set of<strong> 8</strong> regular expression challenges under an hour.<br><em>Folks not familiar with regex can read about it </em><a href="https://en.wikipedia.org/wiki/Regular_expression"><em>here</em></a><em>.</em></p><p>I equipped myself with my then favourite programming language, Python to be able to complete the task.</p><p>Fast forward <strong>5</strong> questions and I’m blazing through the contest in no time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XSY918xaNEqetSDLeEUtfQ.png" /><figcaption>First five questions in no time (duh: they were easy)</figcaption></figure><p>Enter the 6th question, <strong>the Match Maker</strong>. This was labelled as <strong>Hard</strong>. <br>The full question can be found <a href="https://www.hackerrank.com/contests/regular-expresso/challenges/match-maker">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbsEhTtjaEORu4YstMKHUA.png" /><figcaption>The Dreaded question #6</figcaption></figure><p>The challenge talked about how we had to<strong> write a regex</strong> that had to match all of these <em>(insert a list of words)</em> and none of these<em> (another list of words)</em>. The core constraint was that the <strong>length of the regular expression we write cannot exceed 60.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQciIDOA7fxbxzRh3ii1dQ.png" /><figcaption>What we had to fill.</figcaption></figure><p>This, the platform validated using the below stub which we couldn’t override.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sbciG-qePRL0_-2dGYiUpA.png" /><figcaption>Notice the greyed non editable block.</figcaption></figure><h3>The Struggle</h3><p>The challenge to the problem was to first, find the regular expression and then (second) optimize it to a minimum length.</p><p>I did find a regex but the length was exceeding <strong>60</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X4bylfmT5m_GwD5l65NqVg.png" /><figcaption>This works but is at 75 characters</figcaption></figure><p>After struggling for <em>30 minutes</em> with the time running out, I thought I’d try to game the system. I knew the Hackerrank stub checks that the length of my input using the len() and discards it if it is greater than <strong>60</strong> right away.</p><p>len() is an inbuilt function on the Python module __built__in</p><blockquote>If only I could modify the logic that aims to find the length of a string</blockquote><h3>The Hustle</h3><p>It hit me that in Python, you can override<strong> even the in built functions</strong>. This I thought was pure golden and evil while I was reading through its documentation back in the day.</p><p>I decided that I will override the len() function to always return a number that is less than or equal to 60. This way, I hoped I’d pass the stub validation as it only checks if the length is greater than 60</p><pre>def len(string):<br>&quot;&quot;&quot; An evil implementation that is only aimed to pass the test&quot;&quot;&quot;<br>   return 60  </pre><h3>And I did xD</h3><p>I pressed submit and then to (not) my surprise all the <a href="https://www.hackerrank.com/contests/regular-expresso/challenges/match-maker/submissions/code/4546340">tests passed.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqeoIv46FSY1hcxpagrU3Q.png" /><figcaption>The Joy of passing all tests :D</figcaption></figure><h4>How Hackerrank could have avoided this?</h4><p>Well, for starters, detect overrides of built in functions, maybe? <br>These would now belong to a different module than __built__in__</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*H-_ta9f8AV9Tqyy35XkrFA.png" /><figcaption>An overridden function has a different memory address</figcaption></figure><h3>The Aftermath</h3><p>I went on to solve one more problem in the challenge but the joy I experienced in<em> Problem #6 </em>surpassed it all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSwqPUe8rqCDFvX2wItvFg.png" /><figcaption>The #8th problem for which I’d bought time because of the hack in #6</figcaption></figure><p>So, that’s all folks. The takeaway is that Reading documentation helps.<br>Hope you all enjoyed this piece.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d61549e37ee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Profiling and Tracing MySQL Queries]]></title>
            <link>https://levelup.gitconnected.com/advanced-profiling-and-tracing-mysql-queries-c0c95dc7db24?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0c95dc7db24</guid>
            <category><![CDATA[jdbc]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[profiling]]></category>
            <category><![CDATA[mysql-performance]]></category>
            <category><![CDATA[query-performance]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Sun, 03 Jan 2021 20:19:15 GMT</pubDate>
            <atom:updated>2021-03-12T17:42:39.857Z</atom:updated>
            <content:encoded><![CDATA[<p>Explore the various ways to tackle unoptimised MySQL queries.</p><p>Many times you’d like to profile the query that is being executed. This could be due to a myriad of reasons.</p><ul><li>Current execution is latent and we need to optimize.</li><li>To figure out unused indexes or to determine if indexes are even potentially on the query path or not.</li><li>To figure out the query execution plans.</li><li>To profile the time taken for the queries at the server.</li></ul><h3>Slow Query Logs</h3><p>If you are experiencing misbehaving or slow queries, first and foremost, the server should be set to log them. Without knowing what queries are actually hogging the server, we can never optimize them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ky-hpVerrmjbDjlmVPZlOg.png" /><figcaption><strong>show global variables like ‘slow_query%’</strong></figcaption></figure><p>Set a particular threshold of time units so that the logs are less noisy and occupy less disk,<em> say, logging requests that take over 10 seconds</em> <em>(be careful with your production load)</em>. A sample logged query looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AF_SG2I37GFD-bHCDIBmzQ.png" /><figcaption>A Sample Slow Query is logged</figcaption></figure><p><strong>Tailing the log tells us quite a lot,</strong></p><ul><li>Which process is invoking the slow query. <br><em>Sometimes a rogue client blocks resources in indirect ways</em></li><li>How long did it take? <br><em>This is not the estimation, rather the post query execution true number. Highly time taking queries need to be isolated first.</em></li><li>Number of rows it examined and the number of rows it returned. <br><em>If you are examining lots of rows and returning few, an index might be due.</em></li><li>Bytes sent over the network. <br><em>Queries that clog the network need to be offloaded from the OLTP to OLAP.</em></li><li>When this query was executed (Logged before the query has run).<br><em>Timelines help when the issue is discussed for auditory purposes.</em></li></ul><blockquote><strong><em>Caution</em></strong><em>:</em></blockquote><blockquote><em>1) Slow query logs on the server are not rotated by default. If unchecked, the log partition may choke on disk. Please set the threshold accordingly.</em></blockquote><blockquote><em>2) Enabling slow query logs has some performance hit, so be sure to monitor.</em></blockquote><h3>Explain Query</h3><p>Now that we’ve isolated a few queries, it’s time to explain their existence.<br>The EXPLAIN keyword is used to provide information about how MySQL database executes a query.</p><p>Instead of the usual result set, MySQL would then show its statement execution plan by explaining which processes take place in which order when executing the statement and the estimated cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKsfANOENtViPgPIS9inkg.png" /><figcaption>Note: Use EXPLAIN with UPDATE queries carefully.</figcaption></figure><p><em>EXPLAINing</em> all of your queries is the next best thing to do. You can see whether or not MySQL has optimised your query to leverage “<a href="https://www.burnison.ca/notes/fun-mysql-fact-of-the-day-got-it-covered"><em>covering index optimisation</em></a>” by looking for the <strong>Using index</strong> in the <strong>Extra</strong> column. And if your query isn’t Using index, you may want to find out why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFrhAKCrCtzCJRAaDrlo0w.png" /><figcaption>Output (redacted) of an EXPLAIN Query</figcaption></figure><ul><li>For each of your query within a subquery, try to see the possible keys (index) that can be used (<strong>posssible_keys</strong>). This is MySQL’s way of telling you that it could potentially use any one amongst this set. <strong>Please note MySQL can only use one index at a time</strong>. <em>Not many, are aware</em>. You may create 10 indices, but it’s going to use only one, based on its heuristics.</li><li>You may also force MySQL to use an index but this is not always recommended. For example, you may have to write native queries in Hibernate or other ORM’s to achieve this. Not convenient.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JgYJnVNnqzOItQ2YtqbwNQ.png" /><figcaption>ORM and Force Index queries do not gel well.</figcaption></figure><ul><li><strong>Key</strong> column defines the index actually chosen. Totally based on <a href="https://dev.mysql.com/doc/refman/5.7/en/optimizer-hints.html"><em>MySQL heuristics</em></a>. Our hope is that MySQL chooses an index that best covers the maximum columns in the where clause.</li><li>One thing to note is that if you have just created an index, MySQL query optimiser hints might not reflect that and you’d be confused why a more suitable <strong><em>index you just created isn’t chosen </em></strong>in the MySQL Explain output. Give it some time, fire some queries and then watch the heuristics update.</li><li>Understand the difference between an <strong>Uncorrelated subquery</strong> and a <strong>Correlated / Dependent Subquery.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdqOQ2sPD0dnIe283ZP1qQ.png" /><figcaption>Uncorrelated: MySQL derives a scalar value for inner query against each outer query. <strong>O(outer)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCeozjvm8hmiXcOLvpvtXw.png" /><figcaption>Dependent: Non scalar. MySQL executes inner query once for each row in outer_table. <strong>O(outer * inner)</strong></figcaption></figure><ul><li>When you can’t leverage index optimised sorting, you should try to keep your sort keys as small as possible, and especially smaller than <a href="https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_length_for_sort_data"><strong>max_length_for_sort_data</strong></a>.</li><li>If you see heavy <a href="https://dev.mysql.com/doc/refman/8.0/en/order-by-optimization.html#order-by-filesort"><strong>filesort</strong></a>, trade off some accuracy for sorting efficiency. <em>SET SESSION max_sort_length=32</em> would tell MySQL to compare only the first <em>32</em> bytes of each field, rather than the entire thing.</li></ul><p>For example, if you had an <em>ORDER BY campaign.date_created,</em> <em>campaign.name</em>, we may only want to sort the first <em>32</em> bytes of a campaign name instead of using the default <em>1024</em> bytes.</p><ul><li>Read more about making the most of EXPLAIN output <a href="https://dev.mysql.com/doc/refman/8.0/en/explain-output.html">here</a></li></ul><h3>Query Tracer</h3><p>It’s one thing to be able to read the output of the EXPLAIN query. However, in order to dive deep, you need to exactly see how MySQL is going to structure your query and what each of those potential indexes can or can not cover.</p><p>Optimiser traces are verbose and spit a lot of output. You get things like</p><ul><li>Table dependency</li><li>Potential Range indexes (important)</li><li>How it has estimated the number of rows and the cost</li><li>How it is going to prepare the join.</li><li>How it is going to execute the join.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*CAo7Z39VdZlK18AT5wzTzw.png" /><figcaption>Usage: Enabling Query Tracing. The response is pretty <a href="http://The response is pretty verbose.">verbose</a>.</figcaption></figure><p>Mostly, we are looking for the key <strong>potential_range_indexes</strong> . Here you get to see what all columns are covered by each of the index MySQL has found on this table.</p><p>For each of the columns in a potentially multi column index, MySQL evaluates the parts that are useful. <em>(Note that this is in reference to a specific query and the results would vary for each dissimilar query)</em></p><p>For example, consider the below query</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9bImV2x035ZDC34SvrukiA.png" /><figcaption>A query with “3” where parts, the last one is not trivial</figcaption></figure><p>Our clauses in the where parts are:</p><ul><li><strong>creative_template_id</strong></li><li><strong>status</strong></li><li><strong>concat(content_id, content_version)</strong></li></ul><p>If we examine the tracer output, of all the indices, MySQL told us that the last one, <strong>idx_vcg_creative_template_status_idversion</strong> will cover the most keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*H-1qkAz_XKXWF9hBHJIX9A.png" /><figcaption>The last usable covers most keys even according to MySQL</figcaption></figure><p><em>How does this help?</em></p><p>It’s a confirmation that the index is in fact <strong>usable </strong>and that the index does cover the <strong>key_parts</strong> as intended. You can imagine that for non trivial queries, this might help ease out the flow because only one index is going to be used.</p><p>If that is sub optimal, your query would perform poorly as well.</p><p>Note: In the above case, <strong>concatenation(a,b) != index(a,b)</strong>. Things like <a href="https://confluence.fkinternal.com/display/SKADOOSH/2020-10-22+%7C+DB+Queries+%7C+CM+%7C+Virtual+Column+and+Index+%7C+Content+API+Optimization">virtual columns</a> can help here.</p><h3>Query Timing Profiling</h3><p>Now that you’ve figured out the weaker paths in your query, you may have worked upon a set of optimised query candidates. How do you know which candidate to take it to production?</p><p>The <strong>Standard Query profiler</strong> is available that will help you time the results of the query execution at the server. However, it’s demoralising to know that majority of the <em>developers measure query execution time incorrectly</em>.</p><p>They include the time to start transfer and other network factors into consideration. For example, if you are using a GUI client and performing analysis, you cannot rely on the number reported in the GUI status bar. Refer this. (<a href="https://github.com/sequelpro/sequelpro/issues/264">source</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WLGnGcM_bM0Z20YzmaCCQ.png" /><figcaption>What are you really measuring ?</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*hB5V0V1tjqJp93SpClYdLA.png" /><figcaption>Steps to enable query timing profiling at the server (The orange window)</figcaption></figure><p>Let’s take the following set of queries on a table named <em>campaign</em> having a primary key with over<em> 3 million rows</em>. However, the only non primary index is on a column called <strong>end_date.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gpYqaXtuRNev3MLM-KXCzQ.png" /><figcaption>Selecting 10 rows with varying query patterns</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RKsU59yjDSi82l7CdL0Tvw.png" /><figcaption>Query timing results on a ~3 million cardinality set</figcaption></figure><p>The results from the profiler are self intuitive. You can see how having to look up an index in second query was a tad bit <em>slower</em> and led to a <em>slight</em> penalty.</p><p>You can also see that not having an index plus filesort led to a massive <em>5700% </em>penalty. In this way, you can get really good estimates of your query executions. The results will directly translate into production numbers.</p><h3>JDBC Profiling</h3><p>In rare scenarios, sometimes, a query might perform well when isolated, but fail poorly when hit through in a sequence of multiple database calls<em> (example being an API that fires 10 calls to the database)</em>.</p><p>In such situations, you can also enable <strong>profiling</strong> on the JDBC driver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QFS0HxFgZv7yFn_7IGw4bA.png" /><figcaption>MySQL JDBC Driver with <strong>profileSQL</strong> set to true</figcaption></figure><p>It will then output each and every SQL that you can use to</p><ul><li>Analyse each and every query being fired to the driver.</li><li>The duration it took</li><li>The rate at which queries are being fired to the driver.</li></ul><p>It even captures the healthcheck or validation queries that are sent from a JDBC pool such as a TomCat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QzSd8TFYIWR4E3i6OvbqQw.png" /><figcaption>Profiled <strong>HEALTH_CHECK</strong> FETCH call from the connection pool to the driver.</figcaption></figure><p>This log is critical to detect if there are any <a href="https://stackoverflow.com/questions/16577907/hibernate-onetomany-relationship-causes-infinite-loop-or-empty-entries-in-json">cycles being performed in ORM frameworks</a> such as when Entity A with one to many B, loads B which loads A one to one. <em>This happened in our production setups leading to a query stall.</em></p><p><strong>If you do not profile (we did), you’ll never know why the query hangs.</strong> However, if you profile, you’d know why you keep seeing multiple calls being fired to the driver. Each query might have perform well in isolation. This profiling will help you catch a bug in your code base and you’ll see cycle and how you can use <a href="https://stackoverflow.com/questions/37392733/difference-between-jsonignore-and-jsonbackreference-jsonmanagedreference">JsonReference</a> to handle this.</p><p>It’s been quite a ride exploring various ways to drill down to the root cause.<br>To summarise.</p><pre>- Log the Query first. How else do we know what to debug <br>- EXPLAIN the Query and check the possible indexes, extra clauses<br>- Trace out query execution to discover deeper structures in joins<br>- Use MySQL profiles to log (various) query execution at the server<br>- Use JDBC profiler and note for any cyclic / unnecessary queries.</pre><p>If you made it this far, thank you for your time. I hope this was useful.<br>I’ll catch you guys in the next one. Happy profiling.</p><p><strong>References</strong></p><ul><li>Read <a href="https://learning.oreilly.com/library/view/high-performance-mysql/9781449332471/">High Performance MySQL</a> It maybe outdated, but the content is solid</li><li>One of the best blogs on MySQL Internals is by <a href="https://www.burnison.ca/notes/fun-mysql-fact-of-the-day-really-temporary-files">Richard Burnison</a></li><li>Standard MySQL documentation is pretty good at <a href="https://www.burnison.ca/notes/fun-mysql-fact-of-the-day-really-temporary-files">dev.mysql.com</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0c95dc7db24" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/advanced-profiling-and-tracing-mysql-queries-c0c95dc7db24">Advanced Profiling and Tracing MySQL Queries</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production logging feature enhancements atop Jersey and JAX-RS]]></title>
            <link>https://aman-garg.medium.com/production-logging-feature-enhancements-atop-jersey-and-jax-rs-991127a36c88?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/991127a36c88</guid>
            <category><![CDATA[restful-web-service]]></category>
            <category><![CDATA[server-logs]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[jax-rs]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Sat, 05 Sep 2020 17:26:32 GMT</pubDate>
            <atom:updated>2021-03-16T01:49:47.063Z</atom:updated>
            <content:encoded><![CDATA[<h3>Production logging feature enhancements atop Jersey/JAX-RS</h3><p>And how to make the most of them for non trivial use cases in production</p><figure><img alt="Logs that are about to rotate" src="https://cdn-images-1.medium.com/max/1024/0*itc0HjvTh2JT-LsN.jpg" /><figcaption>Logs that are about to rotate and topple</figcaption></figure><p><strong>Logs</strong> are the lifeline of any non trivial application running on production. Any one who has been an on-call at their workplace can corroborate the significance of proper application logging when things go south at 3 AM.</p><p>For applications built atop the <strong>JAX-RS </strong>framework with an implementation such as Jersey, the solution is usually to register the <a href="https://eclipse-ee4j.github.io/jersey.github.io/apidocs/latest/jersey/org/glassfish/jersey/logging/LoggingFeature.html">Logging Feature</a> with the environment. <em>This works both on the client side and the server side.</em></p><pre>environment.jersey().register(<br>  new LoggingFeature(<br>     Logger.getLogger(LoggingFeature.DEFAULT_LOGGER_NAME),<br>     Level.INFO,<br>     LoggingFeature.Verbosity.PAYLOAD_ANY, <br>     1000<br>   )<br>);</pre><p>With this simple registration, you immediately start seeing Request &gt; and Response &lt; logs on the server. In between lie the application logs (if any)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iteor8B3cks8pY4Zipc9hw.png" /></figure><p>This, I admit is pretty neat and extremely handy when starting out. However, there are some problems.</p><ul><li><strong>Every request and response is logged. No matter the use case.</strong><br>Requests that are aimed at monitoring, say health checks or other clean up APIs receive the same treatment as other potentially business critical requests. We have no control over the requests that we want to actually see the logs for.</li><li><strong>No way to log only the requests that failed, say with a 4xx / 5xx</strong><br>Checking the source code of <a href="https://github.com/jersey/jersey/blob/faa809da43538ce31076b50f969b4bd64caa5ac9/core-common/src/main/java/org/glassfish/jersey/logging/ServerLoggingFilter.java#L112">Logging Filter</a> explains why. In the <a href="https://www.baeldung.com/jersey-filters-interceptors">Jersey request lifecycle</a>, the <em>filters</em> are executed prior to serving the actual <em>request</em> and handling the <em>response</em>. Hence, <em>request</em> logging is pretty much guaranteed to happen regardless of the result.</li></ul><blockquote>The impact of this is at least two fold.</blockquote><ul><li><strong>Log files fill and rotate at a faster pace</strong><br>Because all the requests are logged, there’s very little to be done about the amount of logs being generated. It is O(N) where N is the number of requests. In reality we’d like this to be O(K) where K is the number of requests we are interested in (log worthy) and K &lt; N</li><li><strong>Log files contain a majority of non actionable requests</strong><br>Usually we want to debug requests that failed. Since there’s no distinction between a successful request (2xx) or a failed request, we usually end up grep-ing for the failed responses. This coupled with a faster rotation means that for high QPS services, we have a lot to look through.</li></ul><p>We encountered the same problems in our production applications. Some of them were consistently hitting 600 QPS and with an error rate of 0.01% the logs were being rotated at a fairly high pace.</p><p>We thought to tackle both the problems independently. <em>Our first goal was to filter the logging at a request level.</em></p><p>We dove deep on the Jersey implementations of these filters. We faced certain hiccups extending these classes. Probably because they were internal and not meant for extension. But our use case demanded it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lO9FY2nfj9YBCjbHwOtnUA.png" /><figcaption>Why are you final ? :/ Hmph :</figcaption></figure><h4><strong>WhiteListed Server Logging Filter</strong></h4><p>To solve the first problem, we followed suit with the Jersey Implementation and created a class that implements the necessary filters. The logic is simple.</p><ul><li>Maintain a set of <em>resource URI paths</em> that you are interested in.</li><li>Override the filter methods for both request and response bodies. Check whether request URI context is matched by any of our eligible requests. If yes,<strong><em> log it</em></strong> else do <strong><em>nothing</em></strong>.</li></ul><p>This overhead is O(K) * O(1) where K URIs have to be matched backed by a set. This is a good trade-off for the amount of disk we’d save on logging.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f37133c75d015cef89211ff7873e7c1/href">https://medium.com/media/2f37133c75d015cef89211ff7873e7c1/href</a></iframe><p>This variant of the logger can now be installed in services to exclude certain requests with heavy duty payloads.</p><pre>Set&lt;String&gt; excludedPaths = ImmutableSet.of(...);</pre><pre>environment.jersey()<br>  .register(new WhitelistedServerLoggingFilter(excludedPaths));</pre><h4>Delayed Request Response Logging Filter</h4><p>Solving the second problem of logging only the requests where responses have failed or met a certain condition, requires a bit of a thought.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHieeI-6K-FQH6kTg--Z-Q.png" /><figcaption>Server Request Response Processing Lifecycle</figcaption></figure><p>To see why this is non trivial, we need to understand the lifecycle that is followed for a request in the Jersey world.</p><p>The request is done logging in the ContainerRequestFilter layer where the response is not yet available.</p><p>Since, there’s no way to “<strong>unlog</strong>” a logged request, we need to find a better way. The better way would some how delay request logging until the very end when the response is available.</p><p>One thing we can latch on for sure is that all the major frameworks follow a <strong>thread per request</strong> model. This means there’s a dedicated worker thread that is dispatched to serve an incoming request.</p><p>We can use this to our advantage. What is the data structure that can be used to store properties local for a thread?</p><p>You guessed it right. It’s a <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html"><strong>Thread Local</strong></a><strong>. </strong>We could set the request log in the request thread’s local. Once we do get the response later, we match a <em>user defined predicate</em> for success against the response code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3665a217c69665a28db65055dc16f358/href">https://medium.com/media/3665a217c69665a28db65055dc16f358/href</a></iframe><p>If the test passes, <em>both the requests and responses are logged</em>. If not, <em>both are discarded</em>. This can be configured and enhanced on a need basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5Bmxj4mMtQariHRGg2zaA.png" /><figcaption>Blue print for the delayed logger</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcb66ffa110e6c4a9734b96993eb8bab/href">https://medium.com/media/bcb66ffa110e6c4a9734b96993eb8bab/href</a></iframe><p><strong>This simple variation trades space for space</strong>. We save a lot of unnecessary disk usage due to logs being generated only against the response predicates we want. But we also occupy memory by caching request log at every thread.</p><p>In our case, this tradeoff was also worth it. Our applications only logged what we required. Setting the response predicate as{4xx, 5xx} — {404} helped us achieve the required goals of debuggability and disk usage.</p><h4>Gotchas</h4><ul><li>The astute reader might have noticed that because the request logging is delayed, the application logs are written to disk prior to the actual logs thus hampering the sequence of events.</li><li>We’ve only delayed the request logs, the application logs log.info({}) will continue at the usual pace. This wasn’t a problem for us at all. It could be for you.</li><li>One needs to be careful to unset the thread local and make the objects eligible for Garbage collection. For very high QPS, the extra memory might be a problem. But then they might not require this level of logging anyway.</li></ul><p>All in all, these enhancements helped us in maintaining our production services at high QPS with ample balance between debuggability and log rotation. For client side enhancements, we followed suit.</p><p>I’ve open sourced the <a href="https://github.com/isopropylcyanide/jersey-log-utils">components</a> published a <a href="https://mvnrepository.com/artifact/com.github.isopropylcyanide/jersey-log-utils">package</a> to Maven Central for the same. Integrating these utilities within the application is straight forward.</p><figure><img alt="https://github.com/isopropylcyanide/jersey-log-utils" src="https://cdn-images-1.medium.com/max/934/1*FIFdZTmWp_u6kRJgPeVjgQ.png" /></figure><p>Hope you learnt something new. Please feel free to leave some comments on what could be improved further. Contributions are always welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=991127a36c88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Unit Of Work Support for JDBI in Dropwizard Part — III]]></title>
            <link>https://aman-garg.medium.com/building-unit-of-work-support-for-jdbi-in-dropwizard-part-iii-d9d0b1e245c3?source=rss-1f6d3dd56261------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9d0b1e245c3</guid>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[unitofwork]]></category>
            <category><![CDATA[jersey]]></category>
            <category><![CDATA[jdbc]]></category>
            <category><![CDATA[dropwizard]]></category>
            <dc:creator><![CDATA[Aman Garg]]></dc:creator>
            <pubDate>Sun, 28 Jun 2020 09:52:41 GMT</pubDate>
            <atom:updated>2021-03-16T01:50:21.828Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Unit Of Work Support for JDBI in Dropwizard Part — III</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bu4SdrkLuBCMk3SGyYMV3w.png" /></figure><p>In this article, we continue our <a href="https://medium.com/@aman_garg/building-unit-of-work-support-for-jdbi-in-dropwizard-part-ii-4857ca6c48cf">journey</a> of building the support for Unit Of Work for JDBI in Dropwizard.</p><p>Having described how the Jdbi Handle Manager, Event listeners, Transaction Aspect and the resource layers work in liaison, we now turn our attention to the possible implementations for the handle manager.</p><p>Recall, the contract for a Jdbi Handle Manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kf2NuZzo3MMAB_qcnsk-dw.png" /><figcaption>Contract for a handle manager</figcaption></figure><p>While the contract looks simple (<em>and the implementation is too</em>) it is here that the most important decision of providing a handle to a thread is made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuNZ6k_sy3TtUSbPBwGT6w.png" /><figcaption>Invocation site for the manager #get()</figcaption></figure><p>Let’s talk about HandleManager#get() first</p><p>Our proxied Dao’s need to get an active handle before they go on about using it in the context of a SQL Object. If it is not supplied or worse, is null it’ll blow an exception up the call stack.</p><p>More over, we’d need to provide a way to clear the handle, once the task is done or the pool may run out of handles very quickly.</p><p><strong>Default Jdbi Handle Manager</strong></p><p>So, what’s the most naive way of implementing such a handle manager?<br>We could, for instance, open up a real handle using the data source each time the #get() method is invoked. And our clear() implementation would be empty or throw a NotSupportedOperationException</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcqj_9nOJGBRDIMkYbElJA.png" /><figcaption>Default Jdbi Handle manager implementation</figcaption></figure><p>Since a new handle is opened and returned every time, the application will quickly run out of handles.</p><p>More over, as the Dao is proxied, if a thread issues requests on multiple methods of the Dao, or a method on another Dao, it will create a new handle. Hence there’s no way for us to achieve a transaction context with multiple methods.</p><blockquote><em>This implementation can be used to service requests which interact with only a single method in a single handle.</em></blockquote><p>While clearly not suitable for a production use case, it may find its place in embedded database tests.</p><p><strong>Request Scoped Jdbi Handle Manager</strong></p><p>Drawing up on the limitations of the default manager, we realise that our handle manager needs to be equipped with the following capabilities in order to stitch the entire picture.</p><ul><li>Every Dao that is touched by the thread servicing the request needs to share a common handle . Period.</li><li>Handle is scoped to the thread for its lifetime. Once the thread is done processing, it should release the handle.</li></ul><p>The above two use cases call for us to implement a <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/ThreadLocal.html">ThreadLocal</a> variable that represents the handle associated with the current thread. No other thread needs to get a handle on this handle (pun intended)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vrDiT1IYR9xTidw14NDZbw.png" /><figcaption>Request scoped Handle Manager (Logs omitted)</figcaption></figure><p>All SQL objects are attached to a common handle. This makes a transaction boundary span multiple Dao access methods in different Dao classes.</p><p>What this means is, you can now have a transaction across multiple data sources / tables backed by different Dao classes</p><p><strong>This works. It really does.</strong> <br>However, it relies on us to properly call the #clear() method.</p><p>We do this, through the JdbiTransactionAspect which is invoked through the FINISHED Jersey Request monitoring event being handled in the JdbiUnitOfWorkEventListener . Both these classes were discussed in detail in the previous parts of this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*POIYAqUJpRvDTZzZWR4TQQ.png" /><figcaption>Our event listener clean up hook</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*j4rpbQipQeG5tKHgaNmsjg.png" /><figcaption>Through the aspect, we dereference the handle in a finally block</figcaption></figure><p>While we do need to account for exceptions here and there, a finally block helps us always clear the handle.</p><p>That’s it. We’ve now achieved a request scoped handle manager that is invoked from proxied Dao classes called by the request thread whose event are being monitored by an event listener</p><p>The <strong>request scoped thread handler</strong> works for most production use cases. However, there’s just one small limitation . <em>See if you can figure it out.</em></p><p>Since the ThreadLocal handle is assigned to each thread, a single thread can never spawn multiple IO threads that talk to the database concurrently.</p><p><strong>Why</strong>? Because each new thread will get a new handle and threads cannot inherit thread locals. This is something that even Dropwizard’s original @UnitOfWork doesn’t support. And for good reason. It is unreliable.</p><p><strong>Linked Request Scoped Jdbi Handle Manager</strong></p><p>For use cases that require spawning multiple threads to access the database from a single request thread, it gets <a href="https://stackoverflow.com/questions/1531073/is-java-sql-connection-thread-safe/1531103#1531103">tricky</a>.</p><p>However, there’s a trick (<em>although not totally fail safe</em>) to circumvent this issue and share the handle among new threads.</p><p>Since we own the thread that spawns multiple threads, we can create new threads using a <strong>custom thread factory</strong>. This thread factory constructor utilises the current thread’s ID <em>(unique monotonically increasing number).</em></p><p>We call this the <strong>conversation ID</strong>. Thread Id is a good enough way of ensuring uniqueness. All we’re trying to do is to create a unique identifier between parent and child threads. Loosely, this is what <a href="https://docs.spring.io/spring-webflow/docs/current/api/org/springframework/webflow/conversation/ConversationId.html">Spring uses</a> for one of its components</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*93mNS0cYC0SG7q3WQTZNcQ.png" /><figcaption>Creating a conversation ID</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ppU0naqK2Gz_VMe5kkv2BQ.png" /><figcaption>Custom thread factory to establish a conversation.</figcaption></figure><p>Now, the request thread creates multiple threads but it does so using the custom thread factory. Our name format looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*HzmNcQY8Og75GAXZNXo_tg.png" /></figure><p>We know the request scoped implementation fetches from a thread local. So, we switch from a ThreadLocal to a ConcurrentMap&lt;String, Handle&gt;</p><p>Since the original request thread is guaranteed to have a handle <em>(already initialised through the aspect-event listener) </em>we’d at least have one entry in the map or we’d panic.</p><p>Whenever the child thread wants to get the handle, it searches for the key in the map. There are two problems here.</p><ul><li><strong>How do we know whether a thread is a parent or a child? <br></strong><em>Through the thread factory.</em> Since all request threads are created with a custom thread factory, we can look up their name using some String logic to figure this out. By default all threads not explicitly created are a parent.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z-llI1Oc5ie6S29PoyNI9A.png" /><figcaption>A thread is not a parent if it is created using our own NAME_FORMAT</figcaption></figure><ul><li><strong>How does a thread finally access the handle?<br></strong>By accessing the map with the key being the identifier of the parent. If the thread is not a parent, the handle opened by its parent is returned. If it is in fact a parent, a handle is opened explicitly and put into the map.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UFf46lSUVgoVQl75qOApqQ.png" /><figcaption>The handle is selectively created for the first time and cached for its child threads</figcaption></figure><p>Here’s a test that asserts on new threads sharing the same handle, when created using the specified thread factory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QY3eMcZG59tjU2lLSogwnA.png" /><figcaption>6 child threads, each starting at once and sharing the same handle.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VSHs0z040y6h4cCbW8IcNg.png" /><figcaption>All 6 child threads share the same handle. Notice the name format for child threads where main’s ID is logged. The hashcode for handle is same at 109532419</figcaption></figure><p>On the other hand, if we don’t set a custom thread factory, the test fails because each thread tries to get a handle as a parent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLiIxIxvbDJGQYhiV11GQA.png" /><figcaption>All 6 child threads get a new handle. No name format for the thread. The hashcode for handles are different</figcaption></figure><p>Having said this, <strong>Linked RequestScoped Jdbi Handle Manager</strong> needs to be used with care. Playing and relying on a thread factory format which developers are free to violate (hard to detect) is not reliable.</p><p>If you’re going to dole out your own version of a handle manager, ensure that there are <strong>no handle leaks</strong>. This implementation was discussed just to demonstrate that the JdbiUnitOfWork can work across multiple threads spawned from a single request thread.</p><p>With this, we complete our journey of building the UnitOfWork substitute for the Jdbi backends. Be sure to check out the complete source code here. <a href="https://github.com/isopropylcyanide/dropwizard-jdbi-unitofwork">https://github.com/isopropylcyanide/dropwizard-jdbi-unitofwork</a></p><p>Hope you’ve found something useful along the way. <em>I sure, did.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9d0b1e245c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>