<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Monark Dedakiya on Medium]]></title>
        <description><![CDATA[Stories by Monark Dedakiya on Medium]]></description>
        <link>https://medium.com/@monarkd?source=rss-7aa083401e08------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vBxmc01TkQRCLRC9tO0sWQ.png</url>
            <title>Stories by Monark Dedakiya on Medium</title>
            <link>https://medium.com/@monarkd?source=rss-7aa083401e08------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:24:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@monarkd/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Behind the Scenes: Solving deadlocks worth millions]]></title>
            <link>https://engineering.razorpay.com/behind-the-scenes-solving-deadlocks-worth-millions-8b0c8aa22eac?source=rss-7aa083401e08------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b0c8aa22eac</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[mysql]]></category>
            <dc:creator><![CDATA[Monark Dedakiya]]></dc:creator>
            <pubDate>Tue, 09 Nov 2021 14:34:50 GMT</pubDate>
            <atom:updated>2021-11-09T14:34:50.368Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lLJXIx4-ZA3uyfsXdBmc_g.png" /></figure><p>It all started one fine morning when we received alerts of a few skipped settlements for a merchant. Settlement is the process of transferring customer payments funds to the merchant’s bank account. We also noticed that the concerned merchant received a “negative balance error” which means the amount that needed to be deposited in the merchant’s bank account was higher than their current Razorpay balance. This seemed quite out of place as you can imagine and got our team wondering what could be causing this? It was clear that we needed some serious data analysis to get to the bottom of this situation.</p><h3>Data Analysis proved dirty reads</h3><p>Just some context before we dig in: The <em>balance</em> table holds the live balance which gets updated on each credit or debit transaction. <em>balance</em> has one entry for each merchant. The same row is updated on each transaction, so it’s not possible to find the history of balance updates here. However, we have another table called <em>transaction</em> which holds records of each credit or debit in separate rows.</p><p>To get started, we first found the history of updates in the <em>balance</em> table from the order of entries in the <em>transaction</em> table. The merchant balance flow first updates the <em>balance</em> table, and then a new <em>transaction</em> entry is created with the updated balance value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*P99ZtB-HXUVvBA5o7s-BkA.png" /><figcaption>Balance dirty read example</figcaption></figure><p><em>New balance = balance from balance table + current transaction amount.</em></p><p>We dumped the production transaction table data into an excel sheet. We calculated the current balance after each transaction by deducting the amount from the balance. We also calculated the ideal balance of each transaction by adding the current transaction amount to the previous transaction balance. Ideal balance indicates what value the balance column should have in the transaction table if everything went fine.</p><p>It was quite a challenge given the size and scope of the data since we had multiple transactions within one second. Such huge volumes of data make it complex to find the exact transactions that cause such discrepancies in the balances.</p><p>On further investigation, we found that multiple transactions read the same balance concurrently. Few transaction amounts were overwritten by subsequent transactions, causing the balance to drop. For example, in the above table, rows #2 and #3 read the same balance and update the value accordingly.</p><p>We also observed that merchants who had credits were experiencing this problem. <a href="https://razorpay.com/docs/payments/dashboard/my-account/credits/">Credits</a> are a type of transaction fee source. Team also noticed that merchants who were impacted by the issue were using the <a href="https://razorpay.com/route/">transfers</a> feature offered by Razorpay. Transfer feature allows merchants to move the payment funds to other accounts or make a payout to vendors. The focus was shifted to the payments and transfers side of the system. We started diving deep into code to look out for misses in mutex and DB transactions.</p><h3>No Errors and Perfect Code</h3><p>We checked application logs of all problematic payments, and we found no issues, but their data reads were dirty. It was clear, at this point, that this would not be like any other bug, and needed all hands on deck.</p><p>Analyzing the excel sheets, it was noticed that balance discrepancies were occurring when <a href="https://razorpay.com/route/">transfers</a> and payments were trying to update the balance within the same millisecond. The case today was one of concurrency and race conditions. The code was taking the lock on FOR UPDATE on the merchant balance row in a DB transaction while reading the old balance so no other DB transaction can read or update the balance while the current DB transaction is getting processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z7Nwn5I0M48xoagFPiMZlA.png" /><figcaption><a href="https://gist.github.com/dmonark/18cfdec7fc4b889123ce9ce475e42784#file-balance_update-php">https://gist.github.com/dmonark/18cfdec7fc4b889123ce9ce475e42784#file-balance_update-php</a></figcaption></figure><h3>Insights from Issue Reproduction</h3><p>Replicating the issue in a staging environment is crucial to get insights into the root cause. Razorpay’s functional test automation framework suite, built on Java &amp; TestNG was used. We began with a test that created a payment and transferred funds. That test created a new merchant on each run, which makes debugging a lot easier since it creates new data. We removed all the unnecessary steps like irrelevant API calls to make it faster which offers more chances of producing concurrent scenarios. The aim was to reproduce concurrency issues without any network errors and memory / CPU issues. The stage server capacity was also modified to help in replicating concurrency issues at all times.</p><p>We were able to successfully reproduce the issue after several attempts and identified the infra parameters needed to reproduce it. It was not straightforward to assess whether we were able to reproduce the issue after each test attempt. The database alone was not sufficient to confirm that the issue was reproduced. Each test run of these transactions was analysed in the same way as the production transactions were. The cause of the problem so far remained a mystery. It was necessary to thoroughly examine the database logs.</p><h3>Locks on FOR UPDATE not working as expected</h3><p>The initial assumption was that balance update queries were slow and getting terminated by the app. Slow query DB logs however did not have any queries associated with balance update flows, confirming that slow query was not the cause.</p><p>In addition, performance schema table snapshots were taken and printed as application logs to see what is going on with the database during lock and update operations. Performance schema logs confirmed that 2 different DB threads were reading the same dirty balance.</p><p>Checking the database log files was the next step of the debugging process. The general logs of the production database aren’t enabled due to performance constraints. Stage DB general logs were enabled and the issue was recreated on stage to gain more insight. This was an excellent bet. The DB logs revealed some important information and valuable insight. However, DB logs were extremely large in number. Out of the many, we started looking for one particular balance discrepancy in DB logs. The DB logs revealed that the lock on the FOR UPDATE was not working as expected. The same rows were locked by two different DB threads at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*JinuIdPDH3Twu4FR7cVUjQ.png" /><figcaption>DB logs of Balance dirty read example</figcaption></figure><h3>Experiments with Database</h3><p>Since locks were not working as expected, we changed a few database configurations in an attempt to find the source of the problem.</p><p>The First attempt was checking the <a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html">isolation level</a>. The isolation level defines how the MySQL server (InnoDB engine) separates each transaction from other concurrent transactions in the server. There are multiple types of isolation levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hCusVtYPNQ2KU0m-" /><figcaption><a href="https://alxibra.medium.com/isolation-level-in-rails-847edc9e347d">https://alxibra.medium.com/isolation-level-in-rails-847edc9e347d</a></figcaption></figure><p>The isolation level was changed from repeatable read to serializable. Taking this action removed balance discrepancies but reduced system performance by 70%, so these changes are not suitable for production use at all.</p><p>The next idea was to try <em>PDO_MYSQL</em>, a driver that implements the PHP Data Objects (PDO) interface to enable access from PHP to MySQL databases. A <em>PDO_MYSQL</em> attribute, <em>ATTR_EMULATE_PREPARES</em> was tried and was set as true. Razorpay systems run various checks on SQL queries, but this setting will bypass them, hence the setting was not suitable for production. After applying <em>ATTR_EMULATE_PREPARES</em> changes in stage env, the application started reporting deadlock errors while updating certain payment and transfer balances. The deadlock errors gave a different perspective and new direction to approach this problem.</p><h3>Root Cause</h3><p>Balance update flow takes a lock on FOR UPDATE on 2 entities, one on credits (an entity that represents merchant credits) and the other on merchant balance. Payment flow was taking locks on the credits table first and then on the balance table while the transfer flow was taking the locks in reverse order. Payment capture and Transfer flows initiated the process at once and after their lock on FOR UPDATE on credits and balance respectively, both processes waited for each other resulting in a deadlock.</p><p>The DB transaction was also getting terminated because of the deadlock errors. When a lock on FOR UPDATE is done within a transaction on a table/row, it will get released only in 2 circumstances. (a) Once the transaction gets committed (b) the transaction gets terminated due to an error. In this case, a deadlock error was thrown inside the DB transaction, which released all the DB locks. If someone tries to take a lock on a FOR UPDATE statement outside of a DB transaction, it is treated as a select query only. No lock will be taken. If the table’s values are updated, it will be committed immediately, since the surrounding DB transaction was terminated and the connection was in auto-commit mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7eY0fiQDkC3F199tF69QA.png" /></figure><p>When two different processes, a payment flow and a transfer flow, start at the same time, they both took their first DB lock on their respective entities. When they attempted to take a DB lock on a second entity, a deadlock occurred, which terminated the DB transaction of one flow and released all DB locks of that flow. The application ignored the deadlock error and updated the balance. The other flow took a lock of both entities. It read the same balance and continued to run without any errors and updated the value.</p><p>The next question is why the application continued to run when DB was throwing a deadlock error. PHP ignores all deadlock errors when <em>PDO::ATTR_EMULATE_PREPARES</em> is set to false due to a bug.</p><p><a href="https://bugs.php.net/bug.php?id=76742">PHP :: Bug #76742 :: PHP PDO mysqlnd ignores InnoDB deadlock errors (1213)</a></p><h3>Fix</h3><p>To prevent deadlocks from appearing, the easiest solution was to swap the order of locks in transfer flow. It was decided not to release it all at once to avoid risks. The fix was rolled out for a few merchants for whom balance discrepancies were being found frequently. This was accomplished using our internal AB testing &amp; ramp up platform called <a href="https://engineering.razorpay.com/building-splitz-razorpays-platform-for-high-scale-experimentation-c917fe15d8f6">Splitz</a>. The team monitored merchants’ traffic and analyzed data for a few days, then once everyone felt confident that the solution was working, released it to all merchants.</p><h3>Automate Reconciliation Data Analysis</h3><p>The team began researching an automatic solution to identify discrepancies in balance without any manual intervention. The in-house solution was developed which verifies the sum of the amount for an entire day, then verifies the difference between the beginning balance and ending balance for one particular day. A system regularly monitors merchant accounts to find balance discrepancies as quickly as possible. An alert is sent if there is a mismatch.</p><h3>Learnings</h3><p>This debugging journey had lots of learnings. Here are a few:</p><h4>Good Test, Good Life</h4><p>A good test case is always helpful for reproducing issues quickly. An important factor to consider when conducting concurrency tests is to remove all unimportant possible bottlenecks. Test environments should mimic production environments as much as possible. Try to avoid errors associated with the network, the CPU, and the memory. Each test run should create/use new data to simplify debugging.</p><h4>History of changes</h4><p>Every important entity should have versioning capability or proper audit. We spent lots of time in data analysis to fund problematic entities, which could have been reduced if we had the balance entity version-able or proper audit logs.</p><h3>Hiring</h3><p>We are hiring! If our work excites you, we are actively hiring for our Engineering team. We are a bunch of ignited minds simplifying financial infrastructure for businesses and we’re always looking out for great folks. We are looking for both Engineering Managers and Senior Engineers to break barriers and build for the future. Apply via our <a href="https://razorpay.com/jobs/?utm_src=medium&amp;utm_medium=blog">jobs page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b0c8aa22eac" width="1" height="1" alt=""><hr><p><a href="https://engineering.razorpay.com/behind-the-scenes-solving-deadlocks-worth-millions-8b0c8aa22eac">Behind the Scenes: Solving deadlocks worth millions</a> was originally published in <a href="https://engineering.razorpay.com/">Razorpay Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>