<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark McDonald on Medium]]></title>
        <description><![CDATA[Stories by Mark McDonald on Medium]]></description>
        <link>https://medium.com/@mark.mcdonald_63206?source=rss-4bec727a2206------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cITNcmkNdiFzmLAkHHdbFw.jpeg</url>
            <title>Stories by Mark McDonald on Medium</title>
            <link>https://medium.com/@mark.mcdonald_63206?source=rss-4bec727a2206------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:03:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mark.mcdonald_63206/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating amending to older commits]]></title>
            <link>https://medium.com/in-the-weeds/automating-amending-to-older-commits-c33740c1beb3?source=rss-4bec727a2206------2</link>
            <guid isPermaLink="false">https://medium.com/p/c33740c1beb3</guid>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Mark McDonald]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 21:56:55 GMT</pubDate>
            <atom:updated>2019-03-05T21:04:06.355Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Automate Amending to Older Commits</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PHti5O-eOlplM_mYcD5Fbw.jpeg" /><figcaption>Tools of the Trade for Mending</figcaption></figure><p>I like to maintain a clean git history — it helps code reviewers go step by step, and it isolates changes for the poor future souls spelunking through the code base.</p><p>Often times I make a long string of small commits, then make a small change related back to an old commit. It’s easy to squash this small change into the last commit with git commit --amend, but it’s trickier to squash into an older commit.</p><p>I follow the flow outlined by George Brocklehurst in <a href="https://thoughtbot.com/blog/autosquashing-git-commits">Auto-squashing Git Commits</a>:</p><ol><li>Find the old commit’s SHA that I want to amend to</li><li>Create a ‘fixup’ commit referencing that sha: git commit --fixup $SHA</li><li>Perform an interactive rebase with autosquash: git rebase --autosquash --interactive $SHA^</li></ol><p>Autosquash is great because it sets up the ‘fixup’ commit to be squashed into the commit it references when rebasing, but it will only do this when rebasing interactively. This forces us to confirm the autosquash looks good every time using an editor.</p><p>I realized I was typing the same set of lines over and over again and always accepting the confirmation screen, so I created a small bash function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdc95ad7efdbc9291820bf13f4e42690/href">https://medium.com/media/cdc95ad7efdbc9291820bf13f4e42690/href</a></iframe><p>The neat trick here is that using “cat” as our editor skips the “interactive” part of the rebase.</p><p>So now I can update an old commit without thinking about the fixup and rebase flow I need to perform. It’s as simple as git add . &amp;&amp; amend-ng 18740ae.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c33740c1beb3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/automating-amending-to-older-commits-c33740c1beb3">Automating amending to older commits</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A small bit of open source: withModals.js]]></title>
            <link>https://medium.com/in-the-weeds/a-small-bit-of-open-source-withmodals-js-46791f4451dc?source=rss-4bec727a2206------2</link>
            <guid isPermaLink="false">https://medium.com/p/46791f4451dc</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[higher-order-components]]></category>
            <dc:creator><![CDATA[Mark McDonald]]></dc:creator>
            <pubDate>Fri, 19 Oct 2018 14:42:27 GMT</pubDate>
            <atom:updated>2018-10-19T14:42:27.621Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://reactjs.org/docs/higher-order-components.html">Higher ordered React components</a> are great for reducing boilerplate and generalizing patterns. We’ve written a few here at Greenhouse, and I think you might find them as useful jumping off points to write your own.</p><p>To start out with, I’m sharing our smallest, simplest, and most used higher ordered component: withModals.js. It helps out by managing the state of modals (open or closed).</p><p>Here’s the documentation at the top of withModals.js:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a086217358e8fac3f8bcfe88371ca1e3/href">https://medium.com/media/a086217358e8fac3f8bcfe88371ca1e3/href</a></iframe><p>I whipped up an example for you that uses a &lt;Modal&gt; component to display some content on demand:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae9e28aad028d4ce457e8a0fcb2115c9/href">https://medium.com/media/ae9e28aad028d4ce457e8a0fcb2115c9/href</a></iframe><p>And here’s the source:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6c20f80224d9964ef78226a28ef1ea2/href">https://medium.com/media/b6c20f80224d9964ef78226a28ef1ea2/href</a></iframe><p>I’ve also created a code pen for you to try out and experiment with. It hides and displays text instead of popping open a modal, but I think it gets the point across.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2Fmarkymarkmcdonald%2Fembed%2Fpreview%2FVExLMK%3Fheight%3D600%26slug-hash%3DVExLMK%26default-tabs%3Djs%2Cresult%26host%3Dhttps%3A%2F%2Fcodepen.io&amp;url=https%3A%2F%2Fcodepen.io%2Fmarkymarkmcdonald%2Fpen%2FVExLMK&amp;image=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fi.cdpn.io%2F2493921.VExLMK.small.8e1f99fd-41ac-4bc9-bb7b-2e73b97abead.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/f34acb72a18ff66b432ed15f16b5cf36/href">https://medium.com/media/f34acb72a18ff66b432ed15f16b5cf36/href</a></iframe><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46791f4451dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/a-small-bit-of-open-source-withmodals-js-46791f4451dc">A small bit of open source: withModals.js</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fixing a Race Condition]]></title>
            <link>https://medium.com/in-the-weeds/fixing-a-race-condition-c8b475fbb994?source=rss-4bec727a2206------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8b475fbb994</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mark McDonald]]></dc:creator>
            <pubDate>Thu, 15 Feb 2018 15:54:49 GMT</pubDate>
            <atom:updated>2018-02-15T18:30:37.285Z</atom:updated>
            <content:encoded><![CDATA[<p>I’d like to share my journey of fixing a race condition and the things I learned along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SkwhXuTw8S9qSiVWdezC0Q.png" /><figcaption>Uh, not these kinds of race conditions. Race conditions within nondeterministic systems!</figcaption></figure><p>I work as a software engineer on the recruiting app here at Greenhouse. We help customers throughout all parts of the hiring process, from sourcing to interviews to offers.</p><p>In today’s journey we’ll focus in on one of these aspects: scheduling interviews through Greenhouse.</p><h3>An introduction to scheduling at Greenhouse</h3><p>The typical flow for a recruiter scheduling an interview with a candidate is:</p><ul><li>Ask the candidate for their availability</li><li>Set up a time and place for interviewers to meet the candidate based on availability</li><li>Send calendar event invitations to interviewers so they can accept/decline and remember it when dealing with their schedules</li><li>Make updates to the calendar event time and attendees as life happens</li></ul><p>You may be wondering how we keep up with updates to calendar events in external systems — we stay in the know by having users add an extra attendee to every calendar event. Our background workers process email updates in that attendee’s inbox and also periodically poll for the status of events that user was invited to.</p><h3>The Problem</h3><p>Very rarely, an interview would list the same interviewer for an interview twice. We would investigate how it could have happened, fix up the data, and then move on to the next customer issue. Our codebase already tries to prevent duplicates: we track interviewers in a set on the front end and have explicit “does this interviewer already exist” checks on the back end.</p><p>Unfortunately we didn’t add uniqueness constraints at the database level when we created the scheduling system back in 2013, so a race condition somewhere can still add duplicate interviewers into the database. Time to pay down some tech debt!</p><h3>Isolating the race condition</h3><p>I started by chatting with the developers that had previously worked on the problem. They believed the race condition was caused by an interview update being handled by more than one process at the same time. One of the developer’s previous fixes was enabling <a href="https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs">Sidekiq’s unique jobs feature</a> for the background worker that handles event updates. This greatly reduced the frequency of the problem by preventing multiple workers from handling the same event update, but duplicates still popped up occasionally.</p><p>I scoured the codebase for the places event updates are being processed — and found a service that was called from both the now-unique background job and the controller action displaying the edit interview page. This means our web server and the background job could both handle the same update if our periodic update poller found the update at the same time that the user tries to view the interview page. Our product team let me know that we check for interview updates before displaying the edit interview page to help make sure users aren’t making updates to outdated information — and we’d like to keep this functionality.</p><p>To confirm that this was a likely cause of duplicate interviewers, I searched the logs for all the actions related to an interview with duplicates. Luckily we already log information about the event right before doing any processing. This lead to the smoking gun of double “about to update interview x” log statements within a few seconds of each other for all the interviews with duplicate interviewers. One of the updates was triggered by a user visiting the edit page, and the other was triggered by a background poller at about the same time.</p><p>To make absolutely sure the coinciding updates weren’t a red herring, I tried to reproduce the conditions locally. I followed a great <a href="https://blog.arkency.com/2015/09/testing-race-conditions/">blog post</a> by Robert Pankowecki outlining how to write a test that spun up multiple threads to mimic multiple processes updating the same interview.</p><p>Here’s the test I came up with (with tweaks to be more understandable as a stand alone spec):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/544f7f14c0dd712ceaca209300f1e2c5/href">https://medium.com/media/544f7f14c0dd712ceaca209300f1e2c5/href</a></iframe><p>I ran the test and it failed. Good news everyone! I ran it a few more times to make sure it was consistently failing — I wanted to make sure the test passing in the future wasn’t a false positive once I started working on solutions.</p><h3>Why does this test case create duplicates?</h3><p>We check that an interviewer doesn’t already exist for an interview with something equivalent to: Interviewer.find_or_initialize_by(user_id: added_user.id, interview_id: interview.id).</p><p>This method of checking that an interviewer isn’t already associated with an interview before adding them isn’t an atomic database operation. Here’s the flow for a single update of “interviewer added” being processed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*2gDKh2dkLDfZMyjdYrq56Q.png" /></figure><p>Because our existence check and interviewer insertion are separate database statements, it’s possible for another process to insert an interviewer right after our check.</p><p>But our logs show that the updates are happening within a few hundred milliseconds of each other. I’d expect the difference in time between log statements to be much lower — closer to our database latency time — in order for two processes to run the existence/insertion combo at the same time. So a likely flow diagram for two processes updating the same event looks more like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*Mg-NbL3tM_-vvlfs8iyBXA.png" /></figure><p>So it seems like Postgres does not recognize the interviewer added in connection 1 when connection 2 checks if they exist. It’s actually the usage of database transactions that is increasing the potential interval for a duplicate interviewer. Let’s dive into database transactions real quick to understand more.</p><h3>A quick refresher on database transactions</h3><p>Database Transactions:</p><ul><li>Are a list of SQL statements that the database takes and promises that either none or all will succeed (useful for modeling a business process as an atomic action).</li><li>Can be isolated from other transactions (so we don’t need to worry about one transaction’s statements impacting another transaction’s statements while either can still be rolled back)</li></ul><p>We use Postgres as our database of choice. By default, Postgres runs transactions at the “Read Committed” isolation level: any reads in transactions won’t see the impact of other transactions that have not been committed yet. So the second transaction does not see that the interviewer has already been inserted by the first transaction until the first transaction has been committed.</p><p>This is one of the reasons you often hear advice like “keep transactions as short as possible” — the likelihood of the duplicate interviewer check failing increases the longer it takes our transaction to commit.</p><h3>The missing constraint</h3><p>It’d be great if back in 2013 we had set a database uniqueness constraint based on interview id and user id. Then, the second transaction would violate the constraint and be rolled back unless we provided an <a href="https://www.postgresql.org/docs/9.6/static/sql-insert.html#SQL-ON-CONFLICT">“on conflict”</a> option to our insert statement. The “on conflict do update” upsert strategy fits our use case perfectly — we’d update the existing interviewer instead of creating a duplicate.</p><p>So going forward we could switch to the “on conflict” upsert syntax and then add the uniqueness constraint to the existing table. Unfortunately, uniqueness constraints require a backing index that takes a bit of time to build up for large tables. At Greenhouse our database tables are large enough that we have to build any new indexes concurrently — no table locking allowed. Otherwise, we would need to schedule downtime as any request related to interviewers would timeout until the index was built.</p><p>It might take several tries and lots of crossed fingers to successfully create the index while duplicates can still be created in the system. This is because Postgres marks a unique index as “invalid” if a row is inserted that violates uniqueness while the index is being built up. Postgres won’t be able to use “invalid” indexes to enforce the constraint, and we’d have to drop the index and try again.</p><p>So that means our goal is to achieve data integrity without the database constraint, fix all existing violations, and then add the database constraint as a safeguard against future changes.</p><h3>How can we achieve data integrity without adding the constraint?</h3><p>Here are some strategies I looked into:</p><ul><li>Changing the transaction isolation level</li><li>Periodically running a process that cleans up the data (a “reaper”)</li><li>Using database locks</li></ul><p>I think there are other viable ways to solve this problem (changing up the data modelling, breaking up transactions, etc), but these are the strategies I ranked highly in the trade off of complexity and confidence.</p><h3>Transaction Isolation Levels</h3><p>The Postgres docs go in depth on the <a href="https://www.postgresql.org/docs/9.6/static/transaction-iso.html">isolation levels</a> available for a transaction and the kinds of race conditions that each level avoids. Here’s a table from their documentation summarizing the race condition anomalies possible at each isolation level:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*PpjOKmD3jEFpLQzeXA-M6Q.png" /></figure><p>It looks like the “Serializable” level avoids all race conditions. So we can set the isolation level of our transactions to “Serializable” and call it a day, right?</p><p>Unfortunately it’s not that easy. My assumption was that Postgres would prevent these anomalies from occurring through some implementation detail like locks. But it actually checks at the end of the transaction to see if an anomaly would occur right before committing and roles the transaction back if so. This means that increasing the isolation level would give us the guarantee that we wouldn’t have any more duplicates, but we’d need to add application level retry logic for the case when an anomaly occurs.</p><p>We’d also be increasing the isolation level for the entire transaction, not just our uniqueness check. Other statements in the transaction could produce anomalies that we don’t care about right now because they don’t violate our business requirements — so we can’t assume that a transaction failing to commit is always caused by our “double update from separate processes” and ignore the failure.</p><h3>Fear the Reaper</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*QA5v11EcT_1NENe2RAyhzw.gif" /></figure><p>As part of diagnosing the issue I created a database query to find all the interviews with duplicate interviewers. We could piggyback on this to create a background job that finds duplicates and deletes all but one of them. This is one of the easiest solutions if you already have infrastructure set up to run periodic jobs and don’t have a lot of duplicates. The interval between runs would need to be tweaked based on the tradeoff of performance impact and stale data.</p><p>Unfortunately it’s still possible for duplicates to be added to the system and temporarily seen by users. Displaying information that changes over time due to an eventual consistency strategy reduces our user’s trust in the system. It also may still take several tries to add the uniqueness constraint if a duplicate is found while building up the index before the reaper deals with it.</p><h3>Using database locks</h3><p>Database locks are automatically acquired by Postgres when you run statements. For example, ALTER TABLE interviewers ADD COLUMN surname VARCHAR(255) NOT NULL DEFAULT &#39;smith&#39;; will implicitly acquire a &quot;AccessExclusiveLock&quot; on the interviewers table while it populates the default value of surname for all existing rows. We can add our own locks based on our business logic requirements with <a href="https://www.postgresql.org/docs/9.6/static/explicit-locking.html">explicit locking</a>.</p><p>Locking the interviewers table for writes every time there is an update would destroy the performance of our concurrent background workers (along with our web processes). I didn’t really investigate this option that much.</p><p>I ended up settling on acquiring a row-level write lock on the interview that interviewers were being updated for. Adding the lock to the start of each transaction forces the second transaction of separate processes to wait on the first to commit and release the lock before continuing.</p><p>Here’s an example of using ActiveRecord’s <a href="http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html#method-i-with_lock">with_lock</a> method to acquire a row level lock based on the interview id:</p><pre>interview.with_lock do  <br>  new_calendar_attendees.each do |attendee|<br>    add_attendee(attendee, interview)    <br>  end<br>  ... # Rest of the update logic<br>end</pre><p>The SQL statement with_lock executes before running the passed block looks similar to SELECT * FROM interviews FOR UPDATE WHERE interviews.id = ?. The FOR UPDATE clause is what tells Postgres to acquire a <a href="https://www.postgresql.org/docs/9.6/static/explicit-locking.html#LOCKING-ROWS">row level lock.</a></p><p>With this code change in place, the test we wrote earlier started passing. Wahoo!</p><p>We shipped this change to production and I monitored the count of duplicate interviewers for a few days to make sure it didn’t increase.</p><p>Next I created a data migration that moved duplicate interviewers to a temporary table, and watched the duplicates count stay at 0 for another week. Now that we’ve achieved data consistency we can add the database uniqueness constraint so we don’t run into this problem in the future again.</p><p>Let’s pat ourselves on the backs and move on to the next story!</p><h3>The Aftermath</h3><p>Not all development is rainbows and sunshine though — two weeks later someone tracked down a database error. It was possible to have a deadlock: transaction A has lock 1, transaction B has lock 2, and transaction A is trying to acquire lock 2 while transaction B is trying to acquire lock 1. Luckily Postgres recognizes this deadlock state and aborts one of the transactions to free up the locks.</p><p>When adding the row-level write lock on interviews, I missed a conditional code path where we had already acquired write-locks by updating other interview rows before handling the calendar event updates.</p><p>The unexpected lock acquisition was coming from the implied locks of update statements related to interviews we weren’t updating the interviewers of. We don’t actually care about blocking writes to any interview rows — we were using the row level lock as a stand in for “I want to lock this interview’s interviewers for updates without locking the interviewers table”. I think that using an <a href="https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS">advisory lock</a> would better fit this use case and would have avoided the deadlock.</p><p>Advisory locks were created to help model business processes that require locks without the overhead of creating empty tables that are only used for locks. Postgres never acquires an advisory lock implicitly and the locks aren’t associated with any table or row.</p><p>We could switch our SELECT * FROM interviews FOR UPDATE to SELECT pg_advisory_xact_lock($interview_lock_constant, $interview_id) to acquire an advisory lock instead of a row lock. Postgres can accept two 32-bit integers to identify an advisory lock, so we could use a pattern of having the first argument be a constant denoting the kind of application rule we’re enforcing, and the second argument being metadata about that rule. Check out <a href="https://www.postgresql.org/docs/9.6/static/functions-admin.html#FUNCTIONS-ADVISORY-LOCKS">these docs</a> for all the available SQL functions to deal with advisory locks.</p><p>I hope you learned as many new things as I did!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8b475fbb994" width="1" height="1" alt=""><hr><p><a href="https://medium.com/in-the-weeds/fixing-a-race-condition-c8b475fbb994">Fixing a Race Condition</a> was originally published in <a href="https://medium.com/in-the-weeds">In the weeds</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>