<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Austin Pray on Medium]]></title>
        <description><![CDATA[Stories by Austin Pray on Medium]]></description>
        <link>https://medium.com/@austin.pray.mxpnl?source=rss-f8ae54cfd006------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*NZhtDKvV_7M7oPHB</url>
            <title>Stories by Austin Pray on Medium</title>
            <link>https://medium.com/@austin.pray.mxpnl?source=rss-f8ae54cfd006------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:21:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@austin.pray.mxpnl/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Enforcing required checks on conditional CI jobs in a GitHub monorepo]]></title>
            <link>https://medium.com/mixpaneleng/enforcing-required-checks-on-conditional-ci-jobs-in-a-github-monorepo-8d4949694340?source=rss-f8ae54cfd006------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d4949694340</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Austin Pray]]></dc:creator>
            <pubDate>Fri, 28 Apr 2023 20:26:41 GMT</pubDate>
            <atom:updated>2025-05-14T20:15:21.273Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHaxUfdy0mkC-xx7SIhaZA.png" /></figure><p>If you have only configured a few CI jobs that always run on every pull request, enabling GitHub’s <a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#require-status-checks-before-merging">“required status checks” branch protection rule</a> is a simple way to prevent pull requests from being merged without proper CI verification. Simply mark your important CI jobs as required, and you’re done!</p><p>However, what if you have a large monorepo with conditional CI job execution? In that case, things get complicated. As far as GitHub is concerned, a CI job must always execute on a pull request in order to be required. Many repositories <a href="https://github.com/orgs/community/discussions/13690">are impacted by this very limitation</a>.</p><h3>The problem</h3><p>Mixpanel uses a large monorepo shared by many engineers hacking on Python, Go, Typescript, C, and C++ code daily. A necessary optimization in this type of environment is to only run CI jobs related to files changed in a pull request. At best, running unnecessary CI jobs wastes time and money. At worst, it’s frustrating to be blocked by a test flake from a service completely unrelated to your change.</p><figure><img alt="List of a ton of different checks for different environments" src="https://cdn-images-1.medium.com/max/1024/1*AxZPfNCObh9A6IDrJVdtZA.png" /></figure><p>Fortunately GitHub Actions provides <a href="https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore">path filtering for pull request triggers</a>.</p><pre>on:<br>  pull_request:<br>    paths:<br>      - &#39;**.js&#39;<br>jobs:<br>  lint_javascript:<br>    # ...</pre><p>“Only run the javascript linter if you are changing javascript files.” Makes sense to me! We don’t want people merging to master without fixing the javascript lints, so we want to make this a required check.</p><figure><img alt="Setting lint_javascript as a required check" src="https://cdn-images-1.medium.com/max/1024/1*nKJF1mtbghBlnVGjh7uOqA.png" /></figure><p>Uhoh! Now if you submit a pull request that does not modify any JavaScript files, you will be blocked. GitHub will wait indefinitely for the completion of the “lint_javascript” check, preventing you from merging your pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hYAo3DHoB6MUHE5_h9766w.png" /><figcaption>GitHub waiting for a job that will never come :’^(</figcaption></figure><p>GitHub is smart enough to avoid running unnecessary CI jobs, but it will happily wait forever for the same job that it just purposefully ignored.</p><p>The behavior we are looking for is <strong>“if a CI job runs, it is expected to pass.”</strong> This way we reap all the benefits of conditional CI jobs but still prevent people from shipping broken code.</p><h3>The workaround: one check to rule them all</h3><p>One straightforward solution is to have an ever-present check that reports a summary of the conclusions from its sibling checks. We can then target this pass or fail outcome in our required check rule. We elected to call this a “status rollup,” named after <a href="https://docs.github.com/en/graphql/reference/objects#statuscheckrollup">the GitHub API concept of the same name</a>. This check shows up on our pull requests as a “PR is mergeable” check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*akzLqHNoViAkC33tguu-ew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ifMRxR2GqjFM2wATZaazRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LaTelqt3edjXq4iG8W7A-Q.png" /></figure><p>After a couple iterations we settled on implementing this status rollup check as <a href="https://docs.github.com/en/apps/creating-github-apps/setting-up-a-github-app/about-creating-github-apps">a GitHub App</a>. We named our GitHub app “Mixpanel Change Management” or “MCM.” The MCM GitHub App waits for webhook events related to <a href="https://docs.github.com/webhooks-and-events/webhooks/webhook-events-and-payloads#check_run">check runs</a> or <a href="https://docs.github.com/webhooks-and-events/webhooks/webhook-events-and-payloads#status">statuses</a>, does a rollup of the current status of all the pull request checks, and then posts the pass fail result back to the pull request.</p><p>We were able to quickly stand up a GitHub App using <a href="https://probot.github.io/">the Probot framework</a>. Writing your own GitHub app web server is pretty simple, but you are stuck implementing your own development and onboarding process. Using Probot as a framework, all this stuff is completely handled and fully documented. You can just focus on writing your simple webhook handlers and be done with it.</p><p>Our first iteration looked like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U-fBdEpffDuy2Dm6BB8PPA.png" /></figure><p>We eventually dropped the redis locking mechanism and introduced a Google Cloud Pub/Sub queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tX3MNsAdwwokiQ9gLd67wg.png" /></figure><ol><li>Processing status rollups sequentially eliminates the need for a distributed locking solution.</li><li>We added resiliency by nacking messages if a bug prevents the processing of status rollups, or if there are GitHub API outages that prevent rollups from completing.</li><li>GitHub experiences regular webhook outages. To backfill the missing webhook deliveries, we need the ability to manually enqueue a status rollup request for all open pull requests.</li></ol><p>At this point, we have a highly reliable way to always perform a required check on every pull request. Moreover, we can arbitrarily code the pass/fail logic to be whatever we want. This has unlocked some further enhancements to our pull request process.</p><h3>Invalidate stale CI results</h3><p>By default, GitHub’s required checks will accept a CI result that passed at some point, even if it is months old. However, merging with a super stale CI result can break the default branch that everyone relies on being stable. While GitHub does have an option to enforce that branches be up to date before merging, this is an unrealistic requirement in repositories with even a moderate amount of traffic.</p><p>We updated our status rollup logic to check that the CI results on pull requests are within a given staleness threshold. Then we added a cron that goes through and periodically enqueues status rollups for every active pull request. If a pull request is too stale, the failing status rollup result will block the PR until the author retriggers CI.</p><p>Once <a href="https://www.notion.so/https-github-com-mixpanel-analytics-pull-49618-366e2416618c4d0fb2471351793853b5">GitHub’s merge queue feature</a> becomes generally available, this feature might not be necessary. However, until then, it prevents nasty silent merge conflicts and other related issues from being shipped.</p><h3>Break glass: bypass branch protections</h3><p>So now GitHub fully enforces our change management process. All changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7HQWqp8hY5bHDxLu-P2MDw.png" /></figure><ol><li>Require a pull request before merging</li><li>That pull request needs an approving code review</li><li>You have to have a passing CI result, as determined by MCM’s status rollup</li></ol><p>Awesome! But what about emergency situations where you need to bypass this process and merge something for an immediate deploy? Without anything in place, incident responders have to go get ahold of someone with admin rights to the repository to bypass branch protections on their behalf. This is not ideal:</p><ol><li>This is an auditable event for us in our SOC2 process. So we would have to trust that the people with bypass rights would diligently document the context of each bypass. Someone has to be assigned to annoy these people each time a bypass occurs.</li><li>The harm caused by delaying the resolution of an incident is magnitudes higher than anything that could be prevented by having an admin double check the contents of an emergency PR before merging. 99% of the time the repo admin would immediately approve the emergency pull request anyway.</li></ol><p>So we added a “break glass” feature to MCM:</p><ol><li>A user adds a label to the PR marking it as an “emergency deploy”</li><li>MCM will immediately approve this PR</li><li>The CI status rollup is immediately set to passing</li><li>All the necessary audit events are generated and proper people are notified</li><li>User can merge their PR unobstructed</li></ol><p>It looks like this:</p><figure><img alt="example break glass scenario" src="https://cdn-images-1.medium.com/max/1024/1*eVE7xdPNWUDkCS5wRX_pGg.png" /></figure><h3>Conclusion</h3><p><a href="https://github.com/orgs/community/discussions/13690">Many repositories</a> encounter the same issue with required checks and conditional CI jobs. While it would be ideal for GitHub’s branch protection to natively support these use cases, we have been satisfied with our workarounds. Enforcing our change management process entirely through GitHub has given everyone peace of mind and made our SOC2 audits was easier to conduct. I hope this write-up gives you some ideas on how to tackle this in your own repo!</p><p>If this kind of thing interests you, the DevInfra team at Mixpanel is hiring! Apply <a href="https://mixpanel.com/jobs">here</a> and mention this blog post in your application!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d4949694340" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mixpaneleng/enforcing-required-checks-on-conditional-ci-jobs-in-a-github-monorepo-8d4949694340">Enforcing required checks on conditional CI jobs in a GitHub monorepo</a> was originally published in <a href="https://medium.com/mixpaneleng">Mixpanel Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>