<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ryan Clark on Medium]]></title>
        <description><![CDATA[Stories by Ryan Clark on Medium]]></description>
        <link>https://medium.com/@rynclark?source=rss-7d32937bc4ea------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*K0Vn12c3KArYln3MKc3zLQ.jpeg</url>
            <title>Stories by Ryan Clark on Medium</title>
            <link>https://medium.com/@rynclark?source=rss-7d32937bc4ea------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:11:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rynclark/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Enforcing code ownership at Udemy]]></title>
            <link>https://medium.com/udemy-engineering/enforcing-code-ownership-at-udemy-9daf7a83ee96?source=rss-7d32937bc4ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/9daf7a83ee96</guid>
            <category><![CDATA[udemy]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[codeowners]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Ryan Clark]]></dc:creator>
            <pubDate>Fri, 02 Oct 2020 12:53:35 GMT</pubDate>
            <atom:updated>2020-10-02T16:16:29.638Z</atom:updated>
            <content:encoded><![CDATA[<p>Code ownership is a tricky thing to get right. Things become even trickier when you throw a mono-repo into the mix, with no single team having ownership over the whole thing.</p><p>This topic wasn’t something that interested me at first, I must admit. The task of coming up with a solution that enforces ownership of every single file in a repo isn’t something I’d normally drop everything for. The more I thought about it, however, the more I wondered — how can it be done well?</p><h3>GitHub Code Owners is half the solution</h3><p>GitHub offers up a really <a href="https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners">neat feature called code owners</a>, something that we use heavily at Udemy, in all of our repos. Alongside this, you can configure your main branch to require that at least one code owner approves the pull request before it’s eligible to be merged. This works great for simple solutions, but when it comes to mono-repos, with teams overlapping on ownership and no individual team owning all of the code, GitHub’s code owners offering was no longer enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*GgJWCCRVCr4iC4hyLk_5mQ.png" /><figcaption>We already enforced code owners on our repos, but this doesn’t cover if some of the files aren’t owned by anyone</figcaption></figure><p>See, our mono-repo is not only a mono-repo, but it was a mono-repo with thousands of files without an owning team (around 1 in 8 files did not have an owner entry in the CODEOWNERS file). This meant that it was very easy to edit something that you or your team did not own without the actual owners being any the wiser. This is a huge no-no for security and compliance.</p><h3>How can we solve the other half?</h3><p>I work on the Build Engineering team — if you want to read more about that,<a href="https://medium.com/udemy-engineering/how-udemy-does-build-engineering-9722e98a4208"> check out my previous article here</a>, where I go over what my teams and I get up to. To put it simply, part of our job is creating tooling and solutions for developer workflows, and this is where we came into the puzzle.</p><p>One of the other parts of our job is providing a CI platform to the engineering org. Aha! There’s the solution. How can we get every file in this repo to have the correct owner and enforce that every new file added in the future is also owned by a team?</p><p>Well, if we’re going to be enforcing it on the policy level <em>anyway</em>, then there’s no point running CI for a pull request if there are changes to files that aren’t owned by anyone. It would be a waste of resources — the pull request <em>shouldn’t </em>be merged anyway and a future modification to the CODEOWNERS file would cause a new build to fire when the changes are merged into the branch too.</p><h3>Changing our development workflow</h3><p>We treat our engineers as real customers of our product. Sure, our user base is an order of magnitude smaller than that of Udemy.com, but to us they are just as important as our customers. Every change we make — big or small — must be thought out, done with good reason, have benefits to the engineering org as a whole, and finally, <strong>be communicated well. </strong>We wouldn’t be happy if someone changed our stuff overnight and we can’t expect our developers to be either.</p><p>This is a huge change in developer workflows. Before, developers could have been blissfully unaware that a file they added or changed wasn’t covered by a code owner, even though the rest of their pull request was. This couldn’t continue, but it’s not the developer’s fault- the “punishment” cannot lay solely on them. We can’t ask them to change the way they work and delay them in doing so, without good tooling around the solution. A compromise, if you will.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrhC9Ap0eGKo8QaUyAeq-g.png" /><figcaption>This is a stripped down sample output of the repository visualising tool. The repos and files here do not reflect our actual setup — there are many, many more</figcaption></figure><p>We calculate the code ownership of every file in our repos whenever there is a merge into the main/default branch. We also calculate the state of the directories with them being able to either be fully covered, not covered at all, or have partial coverage. This helps provide a really nice way of visualizing aspects of the repo that aren’t covered and developers can quickly check if the areas they actually own are reflected in the CODEOWNERS file.</p><p>We also point them to exactly where in the CODEOWNERS file the matching rules are from, so it’s easy to understand why a file is owned by a certain team. This also enables them to fix this issue quicker without having to mentally parse a bunch of globs and apply that to the complicated folder structure of the mono-repo. Hovering over the badge on the file shows you what teams own it and you can click through on the team to get more information about them, such as the Slack channels they use.</p><p>This is a really helpful tool outside of enforcing code owners everywhere — it’ll be kept in place for the foreseeable future. This was our compromise to our engineers — yes, we need to stop your CI running if you change or add a file that doesn’t have a code owner, but in return, we’ll make it as easy as we can for you to fix and visualize your ownership so you’re not disrupted.</p><p>The other side of the puzzle is then checking that the changed files in the pull request are owned by at least one team. Fortunately, we already have custom logic from GitHub through to Jenkins that allows us to control and visualize development workflows a lot easier, and we could slot this right into it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wx_QWh_yeR63ji3dVQGeRg.png" /><figcaption>Sometimes I like to pretend that I can draw so it makes justifying buying the iPad a little bit easier</figcaption></figure><p>Whenever CI would normally run for a pull request, an API request is made to our Build Portal that pulls all the relevant information about the pull request it needs as well as the contents of the CODEOWNERS file at the branch of the commit SHA (enforcing that changes to the code owners outside of the pull request are pulled in before it’s re-evaluated). We then produce an output report that looks very similar to the build view flow that our developers are used to. In this report, we highlight the files that were added or changed without code ownership, as well as the files they’ve changed that are owned by a team. On top of that, it also includes links to the team’s pages and points to exactly where in the CODEOWNERS file the ownership for that file is defined. If the check passes, CI runs as normal. The status of the check is also reflected in the pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*mBL4dDoPKpBnH0fzCiV9KQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*CdsI2Kj_cBBWlFewlL8w9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*lI7LdkCWIDeG0d9BZpnCYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1e1wNPBdTV37UyLJA_RYcw.png" /><figcaption>A sample output of what the developer sees when they push a commit to GitHub, validating that their changes have ownership</figcaption></figure><p>As I mentioned above, we treat communication pretty seriously, so we include links to Wiki pages containing information about why, how, what, when, why (x2) as well as links to the relevant GitHub documentation about code owners. Slack is noisy, we don’t blame people for missing an announcement. But we also don’t want to leave them confused.</p><h3>Extra checks on code owners</h3><p>When we run the code owners computation on a merge into the main/default branch of the repo, we check if for whatever historical reason there is an entry where the team doesn’t have write access to the repo. If there are any entries without write access, it pings us on Slack so we can fix the issue. The same goes for the checks done on pull requests — if the developer tries to add a code owner entry and the team does not have write permissions on the repo, it’ll fail and notify the developer in the UI that this is the case.</p><p>We think this is a really nice solution that fills in the blanks when it comes to the code owners features from GitHub. It provides an awesome easy to navigate tool for developers to find a file’s owning teams super quickly as well as the security and compliance we needed governing our development repositories at Udemy.</p><h3>Author</h3><p><a href="https://twitter.com/rynclark">Ryan Clark</a> is a Senior Software Engineer on the Build team at Udemy. He works on a variety of different tools such as the Build Portal, CI, and development environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9daf7a83ee96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udemy-engineering/enforcing-code-ownership-at-udemy-9daf7a83ee96">Enforcing code ownership at Udemy</a> was originally published in <a href="https://medium.com/udemy-engineering">Udemy Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Udemy does Build Engineering]]></title>
            <link>https://medium.com/udemy-engineering/how-udemy-does-build-engineering-9722e98a4208?source=rss-7d32937bc4ea------2</link>
            <guid isPermaLink="false">https://medium.com/p/9722e98a4208</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[udemy]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Ryan Clark]]></dc:creator>
            <pubDate>Wed, 20 May 2020 19:56:40 GMT</pubDate>
            <atom:updated>2020-05-20T19:56:40.348Z</atom:updated>
            <content:encoded><![CDATA[<p>An overview of Build Engineering at Udemy and the way we provide continuous integration at scale.</p><p>At Udemy our engineering team is growing at a steady rate, with hundreds of developers that are spread across different teams and many time zones. This diversity has resulted in varying technology stacks depending on the team’s purpose, with some teams possibly having no overlap in technologies. There’s one area that they all have in common, however — the need to test and prove our code. We use Jenkins for all of our CI, but we built a custom frontend and backend for developers to interact with in order to remove all the pain points we all have experienced with Jenkins. Let’s dive into how, and why.</p><h3>Build Engineering</h3><p>I work as part of the “Build [Engineering] team” at Udemy — a team with the core mission to empower engineers with the right tools and automation in place to ensure the best developer user experience possible. Our team is small, but we all have different career backgrounds that enhance our empathy towards frustrations developers may experience. With backgrounds from SRE at Microsoft to a GNU contributor, we cover many bases. I come from a feature development, frontend-heavy background, but I’ve always liked to dive into tooling, infrastructure and operational work as much as I can. I’m also a huge open source fan and contribute to webpack and their webpack-contrib projects where possible.</p><p>Our primary focus is the management of some development environments that deal with the core Udemy application (there are a few teams that manage their own) and to provide the infrastructure for all teams to run continuous integration no matter their needs.</p><p>When it comes to continuous integration, it goes without saying that each team, each project, and each service requires a different scale of complexity when testing their code. For example, our main application has 20,000+ tests running across varying test runners (E2E, frontend testing, unit, integration, etc). At the time of writing, the average duration of a CI run for our main application is around 21 minutes — a number we’re always looking to beat. Improvements are made in places, but then more features and tests are added.</p><p>This has caveats — through no fault of their own, developers rely on CI too much to know if their code has introduced regressions as it’s not viable to run the test suites locally. Developer time is a precious commodity that shouldn’t be wasted — not only should we support every team being able to test their code, we should also enable them to do it as quickly as possible. Having to wait for 45–60 minutes for CI to run on every commit is frustrating, and any minute we can optimise to improve that experience stacks up, increases productivity and more importantly, saves developer’s time.</p><h3>Jenkins</h3><p>Ah, Jenkins. We all love to hate it, but it’s without a doubt the most established open-source CI platform around. We rolled out our own Jenkins cluster running in AWS, as a third-party CI provider would be too expensive for our use case. For example, to be able to run CI for our main application in a reasonable amount of time, we use a machine with 72 cores and 144GiB of RAM. This allows us enough parallelisation and compute power for each stage. We run around 5,000 of these per month (our developers deploy changes to Udemy.com around 30–40 times per day).</p><p>The other pipelines we run account for another 8,000 builds a month, meaning we run around 13,000 builds per month, or nearly 700 per day, excluding weekends.</p><p>We don’t require branches to be up to date with master, as this would be extremely annoying when queuing for a release. Instead, we fast-forward merge any build into master, to make sure it’s working with the latest code. As the majority of developers work on widely different aspects of the codebase at any given time, conflicts and regressions not caught by CI are very rare. To reduce the risk, we expire any build that is older than 2 days (excluding weekends, so you don’t have to re-run CI on a Monday morning, eurgh). This means that a pull request is guaranteed to be working against the latest master from at most 2 days ago. Any important large changes to the codebase involve broadcast communication to developers that they will need to rebase master, and we have the ability to set all builds as failed, prompting the developer to push a new commit (therefore fast-forward merging it into master on CI) or for them to rebase with master for the build to pass.</p><p>In our experience, there is one major limitation to Jenkins — the UI. As a feature developer at previous companies, the experience in debugging my builds was frustrating and confusing. As an engineer on the Build team at Udemy, the experience in configuring pipelines didn’t differ — it too was frustrating and confusing. At the time of writing this post, we have 63 pipelines defined in Jenkins and managing them all through the Jenkins UI wouldn’t be a pleasant experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iv03oCDGwDbwJpfXqNEzxA.png" /><figcaption>The configure Jenkins page, with a confusing form for configuring <strong>every </strong>plugin you have installed</figcaption></figure><p>Note — outside of our main application’s build configuration (which is massively complex, with parallel steps, steps that rely on other parallel steps to finish, etc.) we offer a feature we call “<strong>generic pipelines</strong>” to any development team.</p><p>We try to make as few assumptions about what these pipelines do as possible, and only require there to be a Makefile with build test and deploy targets (the deploy target is only run on merges into the default branch, such as master). Our agents are very barebones, so we encourage the usage of Docker to provide repeatable environments no matter where they’re run. We scale these agents automatically depending on demand and usage, so they’re not idle for too long before being destroyed. This also enables us to ensure that the host machines aren’t affecting builds (nothing is perfect, and artifacts or processes that we can’t account for that may have been left behind by previous builds could possibly affect future runs) — all machines are terminated and replaced within 24 hours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ze473B_XSgHD4ChR3JG9nA.png" /><figcaption>It’s not pretty, but it’s useful</figcaption></figure><p>We didn’t always have 63 pipelines. When I joined Udemy in January 2019, we had 18. These were defined in a Job DSL file, with a pipeline specific array that we’d have to update every time a team wanted a new pipeline created. It looked something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5a8301c4df966b3697553d22596cb2e/href">https://medium.com/media/e5a8301c4df966b3697553d22596cb2e/href</a></iframe><p>This wasn’t scalable, both in terms of adding new configurable properties of a pipeline, and adding new pipelines. We ended up blocking teams that had to wait for us to create their pipeline and run the Job DSL for them. We hate blocking teams — we consider our work best done when they don’t need to think about us at all.</p><p>Like most engineers, we love to automate anything that’s repetitive. Building on the groundwork from my team, and my experience with frontend development, we decided to create a portal that allows developers to self-serve when it came to creating pipelines.</p><h3>Build Portal</h3><p>We created the Build Portal, an internal service where all of our developers can go as a reference point for anything related to our team and what we offer within Udemy.</p><p>The original purpose of the Build Portal was to allow anyone to create and manage pipelines. Through many iterations, it’s evolved to a lot more than just that — we will explore how it enables developers at Udemy to not only manage pipelines, but also give a better UI around their builds, and ideally never have to look at Jenkins again.</p><h4>Pipelines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3GyRe9nN3TChxNgn5wvHtg.png" /><figcaption>The page to view all the configured pipelines and what repository they are for</figcaption></figure><p>In addition to allowing developers to create pipelines, we enhance their functionality through configurable options on top of them. We found the GitHub plugin with Jenkins to be very difficult to work with, especially with large repositories, so we removed it and now kick off builds ourselves, using GitHub’s web hooks instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6xC_RBa2g5zt0Kd4VPTC1g.png" /><figcaption>We keep an audit trail of who made changes to a pipeline, as we allow anyone to edit any of them</figcaption></figure><p>Looking at the screenshot above, there are a few features we offer that enhance our Jenkins experience.</p><ul><li>We allow rules for kicking off builds if there are changes inside or outside certain directories in a repository (useful for mono-repo builds)</li><li>We can control how long Jenkins keeps the logs for the builds</li><li>We allow a different “deploy branch” such as release that will run the deploy stage of the generic pipeline</li><li>Different jobs can run on different node labels, which map in our automated Jenkins configuration to a different type of machine (we have all of our infrastructure as code, and can destroy and recreate Jenkins as many times as we want)</li><li>The messages for builds passing or failing can go into a Slack channel if desired</li><li>Although not shown here, pipelines can define parallelisation for their tests</li><li>We can also control which pipelines run at the same time. We can configure it so if two or more pipelines try to run at the same time, the build will automatically fail. This helps with our mono-repo — we don’t allow changes to multiple services at once, and each service has its own pipeline, therefore we fail the build if a pull request is out of scope and edits two services at once</li></ul><h4>Builds</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M9XNq_t_LjHUbqCATc_CUw.png" /></figure><p>When opening the Build Portal, we get straight to the point and show users their own builds. We include the status of the builds in this view, so it’s easy for them to get a glance at how their runs are doing without having to dig down into them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ipb3Y35ytKfRfo852TqKxA.png" /><figcaption>Note that I don’t do this much work, I’ve replaced our developer’s profile pictures and names with my own to protect their privacy</figcaption></figure><p>There’s also the option to see all the builds that are currently running, and a brief history of builds over the last few days. You can also refine by the specific pipeline you’re looking for. We archive builds after three days, so developers can still view historical builds when clicking on the details link in GitHub, but we don’t pollute the main view with the thousands of archived builds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bT2j0mDnAZA3PPU7nLVUlQ.png" /><figcaption>An example view of a build that users are shown when clicking on “Details” in their GitHub pull requests</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cowisjnxlx4-dpP9SpFiOw.png" /><figcaption>Clicking on an item in the flow diagram will show/live-stream the relevant logs for that section</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhCo4T36HbDBW51X0J-_YA.png" /><figcaption>The steps that have failed are marked in red, so developers can tell where their build went wrong at a glance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*HqMah0HgYWZdrBom54kdrg.png" /><figcaption>We also show if your branch has merge conflicts</figcaption></figure><h3>Improving developer user experience</h3><p>We’re huge believers that developers shouldn’t have to browse through logs to figure out why a build has failed, so we are constantly creating deeper integrations to interpret results from the builds, to then be displayed in the Build Portal.</p><p>For example, when an integration test fails, instead of having to browse through the logs (and there are a lot of them), we include a test failures tab in the Build Portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6zg0cgJQoBBt2R_Wr2xWmw.png" /><figcaption>The test failures tab also provides useful instructions on how to re-run the tests locally</figcaption></figure><p>We understand that tests can be flakey, and it’s annoying to have your build fail because of a flakey test related to code that you haven’t touched in your pull request. We offer the ability to only retry the failed steps of a pipeline, or retry all of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D7r5nj_BfybfGpy9LkiKmQ.png" /><figcaption>Code coverage failure reports are integrated into the build page</figcaption></figure><h3>How does it all work?</h3><p>Although we tried, we weren’t able to use the Jenkins API to provide the same level of data and real-time updates that we wanted to implement in the Build Portal. We created our own backend with a database that keeps track of builds and their statuses. From the Jenkins side, we interact with it only where needed, e.g. grabbing the logs to display to developers and to kick off builds.</p><p>When it comes to updating our own database to reflect the status of the build, we have configured Jenkins to push the status updates into a RabbitMQ queue, which are then processed by a worker that saves the results to the database, as well as publish an event for any page that’s currently subscribed to the results. We chose this route to provide redundancy — if for any reason the Build Portal’s API went down, we wouldn’t lose status data and potentially provide incomplete/incorrect build results to our developers. If the worker goes down, the messages would then be processed by its replacement.</p><p>The frontend is built with <strong>React</strong>, and the backend API is a custom built framework on top of <strong>Koa, </strong>utilising <strong>GraphQL</strong>. We use Apollo’s server core to run the query (no need to reinvent <em>every</em> wheel) and then our framework takes it from there.</p><p>GraphQL subscriptions allow us to disconnect the worker processing messages from the server. Whenever an update to a build happens, the worker publishes the result into Redis, which notifies the server, passing on any relevant data to the browser through a WebSocket connection that has an active subscription. This means that the UI updates in real-time, removing any need to refresh the page to see the latest information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ygIMOlGg8eOqgNr-xzdU7A.png" /><figcaption>A simplified overview of the architecture behind the Build Portal</figcaption></figure><p>GraphQL allows for faster frontend development, as the ability to query what you need may already exist within the schema and require no server changes. Therefore, client and server changes can be rolled out at different times. We also provide the ability for developers to create API access tokens, so they can query historical build data, or pull their build statuses into their editor, or terminal, etc.</p><p>The perk of it being an internal tool is being able to use more bleeding edge technology. We use the latest experimental builds of React and Relay, meaning we can test out React’s new concurrent mode, and provide feedback to our main frontend team about the direction React’s heading towards. There isn’t a single React class component — everything is a functional component using the React hooks API, all written in TypeScript.</p><p>We provide a Relay-compliant GraphQL API through a custom backend framework that takes our GraphQL schema (defined in TypeScript, utilising helper functions to auto-create the Payload, Input and Node generated types for the mutations and subscriptions) and maps the resolver functions to a dependency-injected controller based approach.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f3747a9e9cb016bad2d5a71fa7be464/href">https://medium.com/media/2f3747a9e9cb016bad2d5a71fa7be464/href</a></iframe><p>We also provide caching during queries and remove any inefficient queries that may have to grab either the same record or different records from a table multiple times, resulting instead in one query to the database to retrieve all the records needed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63250025084740eed32ba3ee9ef4e046/href">https://medium.com/media/63250025084740eed32ba3ee9ef4e046/href</a></iframe><p>This query fetches the builds for the main page we saw above. However, for each build, we need to query the pipeline to get the nice display name to present to the user.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62f70978e3de72ef6dd36f51ff873754/href">https://medium.com/media/62f70978e3de72ef6dd36f51ff873754/href</a></iframe><p>Notice how we’re selecting the pipeline for every item in the list. This could be extremely inefficient as we’d be firing off a query to fetch the pipeline for each build. To counter this, we figure out exactly what pipelines are being requested, create an array of their IDs, remove any duplicates, and fire off only one query to the database for this connection — doing a SELECT ... WHERE pipeline.id IN [1,2] where 1 and 2 are the ids of the pipelines that are being requested. We also take advantage of GraphQL’s ability to select the fields you want in your query, and map that to our database queries — only selecting the necessary data and reducing query times massively.</p><p>We try to make the Build Portal as fast as we can. We wrote a custom router that prefetches the route chunk when a user hovers over a link (we code split every route out, so the initial page load is quick). We then utilise the preloaded queries that Relay experimental offers and preload the data for the route on mousedown. As a result, the data is loaded before the browser starts to navigate to the next page, and is almost always available at the first render of the route components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VwVaodFJh7sWqmEBaxYpWw.png" /><figcaption>The light theme for the Build Portal</figcaption></figure><p>Oh, and it goes without saying, we offer both a dark and light mode, depending on system preference, but it can be overridden if you prefer to always use one.</p><h3>Tracking build metrics</h3><p>As I mentioned above, build times are an extremely important thing to our team. We create custom APM traces for our builds, so we can monitor the 95th percentile build times in Datadog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFAxHPcogOT_Kxs1Il20Cw.png" /><figcaption>APM traces allow us to see when each stage has started, and how their times change. It also gives an overall view as to what is running at the same time, and where our bottlenecks are</figcaption></figure><p>We also have an extensive dashboard so we can see at a glance how well our CI platform is doing, with some custom metrics, and pulling in metrics from our custom APM traces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*AUmzMtKBNr6YmmeIqYtDRw.png" /><figcaption>Viewing the average wait in seconds a build has to wait for a node to be available to run tests — we tag this based on the node type being requested</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*t0L7wdCaUCnobINc1an-2A.png" /><figcaption>We track every step of every build of every pipeline so we can pinpoint if any changes have slowed down the build time significantly. These are the results for our main application’s CI runs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*SHs98MJ1uEEtLQpIoQQp2Q.png" /><figcaption>We also track the slowest integration tests, so teams can fix and improve the speed of the builds</figcaption></figure><h3>Slack integrations</h3><p>We also provide automated rich Slack comments for users with their build results. From these messages, you’re also able to retry builds from within Slack. This way you don’t have to open up the Build Portal if you’re confident your build failed because of a flakey test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*y-IseijwQ6idNKYOswxw7Q.png" /><figcaption>Beep is our internal bot for providing information about builds</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*NkPV0YB_-S7p_Z354mlNkA.png" /><figcaption>Clicking on “Retry Failed Steps” or “Retry All” will kick of a build and update the message to tell you so, without having to open a browser</figcaption></figure><p>@here — the enemy of most people using Slack, seconded to @channel. We realise that, but we notify the relevant channel for the pipeline of any deployment branch failures. It’s important to keep master healthy, as other developer’s builds will fail because of it (due to the fast-forward merging).</p><h3>Final notes</h3><p>There are a lot more features to the Build Portal that I haven’t gone over in this article. It’s constantly evolving and is one of the most used internal tools at Udemy, and listing out all the features would make this article too long. By creating the Build Portal, we’ve improved productivity within our engineering teams, allowing them to debug builds faster, create pipelines themselves and improve their overall developer experience. We have a lot more planned for the Build Portal in the future and will share our journey along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*Zmb9dBO3qi-LCGWeRV61kA.png" /><figcaption>We love feedback!</figcaption></figure><p>We constantly strive to make the developer’s lives at Udemy easier in any way that we can, and remove any dev-related pain points or blockers that they may have during their normal work day.</p><h3>Author</h3><p><a href="https://twitter.com/rynclark">Ryan Clark</a> is a Senior Software Engineer on the Build team at Udemy. He works on a variety of different tools such as the Build Portal, CI, and development environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9722e98a4208" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udemy-engineering/how-udemy-does-build-engineering-9722e98a4208">How Udemy does Build Engineering</a> was originally published in <a href="https://medium.com/udemy-engineering">Udemy Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>