<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maria Gullickson on Medium]]></title>
        <description><![CDATA[Stories by Maria Gullickson on Medium]]></description>
        <link>https://medium.com/@maria.gullickson?source=rss-86bcc9f00470------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Ju4Yt3NR43KktmKT.</url>
            <title>Stories by Maria Gullickson on Medium</title>
            <link>https://medium.com/@maria.gullickson?source=rss-86bcc9f00470------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:47:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maria.gullickson/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[API Roles]]></title>
            <link>https://medium.com/carmax-engineering-blog/api-roles-aec7999c095c?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/aec7999c095c</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[sdui]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[bff]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Fri, 06 Mar 2026 14:57:14 GMT</pubDate>
            <atom:updated>2026-03-06T14:58:20.447Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been thinking a lot lately about the roles various APIs play in our system architecture, and why it’s important to understand those distinctions. Most of the APIs that power <a href="https://www.carmax.com/">carmax.com</a> fill one (or more) of four basic roles:</p><ul><li>Data Access Layer</li><li>Business Logic Layer</li><li>Server-Driven UI</li><li>Backend For Frontend</li></ul><p>In this article, I’ll talk about what each of those is and how we use them. Understanding these roles helps us reduce duplicate code, lets product teams focus on their areas of expertise, and allows us to scale to meet our customers’ needs.</p><h3>Data Access Layer</h3><p>A Data Access Layer (or DAL) is an API that provides consumers a way to manage and query for data. It abstracts away how and where that data is stored.</p><p>These are classic CRUD APIs. They expose data in a straightforward, predictable way. For example, CarMax has an API to provide details about all of our store locations, and an API to provide details about all of the vehicles we have in inventory.</p><p>These APIs are generally not hard to build and maintain, but they are vital, and underlie everything we do. DALs establish consistency and make it easier to evolve data stores with minimal impact.</p><h3>Business Logic Layer</h3><p>A Business Logic Layer (BLL) is an API that sits on top of one or more DALs and centralizes the business logic and rules around how an application behaves.</p><p>One example of a BLL at CarMax is an API that manages vehicle transferability. Customers can get most vehicles they’re interested in transferred to their local store, regardless of where the vehicle is. But there are some cars that can’t be transferred at all, or can’t be transferred to particular locations, for various reasons. We can’t sell vehicles with incomplete titles at stores in certain states, so we wouldn’t transfer there. We may be temporarily unable to transfer vehicles out of a store in an area having a major weather event. This API centralizes all of those business rules, so every experience can just get a simple Yes/No answer without needing to keep track of the rules.</p><p>Business Logic Layers serve a critical role. Without them, you can find your business logic rules duplicated across systems and experiences. This duplication very rarely results in them being implemented exactly the same everywhere. It also makes it very hard to keep them all updated as the rules evolve over time.</p><p>Beyond saving you from managing duplicate code, there’s another important reason to have BLLs. There’s a lot of know-how required to run a large company like CarMax. There’s no way everyone can understand all the information, best practices, legal requirements, etc. of every area of the business. Centralizing logic means that only the team owning that BLL needs to really understand those business rules. Other teams can simply consume the output. This gives teams more ability to focus deeply on their own domains to provide the best possible experience for our customers.</p><p>In summary, BLLs reduce duplication, keep domain logic consistent, and let teams build confidently without needing deep cross-domain expertise.</p><h3>Server-Driven UI</h3><p>A server-driven UI (SDUI) is an API that controls the structure and content of the user interface, rather than letting the client define a fixed layout. Whereas the BLL decides what’s allowed to happen, an SDUI decides what the UI should show and how to show it. SDUIs do vary in how detailed they get on “how to show it”.</p><p>We’ve got a small amount of SDUIs at CarMax and are starting to move towards more. Ours are not overly prescriptive on the exact layout details. Our SDUIs provide domain-specific UI structure (e.g., an ordered list of widgets to show, along with their data parameters), while clients still own visual presentation and interaction patterns.</p><p>One of our first SDUIs came about three years ago when we rewrote the API used for retail vehicle search. The old version of the API let you pass in some search criteria and get matching vehicles and metadata back. It did a great job. But it also meant that people searching in our Android or iOS apps got a different experience than they did on our website. They might see a different set of search filters and sort options, for example. This causes even bigger problems with our saved search functionality. You might save a search on the web that won’t work the same way when you load it in a mobile app.</p><p>The new version is an SDUI. The API tells the clients exactly what filters to show, for example. It includes details like what increments to use for numerical filters (e.g., you can select Price in $1,000 increments), what order to show the filters in, how to group the long list of Features into categories like “Driver’s Assistance”, that MPG should allow users to select a minimum but not a maximum, etc. When we want to add a new filter, we simply update the UI, and it shows up everywhere.</p><p>Our SDUI does not go down to rendering details like what font to use, how much space each filter should allow, whether to use drop-downs or radio buttons for single select, etc. Leaving those things up to the UIs allows them to better adapt to different screen sizes and device capabilities, areas where those developers have expertise. We focus on centralizing the UI decisions that fit in the vehicle search domain.</p><p>We’ve had great success with this in search, and are currently working on expanding the idea to more areas of the site. SDUIs have given us a consistent experience across interfaces, and have also allowed us to iterate rapidly on experiments, and make our user interfaces more dynamic. It’s important to be aware, however, that this tight coupling between the API and the frontend requires great care to avoid breaking changes entirely or version the API thoughtfully when they are needed.</p><h3>Backend For Frontend</h3><p>A Backend For Frontend (BFF) is an API that is optimized for a single user interface, aggregating and shaping data for that client.​ BFFs orchestrate backend calls, map data to a particular UI’s needs, and absorb complexity that would otherwise live in the client. They can make assumptions about device-specific features in ways that other APIs can’t do safely, and can track a user’s context. They can act as a security layer between your UI and your backend APIs, centralize logging, and provide fault tolerance. By consolidating calls and serving only the data a given UI needs, BFFs often dramatically improve performance, which can be especially important for mobile devices.</p><p>BFFs are useful everywhere, but especially for mobile apps. Deploying changes to a mobile app can be a slow process, going through app store procedures. Even after a release, not all of your users will install the new version. But if you can move as much logic as possible out of the app and into a BFF, you can deploy updates to the BFF as often as you want, and those changes will reach all customers right away.</p><p>At CarMax, we use BFFs a lot, both for our mobile apps and for targeted sections of our website. They have become a core part of how we build software.</p><h3>Playing Nicely Together</h3><p>Having read all this, you might be picturing a beautiful layered system, where a user engages with a UI, built on a BFF, built on an SDUI, built on a BLL, built on a DAL, built on a database. Sometimes this is exactly the right architecture.</p><figure><img alt="A nice neat looking layer cake, with one layer in each color of the rainbow." src="https://cdn-images-1.medium.com/max/894/1*_9Yd8ActIcQ3-mvWf5j0BA.png" /></figure><p>But in practice, systems vary. The goal isn’t to enforce layers everywhere, but to understand the purpose of each type of API so you can use the right tool(s) for the problem at hand. You may find that you don’t need a BLL or an SDUI in some areas. Or it may make more sense to combine your BLL with your SDUI or with your DAL.</p><figure><img alt="A cake with all the colors of the rainbow swirled together throughout." src="https://cdn-images-1.medium.com/max/882/1*svFbhJWmowpR-LYZBDWjlg.png" /></figure><p>Maybe your client is pretty simple and maps cleanly to your domain APIs, so a BFF wouldn’t add value. Maybe you’re quickly iterating on the UI independent of backend deployment, so an SDUI wouldn’t be helpful. Or you might be working on something so small-scale that it doesn’t really warrant all those layers, and you just want one API.</p><figure><img alt="A cake that is consistently purple throughout." src="https://cdn-images-1.medium.com/max/892/1*y2RtSdm5_AqWXHVi54iY7Q.png" /></figure><h3>How Do I Choose?</h3><p>There aren’t strict rules about what to use where, but some guidelines can help you make a good choice for your use case.</p><ul><li>Choose a DAL when you need clean, direct access to data.</li><li>Choose a BLL when domain rules are shared across products.</li><li>Choose an SDUI when multiple clients need consistent UX behavior.</li><li>Don’t choose an SDUI if the UI needs to iterate rapidly, independent of the backend.</li><li>Choose a BFF when a specific client needs optimized aggregation or device-specific logic.</li><li>Don’t choose a BFF when the client is simple and is a clean match to your domain APIs.</li></ul><p>Treat these API roles as tools in a toolbox. You don’t always need all of them, but knowing when to use each one helps you build systems that are flexible, maintainable, and scalable. At CarMax, we have a large enough ecosystem that we’ve got a bit of it all. Some areas have the nice, neat layers, and some areas skip layers or blur the lines. Once you know what’s possible, you can design the best architecture for your system, and be ready to evolve it as your needs evolve.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aec7999c095c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/api-roles-aec7999c095c">API Roles</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Search Primer]]></title>
            <link>https://medium.com/carmax-engineering-blog/azure-search-primer-f43ceea33ea0?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/f43ceea33ea0</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[azure-search]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Wed, 04 Mar 2026 19:44:27 GMT</pubDate>
            <atom:updated>2026-03-04T19:46:18.163Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_THge2MLZhr77TsM2ds4w.png" /></figure><h3>Introduction</h3><p>This article is intended to give an introduction to <a href="https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search">Azure AI Search</a> (formerly known as Azure Cognitive Search, formerly known as Azure Search), and how to use it to filter your data. This is focused on using Azure Search for filtering, rather than text searching.</p><p>In particular, this article will discuss the following:</p><ul><li>Creating a new search index</li><li>Populating the search index with your data</li><li>Filtering a large body of documents on specific fields</li><li>Faceting to build a UI for that filtering, including showing field values to select</li><li>Sorting results</li><li>Managing the search over time, through schema changes</li></ul><p>The following are out of scope:</p><ul><li>Text Search</li><li>Synonyms</li><li>Scoring Profiles</li><li>Vectors &amp; KNN Search</li></ul><p>I also don’t really cover <a href="https://learn.microsoft.com/en-us/azure/search/query-simple-syntax">how to query Azure Search</a>, as that is well-documented and straightforward.</p><h3>Defining Your Search Index</h3><p>The first thing to think about is <a href="https://learn.microsoft.com/en-us/azure/search/search-how-to-create-search-index">your search index definition</a>. The main thing you’ll need for this is the list of fields it includes. <a href="https://learn.microsoft.com/en-us/rest/api/searchservice/supported-data-types">Azure Search supports a pretty complete set of field types.</a></p><p>In addition to defining the type of the field (String or Collection of Doubles or whatever), there are a handful of other things to consider for each field. First, a field can be a Searchable field or a Simple field. A Searchable field should be used if you’re allowing free text searching over your documents and want to include this field in what might match. Free text searching is out of scope for this article, so I won’t go into much more detail on that.</p><p>Whether you choose a Simple field or a Searchable field, there are a few other field attributes to consider:</p><ul><li>Hidden — Use this if you don’t want the field to be returned with the rest of the document in query results, but need it stored for calculations like filtering and sorting.</li><li>Filterable — Use this if you need to filter documents returned based on this field. This is distinct from being Searchable, as Filterable is used to do exact matches or numerical comparisons on a single field at a time. (See below for a gotcha with filtering.)</li><li>Facetable — Use this if you need to get a list of all possible values assigned to a field, including counts for each value. If you’re building a UI, this can be super helpful to create those filters.</li><li>Sortable — Use this if you’d like to be able to sort the result set on this field.</li></ul><p>You’ll also need to pick one field to be the document Key. This is required, and must be a string field containing an identifier that is unique across all documents in the index. For example, you might use the storage path if your data is coming from blob storage, or the ID field if your data is coming from Cosmos DB, assuming those values meet <a href="https://learn.microsoft.com/en-us/rest/api/searchservice/naming-rules">the character requirements</a>.</p><p>Beyond fields, there are various other things you may include in your index definition — normalizers, vector search definitions, scoring profiles, suggesters, etc. But those are out of scope for this article.</p><h3>Getting Data Into Your Search Index</h3><p><a href="https://learn.microsoft.com/en-us/azure/search/search-what-is-data-import">There are basically two options for how to get data into your search index</a>: doing REST calls to directly update Azure Search, or using a data source and an indexer. I’ll explain what each is, and then lay out the pros and cons of each approach.</p><p>Using the REST API directly is pretty straightforward. The API has an endpoint that will take a batch of actions to add, update, or remove documents from the index.</p><p>The alternative is to set up a Data Source, which is a connector to something like Cosmos DB, blob storage, Azure SQL, etc. Then you create an Indexer, which replicates changes from your Data Source into your Index. The indexer runs on a particular schedule, looking for changes periodically. The most frequently it runs is every 5 minutes.</p><p>As I mentioned, there are trade-offs between the two approaches:</p><ul><li><em>To use a Data Source, your data structure in the source needs to be nearly identical to the structure in the search index</em>. If you want to do things like convert a dictionary to a list, or convert a numerical price to a displayable string, you need to make that change in your data source. With a REST call, you can simply map the data appropriately before sending it to the Azure Search API.</li><li>However, <em>Data Sources are more robust to adding new fields</em>. If you have extra fields in your data source that are not present in your index, they will be quietly ignored. If you try to send extra properties to the API, it will throw an error. This means you need to be very careful when adding new fields that the code to reference them is not pushed out before the index has been updated to include them.</li><li><em>With REST calls, updates appear nearly immediately</em>. With an Indexer, they appear on the next run, which is every 5 minutes at best. And an Indexer is more prone to getting backed up if there are a lot of changes in a short period of time.</li><li><em>With Data Sources, building a new index version from scratch is quicker and simpler</em>. (This comes up more often than you might think, as we’ll learn below when I discuss versioning indexes.) If you’re sending updates via API calls, you’ll need to send every document again after creating a new index version. Which means you’ll still need to have a copy of the data lying around or an easy way to recreate it.</li><li><em>With Data Sources, it can be easier to manipulate data for testing.</em> You can simply edit a document through the Azure portal in blob storage, or Cosmos, or whatever your data store is. Then wait for it to get picked up by the index.</li></ul><p>Our team used to use the REST API approach, preferring the immediate updates. But a few years ago, we migrated to using indexers with blob storage for a data source. We found these much simpler to manage, especially when it came to making updates to field definitions in the index. (When we first started using search, blob storage didn’t support soft deletes, which would have complicated the process for removing documents from the index. The move to indexers became more appealing when Azure added native support for soft delete in blob storage, making it easy to remove documents.)</p><h3>Versioning</h3><p>Index versioning is one of the more annoying things to deal with in Azure Search. Though, to be fair, it’s an issue with other search engines as well.</p><p>If you want to add a new field to your index, it’s simple and quick. You just update your index with the new field, and it gets added. This won’t actually populate the data on your existing documents, even if you’re using a data source that already has the data and an indexer. You’ll need to push an update to each document to make that happen. But you can do all that in place.</p><p>If you want to edit an existing field (e.g., make it Facetable) or remove a field, from an index point of view, that’s not allowed. You need to create a whole new index from scratch, then populate your index with all the data, then update your system to point to the new index.</p><p>If you’re using a data source, populating the index is as simple as running the indexer and waiting for it to get through all of the documents. If you’re sending documents directly to the Azure Search API yourself, you’ll need to send them all again to this new index. (Which also means you’ll need to have a copy of them ready to access in these cases.)</p><p>Updating your code to point to the new index is easier than it used to be, thanks to <a href="https://learn.microsoft.com/en-us/azure/search/search-how-to-alias">Index Aliases</a>. You can define a single index alias that your code always refers to, and this acts as a pointer to the current version of the index. When you’ve got a new index ready for use, simply update the alias to point to the new index.</p><p>Index aliases are in Public Preview, not Generally Available yet. But they’ve been out there for years and are very stable. Our team uses them and hasn’t had any issues. The only catch is that if you use their client SDK, you have to use the beta version of the NuGet package.</p><p>So for us, the index update works as follows:</p><ol><li>Try to update the index in place. If you’re simply adding a field, this can happen in place and you’re done.</li><li>If the index update failed, create a new index with the new definition. We just append a timestamp to the index name to ensure they’re unique.</li><li>Wait for the indexer to run through all of the documents. When it actually stops for that 5 minute rest between runs, you know it’s done.</li><li>Update the index alias to point to the new index.</li><li>A separate process we wrote will identify any indexes with old timestamps that aren’t referenced by the alias anymore and remove them. It does this only after a period of time, when we’re sure we won’t need to quickly revert by pointing the index back at the old version.</li></ol><h3>Adding and Removing Fields</h3><p>I spoke above about how to manage the actual versioning of the index. But I also want to be clear on the care needed when changing the set of fields. You’ve probably got two relevant areas of your code here. One is the code that actually defines the index and does the updates. The other is the code that references the index to run searches and return results. When adding and removing fields, you need to be thoughtful about these parts and how they’re deployed.</p><p><em>When adding a new field</em>, you can’t reference it to select, filter, facet, or sort results until it already exists in the index. This means that you can’t push out one update to production that includes and uses the new field. You’ve essentially got two choices:</p><ul><li>Push two separate updates. The first updates the code to create your search index. After pushing and running this, push a second update with the changes to reference the new field.</li><li>Update your index before deploying. This would mean you have to run the index update from somewhere other than your production code base.</li></ul><p>In our case, we use <a href="https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots">deployment slots</a> for blue-green deployments. So we deploy all our new code to the staging slot together. Then we run the index update from the staging slot, before swapping the new code into the production slot. If you are using an indexer, this does the trick nicely. If you are sending updates directly to the Azure Search API, you could have updates processed after building the new index, but before swapping code live, that will fail to populate that new field.</p><p><em>When removing a field</em>, you need to stop referencing it before you update the index to remove it. This is easier, as you can push all of your code out at once, and then update the index after.</p><h3>Gotcha: Case Sensitive Filtering</h3><p><a href="https://medium.com/carmax-engineering-blog/another-azure-search-workaround-case-sensitivity-230426259138">This is covered in a previous article</a>, but I wanted to ensure that it doesn’t get overlooked. Filtering in Azure Search is case-sensitive. If you have a car with Make=Toyota, a search for “toyota” or “TOYOTA” might find it, but a filter for the Make field equal to “toyota” or “TOYOTA” won’t. That link describes your options, but if you want a field to be filterable and have a nice user-friendly display version, you’re going to need two copies of the field.</p><h3>Conclusion</h3><p>All in all, we’ve had a lot of success with Azure Search. There was definitely a learning curve getting started. And <a href="https://learn.microsoft.com/en-us/azure/search/whats-new">it’s worth keeping an eye on what’s new</a>, as sometimes it releases features that can make your life easier. But we created some reusable tooling that we employ across several different search indexes, which has made it a lot easier to work with. We generally only need to update that if there’s a new feature we want to take advantage of. Most of the time, it’s pretty smooth sailing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f43ceea33ea0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/azure-search-primer-f43ceea33ea0">Azure Search Primer</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No Coasting Allowed: A Playbook for Continuous Improvement]]></title>
            <link>https://medium.com/carmax-engineering-blog/no-coasting-allowed-a-playbook-for-continuous-improvement-c978723292f4?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/c978723292f4</guid>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[continuous-improvement]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Tue, 23 Sep 2025 15:31:10 GMT</pubDate>
            <atom:updated>2025-09-23T15:31:10.348Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*l9r8hlgJf_OxKTDj9ptXbA.png" /></figure><p>I am a software engineer. I get to work on cool and interesting problems. I build things that I am proud to put out in the world. I like coming to work because I get to spend most of my time doing fun design and development. I don’t spend much time or energy slogging through tech debt, cleaning up bugs, waiting for things to be ready, getting changes deployed, resolving a production issue, etc. And the teams I’ve worked on lately support each other to achieve common goals.</p><p>This doesn’t just happen. It’s not a coincidence. And it’s not just hiring people who are good at what they do. It’s teams being intentional about <em>how </em>they work, both individually and together. And it’s about not coasting once everything is working. You’ve got to think every day about if and how things might work better. The only way to make things good and keep them good is Continuous Improvement.</p><p>Continuous Improvement can be applied to almost any part of your job (or life). It could be about updating the tools you use, rethinking your meeting schedule, changing the way the team communicates, automating processes that don’t need a human in the loop, etc.</p><p>In this article, I will talk about when and how Continuous Improvement can happen within a team, with lots of examples. These examples are not recommendations for what your team should do. They are to demonstrate the types of changes that can be made. And they are all things that are pretty quick to implement and try out.</p><h3><strong>Opportunities abound</strong></h3><h4>Opportunity: Innovation Day</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*OP7VsbAacMKZss31SeSSUw.png" /></figure><p>An innovation day or hack day is a great time to improve the code. But it could also be a chance to build a new tool to support the team, or rethink how things work.</p><blockquote>Example: Most of the projects at our company’s recent Innovation Garage were cool new products and features. But one that stood out was the Silo Slayers, a project to establish a new format for small-scale community-building across the large engineering department. The year before, there was a project that redesigned how we train and onboard our new college hires in engineering, bringing more continuity and connection across all the training modules. That latter one was successfully put into practice this year.</blockquote><blockquote>Example: Our team has monthly innovation days. Recently, instead of doing something customer-facing, an engineer chose to jump into our team’s internal admin website. As the site has grown over the years, the navigation system has gotten harder and harder to use, especially for the product folks on our team who haven’t been involved in implementing it all. This team member rewrote the navigation system, and used it as a chance to try out a new development framework that made the whole process simpler.</blockquote><h4>Opportunity: Retro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*-hG-RMsCh5wiJ7AtWqAMbg.png" /></figure><p>Retro is a great insight into what is and isn’t working well for a team. Whether you are doing “Stop/Start/Continue” or “Liked/Lacked” or “Sunny Skies/Stormy Clouds” or some other format, there’s always a chance for people to surface things that aren’t working their best. Don’t just talk about them, but pick a couple of things that you can either act on immediately or assign an owner to improve.</p><blockquote>Example: My team changes QA engineers way more often than other team members. One QA resource mentioned during retro that she was unsure how to validate some cards, because she doesn’t have the context on the team’s work that the rest of us do. We made a new team norm to include full details on exactly how the card should/should not change behavior, and how to validate it. We are all responsible for verifying this when cards are refined, with our PM acting as a gatekeeper if we forget.</blockquote><blockquote>Example: My team holds retro meetings monthly. During the retro itself, I complained that I sometimes struggled to remember points I might have wanted to make earlier in the month. They don’t always come to mind during the short meeting time where I’m most focused on the last couple of days. We decided to publish the online retro board well in advance of the meeting, so team members could add notes to it over time, leading up to our discussion.</blockquote><h4>Opportunity: When the team’s direction is changing, or a new project is spinning up</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*L-_rrNhLHsAupRReg4bb7w.png" /></figure><p>If what you’re working on is about to change, there’s a good chance that changing <em>how </em>you work will be helpful too.</p><blockquote>Example: In a recent re-org, our backend team of four developers got two new frontend developers, and responsibility for part of the UI. As all of us had new things to get up to speed on, and the team size was larger, we needed something more. We added a weekly “Dev Chat” meeting with just the developers to get everyone up to speed and reduce “silos” of knowledge, as well as to consider upcoming technical decisions. Originally, we thought the meeting might be just temporary as the new team solidified, but we’ve found it valuable enough to keep on the calendar indefinitely.</blockquote><blockquote>Example: My team recently started planning a significant re-architecture. Some of the requirements from leadership include metrics about uptime, MTTD (mean time to detect issues), and MTTR (mean time to recover from issues). Those aren’t things we measure carefully today. We could wait until after the re-architecture, and then validate that we are hitting those targets. But it would also be nice to know if the re-architecture makes things better or worse than they currently are, and by how much. So it was decided that we should start tracking those metrics soon. The manager and dev lead were tasked with developing a strategy to consistently and accurately have access to this data.</blockquote><h4>Opportunity: When there’s a bug, an alert, or an outage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*FgTQdCCvirUwQ3v3EsIvKA.png" /></figure><p>When things have gone wrong, it is a great time to consider doing things differently in the future. Think about what went wrong and how it could have been avoided, caught sooner, and resolved faster. Are there missing test cases? Or test cases that didn’t run when they should have? Was there a breakdown in communication between developers and QA? Was the acceptance criteria unclear? Does the product need to be scaled up or out? Was the appropriate engineer alerted about the issue quickly? Did the alert have all the relevant details? Are the alerts just being too noisy?</p><blockquote>Example: Our team has a number of backend processes that run on a schedule. Some run daily or weekly, while others run every 5 minutes. If one of them fails, we get an alert. We found that the jobs that run every 5 minutes would have an issue every now and then, but the very next run would work and clean things up, and there was nothing we needed to do. Those alerts were noise. But if a once-a-week job failed, we wanted to know, because it wouldn’t get resolved for another week. And if an every 5 minute job failed several times in a row, that would indicate something we need to look at. We couldn’t disable the alerts. We updated the alert to only fire if more than 50% of the runs in the last hour failed. A single failure for a job that runs once an hour or less will trigger the alert, while a more frequent job will only alert on repeated failures. The noise from these alerts has been eliminated, so they only fire when we need to pay attention.</blockquote><h4>Opportunity: When a team member joins or leaves the team</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*YKXXChC-vZeBRmnAXK601g.png" /></figure><p>A new team member may come with great new ideas for how to improve the way the team works. I recommend giving them a couple weeks to get acclimated and then having a “re-norming” session to talk about any process updates that might be valuable. This is especially true if the new team member brings a new role to the team.</p><blockquote>Example: Our team did not have a QA resource for a while. When one was added, we rethought the columns on our board and the tags we use on cards to support that new resource and their work. We also added a new QA channel in our chat to discuss any issues related to cards going through QA.</blockquote><p>This can also be helpful to do after losing a team member, as team dynamics and best practices may shift again. For example, a team’s strong preference for calls over chat may shift as the team members change. Or the best time of day to hold meetings might vary, especially if the time zones represented have changed.</p><h4><strong>Opportunity: When you are waiting</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*Rx_BBpz4v5u0iF0zIRHq_g.png" /></figure><p>If you find yourself waiting around for the same thing regularly, think about how you might make the wait shorter, or gone entirely.</p><p>If you are waiting too long for a build or deployment to complete, look at ways to speed it up. What part is taking the longest? Could some steps be done in parallel?</p><blockquote>Example: We had a case where all of our builds were taking way longer than we wanted to wait — branch builds, deployments to our development environment for work-in-progress, as well as deployments to our QA and production environments when code was ready. We noticed that running unit tests took about 7 times longer in our build pipeline than locally, because our pipelines were computing code coverage rates. We removed code coverage from all of our regular pipelines, and set up a once-a-week job that computes code coverage. This allows us to keep tracking how we’re doing and trending over time without paying that cost over and over during the builds we’re actually waiting for.</blockquote><p>Maybe you find yourself waiting on a coworker to finish their piece of a task. How could you rearrange the work?</p><blockquote>Example: As a developer, you might be waiting for the designer to finish designing the interface for the new feature. You could schedule things so the designer is engaged earlier. Or maybe you could start working on the backend, even before you know what the frontend will look like. Or maybe you and the designer could pair up on some of the work, helping ensure that the designs are feasible and practical, and having some of that frontend already written as the design phase finishes.</blockquote><h4><strong>Opportunity: When you are doing something tedious</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*IJ2bbdQvbjmVhLVxE5JsuQ.png" /></figure><p>No one likes doing the same mindless task repeatedly. Every time I find myself doing something tedious at work, I think about how I could avoid this in the future. Could I write a script to do it for me? Could Copilot do this task on my behalf? More often than not, the answer is yes. And the time to automate is more engaging than the time spent doing that tedious task would be.</p><blockquote>Example: We’ve got a module in our code designed to try to understand a customer’s search terms and how they map to vehicle attributes. For example, “Blue” in “Blue Toyota” is talking about the exterior color, but in “Blue interior” is talking about the interior color. <a href="https://en.wikipedia.org/wiki/Hyundai_Genesis">Is “Genesis” a reference to a Make or a Model</a>? We’ve been working on overhauling that system. Past updates have required a lot of tedious, manual evaluation that never left us confident that we’d seen everything. Heading into the latest changes, we built a couple of reports. One evaluates how well it does on simple combinations of Makes, Models, and Trims in our inventory. The other evaluates how well it does on a wide variety of customer input that is often less precisely formatted. These reports run automatically once a week, sending us a summary of how the results compare to the previous run. This enables us to evaluate any changes we make without any extra work.</blockquote><blockquote>Example: While building out the reports I just mentioned, I needed to generate a test suite with hundreds of cases. After writing about 5, I was bored. So I wrote a query to get hundreds of real-life system inputs from our production logs. Then I wrote a Python script to execute those queries, examine the output, and generate corresponding test cases in our C# codebase. I still needed to review the test cases and make some tweaks here and there, but that was so much quicker than writing all those cases myself.</blockquote><h4><strong>Opportunity: When you are doing something that’s not your core role</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*yPSMU1wImoDXLWFMcDKGCA.png" /></figure><p>A responsible employee often feels compelled to fill in gaps. Maybe there’s a core meeting that often gets off track, so you step up to lead it. Maybe there’s a report that needs to be updated each week, and you start doing it to make sure it gets to your leadership on time. Maybe you’re working on a cross-team project, and you help coordinate to ensure everything goes smoothly. And then one day you realize you’re spending a significant portion of your time on these tasks that are important, but don’t quite fit the job description of Software Engineer that you love. Or maybe you notice a coworker taking on these items, and not having enough bandwidth to do their core job.</p><p>Ideally, you notice this disparity early, but even if we’re talking about things that everyone has relied on someone to do for months or years, that can still be re-evaluated. Consider each task individually. Is there someone else who would be better suited? Does your team have the right makeup of people? Could this be automated? Could it be handed over to an AI agent? Does the task need to be done the same way it always has? For example, could that report you spend an hour on every week take a different form that can be completed in 5 minutes and be just as useful? If the task really has to be done, and there’s no way around it, maybe you share the load. For example, maybe your team takes turns setting an agenda and leading that meeting, so no one person has to own it.</p><blockquote>Example: When our team first started up, we were focusing on a few of our largest clients, who had lots of requests of us. As time went on, these requests got hard to keep track of, and sometimes conflicted with each other. As the Dev Lead at the time on a team without a Product Owner, I was spending too much of my time trying to keep track and coordinate with stakeholders. I let the team know that I was struggling with it. Our Delivery Manager set up a spreadsheet to track all of our external requests that others could add items to themselves, and scheduled a bi-weekly meeting to discuss those requests with all stakeholders present. This ensured that we were building one solution that would meet everyone’s needs, and left me free to focus more on our development work outside of that meeting.</blockquote><h4><strong>Opportunity: When there’s something off with the agile board</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*DgR2wA6l6wKHBgD6LYp5pg.png" /></figure><p>There are a lot of things that might prevent cards from flowing smoothly across your board. Once you’ve recognized an issue, consider why, and what changes could eliminate the problem.</p><p>Maybe the backlog is growing over time. Cards are being added faster than they can be completed. Does the team need more resources or different resources? Are there cards being added that no one is really going to work on in the foreseeable future? Maybe you just need a regular backlog pruning session.</p><blockquote>Example: We noticed a growing number of cards that were blocked across most lanes of our board. Our team uses Kanban — developers can pull in any card that’s been refined. These blocked cards were cluttering things up, making it harder to see the cards that were really ready to go. Some were waiting on a dependency from another team; some couldn’t be worked until a particular date; some were waiting for a final decision on whether they should be done. We added a new “On Hold” swim lane to the bottom of the board and slid those cards down there, leaving them in their current lane, but out of the way. We make a point to look down there once a week or so during standup, so cards don’t linger, but they aren’t in the way every time we’re trying to look at current work.</blockquote><p>Maybe there are cards that were ready, but just sit there for a long time. Are the card’s details unclear, so developers don’t feel ready to pull them in? Are the cards too large, and need to be broken down? Are the cards working on something fragile that people are nervous to touch?</p><blockquote>Example: A previous team I was on noticed cards sitting for too long in the backlog, and introduced a new process. The DM searched for any “ready to go” cards that hadn’t been touched for over 30 days, and pulled them into a new lane to be discussed at standup each day. This gave developers a chance to ask questions or explain why the card wasn’t being taken. It helped resolve any issues with that card, as well as reminded us how to better define cards going forward.</blockquote><p>Maybe the backlog is getting low. This could happen because the designer or product manager has lower bandwidth lately, the developers had a breakthrough that helped move cards along, or because team resources changed. This is a great time to address tech debt. Developers can usually come up with things about the system that bug them, are hard to work with, or could be updated. Even if the path to fix them seems complicated, take advantage of the time when you’ve got it. Encourage all the devs to add some tech debt cards.</p><p>Maybe cards are piling up in one of the in-progress lanes — development, review, QA, etc. Think about what the bottleneck is and how to improve it. Do developers need to help out with some of the QA tasks to get things moving? Are cards being pulled in before they are really ready to be worked on, leading blocked developers to pull in more and more cards? Does the team need a new policy to help ensure that cards get reviewed in a timely manner?</p><h4><strong>Opportunity: When it’s been a while</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*drtaDR6TxIof0u5LshKA-A.png" /></figure><p>A periodic re-norming meeting (somewhere between quarterly and yearly) can be valuable, even if the team hasn’t significantly changed. Maybe there’s something that seemed minor a while ago that’s become more and more of an issue. Maybe team members work better in new ways as they gain more experience. Or maybe there’s a new tool that you could take advantage of.</p><blockquote>Example: It’s common to have someone send out notes after meetings. In most of my meetings, we’ve stopped asking a person to be responsible for note-taking now that Copilot can do that for us. In fact, I’m seeing notes published from meetings that previously didn’t have notes, because it’s so easy now.</blockquote><blockquote>Example: Our team had a re-norming meeting a little while ago where it came up that we were getting slower at reviewing each other’s PRs. Emails were getting lost in a sea of noise, and outstanding PRs were hard to find in GitHub. We added a new “Pull Requests” channel to our team chat that automatically gets messages about open PRs in a space that’s uncluttered with other things. This makes them much more visible, and gives us a place to add comments like “I’d love to get this out quickly if someone has time this morning” or “this is the PR that I mentioned in standup” with easily accessible links.</blockquote><h4><strong>Opportunity: Any time someone feels like speaking up</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*NJq-TP41j-ZFUKkC2UoOTA.png" /></figure><p>There doesn’t have to be a special trigger to suggest improvements. Sometimes a team member will just mention something that’s not working during standup, or start a chat thread when something comes to mind.</p><blockquote>Example: Our analyst was having trouble finding our team’s test in our analytics system, where data from all teams is mixed together. A new team norm to prefix our tests with our team name, so they’d alphabetically sort together, made this much easier.</blockquote><blockquote>Example: It’s pretty common for one member of our team to learn about a new tool, or a new feature in an old tool, and let the team know how it’s helping them. Recently, one team member pointed out that ADO Boards now support markdown. Another team member shared how prompting GitHub Copilot differently to generate unit tests worked better. Both of these are things that I hadn’t noticed but found very helpful.</blockquote><h3><strong>But how?</strong></h3><p>I’ve talked a lot about how to find friction points or opportunities to improve. Sometimes you may not know what change to make, how to make the change, or how to tell if it worked. Then what? In this section I’ll share a few tips and tricks.</p><p>One thing that helps you get started is tracking data. How many cards are in each column on the board on average, and how does this trend over time? How long are individual cards sitting in each column? How long do various steps of the process take — deploying code, running tests, refining cards, preparing demos, etc.? Are there any outliers taking up a larger percentage of time than they should? How much time do developers spend actually developing?</p><p>Another thing that’s important is to ensure that every person on the team feels empowered to speak up, with the understanding that their opinion will be heard and valued.</p><p>When you’ve identified an issue, the old practice of <a href="https://en.wikipedia.org/wiki/Five_whys">The Five Whys</a> might help you get to what you want to tackle (though it might not always be exactly 5).</p><blockquote>Example:</blockquote><blockquote>1. It takes way too long to set up new A/B tests. Developers are focusing on the tedium of configuring them rather than spending more time on actually building those new experiences to test.</blockquote><blockquote>2. Why? They have to be configured in 8 different places.</blockquote><blockquote>3. Why? We need Adobe Target, and the application configuration, and the controller code, and the orchestrator code, and the telemetry to all be aware of it.</blockquote><blockquote>4. Why? Tests have to be tracked or handled at each of those levels.</blockquote><blockquote>5. But? Maybe you could build a framework that handles the app configuration, controller, and telemetry. Then a test will need to be set up in Adobe Target, and the orchestrator updated to alter behavior based on it, and everything else will work automatically.</blockquote><p>Make sure you’re also paying attention to what’s working well. Maybe it’s a meeting that’s super helpful, or a chat channel that makes life easier, or the order you talk about work during standup. Note those and hold on to them, even as you make improvements.</p><p>Once you’ve got an idea, it’s not going to succeed if the team is split on whether to do it or not. Consensus is important. But remember, you don’t need consensus that “this is a great idea and it’s going to solve the problem”. You just need consensus that “this is something we can try for two weeks and see how it goes”. If you add a new column to your board and it doesn’t help, it’s easy to delete later. If you cancel a meeting, but things start slipping, it can be rescheduled.</p><p>After you’ve made a change, you can go back to that data to see if it helped. You can also poll team members to see if things feel better. Then decide if you want to stick with the change. If you find a win, big or small, make sure to celebrate it, encouraging the team to look for future wins.</p><p>The more you get in the habit of reconsidering your team norms and practices all the time, the more comfortable you’ll be that things can change. And remember — everything is fair game if it improves quality, efficiency, and team morale.</p><h3><strong>Conclusion</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*Af-mkaAyK8HI0lDdIzgmaQ.png" /></figure><p>The reality is that the “best” way to work is a moving target. You’ve got to be intentional about moving with it, and never settle for “good enough”. By doing this, you can ensure that you are efficiently producing quality work, and even enjoying it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c978723292f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/no-coasting-allowed-a-playbook-for-continuous-improvement-c978723292f4">No Coasting Allowed: A Playbook for Continuous Improvement</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering the Art of Frequent Releases]]></title>
            <link>https://medium.com/carmax-engineering-blog/mastering-the-art-of-frequent-releases-ca7d04021244?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca7d04021244</guid>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 13:39:25 GMT</pubDate>
            <atom:updated>2024-10-08T13:39:25.025Z</atom:updated>
            <content:encoded><![CDATA[<h4>Best practices for rapid and reliable deployments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*yc1kxJP5uq4q8VCGXtAF_A.png" /></figure><p>In software development, there are a lot of great reasons to aim for frequent, small releases, rather than collecting changes for a week or a month or a quarter and deploying all of them together.</p><p>Our team of four engineers has seen great benefits from this approach. It’s not unusual for us to have 5–10 releases in a day, most of which are to our one primary service.</p><p>In this article, I’ll briefly talk about some of the advantages of working this way as motivation. Then I’ll go over a variety of strategies that can help teams move toward releasing in quick, small chunks.</p><h3>Motivation</h3><p>Some of the benefits of frequent releases include:</p><ul><li>When a release causes a bug or issue in production, the change will still be fresh in the developer’s mind. This makes it easier to identify and fix the bug more quickly.</li><li>Once implemented, bug fixes can get out quickly.</li><li>Customers can start using the new features right away.</li><li>You’re able to get quick feedback to iterate on changes.</li><li>You get lots of chances to exercise your release muscles, making it easier to do.</li><li>You have lots of motivation to streamline the release process, making it even easier.</li><li><a href="https://dora.dev/capabilities/continuous-delivery/">Frequent releases have been shown to increase developer job satisfaction and reduce burnout.</a></li></ul><p>The releases being small adds a few more benefits beyond just the frequency:</p><ul><li>When there are fewer changes to look at, it’s easy to identify the source of any bugs or unexpected issues.</li><li>Reverting a change only requires reverting that change. When a release includes a bunch of changes at once, if one needs to be reverted, all the other good changes will get rolled back too.</li><li>A smaller proportion of releases contain bugs. When a release includes a lot of changes, the odds that one of them has a bug go up significantly.</li></ul><p>This quote from <a href="https://www.oreilly.com/library/view/accelerate/9781457191435/">the Accelerate book</a> sums it up nicely: “Reducing batch sizes reduces cycle times and variability in flow, accelerates feedback, reduces risk and overhead, improves efficiency, increases motivation and urgency, and reduces costs and schedule growth.”</p><p>You may not be able to get everything on this list perfectly complete on day one. But every step down this path will benefit your team and your customers.</p><h3>Case Study</h3><p>I am the lead engineer on CarMax’s Search Foundations team. We build the backend services that power the retail search experience on the CarMax website and mobile apps. We do a lot of different things as part of this work, but one kind of task comes up regularly.</p><p>When we have access to a new data point about vehicles, we consider whether it’s something we want to include in search. For example, we’ve recently added data about the vehicle’s door count, recent price drops, and vehicle size (e.g., “compact” or “midsize”).</p><p>There are two ways we could include the new data in the vehicle search experience. Both of these are indicated in yellow in the screenshot below for the identifier that a vehicle has had a recent price drop. We might simply return that data for each individual vehicle so it can be shown on the car tiles to the right. We might also add the capability for users to filter their results based on this data in the controls to the left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUctOmY_W5qpiHNomy9boQ.png" /></figure><p>When we add a new piece of data, there are a handful of things to do:</p><ul><li>Ingest the data from the source and store it on our vehicles.</li><li>Update our search schema to include the new data point.</li><li>Populate the data on all of the pre-existing vehicles.</li><li>Add the data to the information on each vehicle returned.</li><li>Add the ability to facet and filter on the new data point.</li></ul><p>Throughout this article, as I talk about different strategies to allow frequent, small deployments, I’ll use this type of change as a case study to talk about how we employ these strategies on my team.</p><h3>Strategy: Easy Deployments</h3><p>If it takes a lot of time or effort to deploy, it won’t happen very often. So the first set of strategies I’ll discuss are all centered around making it low-effort to deploy changes.</p><ul><li>Deployments should be fully automated, including triggering automatically when a change is merged to your main branch.</li><li>Use blue-green deployments with no service downtime to release changes.</li><li>Use a single pipeline that deploys changes to your QA environment, then (possibly after an approval step) to Production. This ensures that you know exactly what is going out, even if other changes were merged to your main branch in the meantime.</li><li>Use build notifications to ensure you know the state of the deployment.</li><li>Empower anyone on the team to deploy code at any time.</li></ul><p>We use all of these in the search service we own, and would use all of them for a change to add a new filter. In addition to these, we’ve got a couple of other custom tools to help make these changes fully automated. One tool automatically updates our search schema to match whatever is in code across all regions of the service. Another tool kicks off data updates for all existing vehicles so that new field gets populated. Each of these is a single button click that a developer can initiate whenever they’re ready.</p><h3>Strategy: High Confidence</h3><p>Before you can feel good about deploying all the time, you need to know that the code will work as expected without bugs. The primary tools that enable this will be no surprise to veteran developers:</p><ul><li>Code review process (e.g., PRs)</li><li>Unit tests</li><li>Integration tests</li><li>Acceptance tests</li><li>Performance tests</li></ul><p>Using these tools can be easier and be enforced by your tool set. For example, our GitHub repos are configured to require PR approval and passing unit tests before code is merged to our main branch. Those unit tests are automatically kicked off any time a change is pushed to a branch. The build pipeline automatically kicks off when code is merged to our main branch. However, it won’t deploy code if unit tests fail, and won’t swap it to the live slot in QA until integration and performance tests have passed.</p><p>These quality requirements are the responsibility of every developer on the team. This means that every change should include any appropriate test additions or updates. It also means that everyone who reviews a PR should validate that these are in place before approving the PR.</p><p>For example, with our case study of adding a new data point to vehicle search, I might include the following tests:</p><ul><li>Unit test: ingesting the new data point, including any normalization</li><li>Unit test: returning the new data point on vehicles</li><li>Unit test: returning the new filter details</li><li>Integration test: using the new filter returns the correct vehicles</li><li>Integration test: using the new filter updates filter state correctly to indicate selected values</li></ul><p>In most cases I wouldn’t spend much time testing how the new filter is showing up in UIs built by consuming teams. My automated tests are sufficient to give me the confidence I need. The one exception would be if it were a new, different kind of filter. For example, we introduced boolean filters such as “Has price drop” not long ago. Prior to that, all filters were either a list of string values (such as car models) or a range of numerical values (such as mileage). Because this new boolean filter type had a new behavior, I did some manual testing through the UI before declaring it done.</p><h3>Strategy: Focused Changes</h3><p>Small focused changes are much easier to validate and deploy quickly. There are a couple of things you can do to make your changes smaller.</p><ul><li>If necessary, break up cards into smaller chunks that can deploy individually.</li><li>Deploy components individually, even if they are ultimately meant to work together. Just because several cards contribute to the same larger feature doesn’t mean they all need to wait to be released together.</li></ul><p>By breaking a change into smaller, focused changes, you can start getting partial value from your work even sooner. You can use learnings from those early changes to inform how further changes are implemented. And if you need to revert one piece for some reason, other pieces might live on and continue to provide that partial value.</p><p>With our example of adding a new filter to search, I often break this task into 3 releases.</p><p>The first change is simply to ingest the new data and store it with vehicles. Once this change is out, I can run the automated process to update the search schema and populate the field on existing vehicles. At this point, no customer-facing changes have been made, but I can see if the data looks as I’d expect to ensure we’ve got good coverage across our inventory, and to get an idea of what values the field holds.</p><p>The second change is to start returning the field on the individual vehicles. This is a very quick change to make, and allows us to get that partial value of showing the data on individual car tiles.</p><p>The third change takes a bit more work, but completes the value of the new data. This is the change to add the filter to the request and response, and implement the logic so that the filter is actually applied in selecting the set of vehicles to return. How this filter works might be informed by the data I learned about in the first step. For example, if I were adding a numerical filter, looking at the spread of data might tell me if I want to allow customers to filter in increments of 100 or 500 or 1000.</p><h3>Strategy: Minimal Coordination</h3><p>If you have to coordinate your release with other teams, you can no longer release any time you want, and their process could delay your work. Ideally, you’d want to decouple your release from others whenever possible.</p><p>There are several things that are required to ensure that your changes aren’t tied to releases from other teams:</p><ul><li>Prefer non-breaking changes whenever possible. This ensures that customers don’t need to make updates to account for your changes.</li><li>Make any new functionality opt-in, leaving the default behavior unchanged.</li><li>Use feature flags to hide code that’s not ready to be used. This helps ensure that the code can be deployed at any time. When using feature flags, we like to set up our test suite to run with the flag both off and on. That way, whenever we decide to flip that flag, the new configuration will have been thoroughly tested.</li><li>Sequence releases so you don’t need to coordinate and deploy at the same time as another team.</li></ul><p>Adding a new property to the individual vehicles returned from search is always non-breaking. Customers can start using the new property anytime after it’s released, on their own schedule.</p><p>There are two different models for how clients (other internal development teams, often building customer-facing user interfaces) might integrate with the search service to implement vehicle filter options for customers. Both strategies allow us to deploy without having to coordinate with those clients.</p><ul><li>The first model involves filters driven by the search service. Our response object includes a list of filters to show, with a standard set of details about how to show those filters. Clients can simply show this set of filters without worrying about what they are. If we add a new filter, it will automatically start showing up to customers without any changes needed on that consuming software.</li><li>The second model involves clients choosing which filters to show, and simply ignoring the ones we return (or asking us not to return them). In this case, we can add a new filter if we want and it won’t impact those clients at all. If and when they want to start using the new filter, they can add it on their own schedule.</li></ul><h3>Strategy: Mitigate Risk</h3><p>There are a variety of ways one can reduce the chances that something will go wrong, and reduce the scope of the problems if it does. This goes hand-in-hand with High Confidence in helping you feel comfortable releasing at any time.</p><p>I think of these in three areas. Each of these is probably broad enough for one or more blog posts of its own, so I won’t cover all the details here:</p><ul><li>Ensure you’ve got strong monitoring and alerting.</li><li>Code defensively when interacting with other APIs or resources (e.g., databases).</li><li>Continuously improve on all of the above areas, or anything else you find slowing down your release process or gumming up the works. Many of the learnings above came from our own experiences, and looking at how things could work more smoothly for us. If something does go wrong, shore things up so it doesn’t go wrong again.</li></ul><p>Looking at our example of adding a filter, we make use of these as well:</p><ul><li>We’ve got an alert that fires if all recent updates set the same value for a given facet. For example, if every car updated in the last 24 hours has its VehicleSize set to “Compact”, we get an alert, as there should be other VehicleSize values represented.</li><li>When pulling in a new data point, if we aren’t able to get the data (either because the service to provide it is unavailable or because the data is missing), we’re still updating all the other data on the vehicle. If the new field is considered critical, we could choose to log its absence or even alert on the failure. But often, we’d rather show what information we do have about a vehicle, even if one or two data points are missing.</li><li>Many of our data points are carefully normalized to ensure that they have standardized display values, and that we aren’t showing malformed or inconsistent data to our customers. Even if our data source isn’t cleaned up enough for customer-facing display, we ensure that the data in our service is.</li><li>Continuous improvement applies in lots of ways. It’s what led us to make our search schema updates simpler and simpler over time, to the single button click they are today. It’s what led us to significantly expand our integration tests a couple of years ago when we decided they didn’t give us enough confidence. It’s what led us to add more build notifications so a developer knows as soon as their code has landed in the QA environment.</li></ul><h3>Conclusion</h3><p>I hope these strategies are helpful in moving your teams forward. I also encourage you to take a close look at how your team is operating to find any opportunities:</p><ul><li>Are there factors that make you less likely to release a change quickly?</li><li>Are there factors that make you release less frequently?</li><li>Are there factors that make you deploy changes in larger chunks all at once?</li></ul><p>This isn’t a one-time examination of your process, but something you should do continually. It’s always worth considering any improvements you can make to get you to that next level.</p><p>We’ve found that fast, frequent releases improve quality for our customers and make work easier for our developers, and setting ourselves up for this success has been totally worth it.</p><p>If you have any blockers that I didn’t address, feel free to leave a comment. Or if you have strategies you’ve already had success with that I didn’t include, I’d love to hear about those too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca7d04021244" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/mastering-the-art-of-frequent-releases-ca7d04021244">Mastering the Art of Frequent Releases</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Another Azure Search Workaround: Case Sensitivity]]></title>
            <link>https://medium.com/carmax-engineering-blog/another-azure-search-workaround-case-sensitivity-230426259138?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/230426259138</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[azure-search]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Mon, 05 Aug 2024 18:01:52 GMT</pubDate>
            <atom:updated>2024-08-05T18:01:52.421Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCw0FukmKNGYHt_8l9A05g.png" /></figure><p><a href="https://medium.com/carmax-engineering-blog/how-to-search-for-data-we-dont-have-6c393d0bc25f">I posted previously</a> about some work we had to do to get searches working for missing data in <a href="https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search">Azure Search</a>. This post is about another bit of functionality that didn’t work for us out of the box in Azure Search, what we considered, and how we got things working as we needed.</p><p>On the CarMax website and mobile apps, we tend to do a lot of fairly structured searches, rather than free-text searches. Customers are shown a collection of facets that list vehicle makes, drivetrains, features, etc. to choose from. These values are then used to filter down our current saleable inventory. For the most part this works as we want it to. But to make our API as usable as possible to internal customers, we wanted to make queries as forgiving as we could.</p><p>In particular, we wanted to allow clients of our API to search for Make=“Ford” or Make=“FORD” or Make=“ford” and get the same set of vehicles back. Unfortunately, this is not supported by Azure Search. A filter on a text field needs to be an exact match, case and all. Make=“Ford” would return the right results, but Make=“FORD” and Make=“ford” would return nothing at all.</p><p>In this post, I want to share the solutions we considered and the one we landed on, because this seems like an issue that is likely to impact other search experiences in other organizations.</p><h4>The Simplest Solution: Lowercase All the Things</h4><p>The most obvious way to work around this is to simply lowercase all of our data and all of our queries. This works great to filter down to the right set of inventory.</p><p>But we also want a pretty version of the data to display to users. We need this in two places. One is simply in showing data about individual cars, and the other is in the list of facet values we allow users to select from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQtC8P8yaUueRZthDOHEog.png" /></figure><p>We can’t simply convert those lowercase values to Title Case (capitalizing the first letter of each token) when displaying them. That wouldn’t work for all of our data. For example, it would produce poor results for vehicle models like Honda CR-V, Toyota RAV4 or Mazda CX-5.</p><h4>The Elegant Solution: Use Azure Search Normalization</h4><p>Looking at Azure documentation, I discovered <a href="https://learn.microsoft.com/en-us/azure/search/search-normalizers">normalizers</a>. Their documentation led me to believe they might have solved this problem for us.</p><p>You can assign a normalizer to any text field. This defines a method to normalize the field for indexing. You can define a custom normalizer if you need to, but one of the pre-defined ones is “lowercase”. Using this, your document is still stored with the original casing (e.g. “Land Rover”), but the normalized version (e.g. “land rover”) is what gets indexed. When filtering on this field, your filter text is normalized in the same way.</p><p>This successfully solves our filter problem. Now filtering for Make=“Mazda” or Make=“MAZDA” or Make=“mazda” will all return the same set of cars for us, the Mazdas. And because the primary document hasn’t changed, what’s returned on the vehicle to show the user still says “Mazda” like we want.</p><p>Unfortunately, it doesn’t solve the faceting portion of our use case. Those normalized values are used when getting facet counts. It makes sense why they did this. It’s to handle the situation of several documents with different casing. For example, let’s say there are 100 “Mazda” vehicles, and one “MAZDA” that snuck into our data somehow. When I get facet counts for the Make property, I want to get a single Mazda count of 101. To do this, they count by that normalized value of “mazda”, and so that’s what’s returned in my facets with a count of 101.</p><p>This means the normalizers won’t work for us. When showing the set of facet values for customers to choose from, we want them to see “Toyota” and “BMW”, not “toyota” and “bmw”.</p><h4>The Functional Solution: Double Down on Our Data</h4><p>What we ended up doing is having two copies of every string field that we want to be able to filter on:</p><ul><li>One field is used for filtering. This field is lowercased. It is created with IsFilterable=true, IsHidden=true.</li><li>One field is used in places where it might show to users. This field uses the original, user-friendly casing for the data. It is created with IsFacetable=true.</li></ul><p>This changed how we work with these fields a bit:</p><ul><li>When adding new data properties, we have to consider whether this is a case where we’ll need the double fields.</li><li>When ingesting data about new vehicles, we need to create two copies of the data for each of these fields.</li><li>When generating Azure queries, we need to consider which field to reference for searching, faceting, selecting, and sorting.</li></ul><p>That said, now that the pattern is established and works pretty much the same for all fields, it hasn’t been hard to keep up with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=230426259138" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/another-azure-search-workaround-case-sensitivity-230426259138">Another Azure Search Workaround: Case Sensitivity</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Search for Data We Don’t Have]]></title>
            <link>https://medium.com/carmax-engineering-blog/how-to-search-for-data-we-dont-have-6c393d0bc25f?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c393d0bc25f</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[azure-search]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Mon, 29 Jul 2024 17:32:49 GMT</pubDate>
            <atom:updated>2024-07-29T17:37:05.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYewuTsJihIDQ3zAbmQr_Q.jpeg" /></figure><h3><strong>Search Facets</strong></h3><p>The CarMax website and mobile apps allow customers to search for vehicles based on a variety of data points. These include a few different types of data:</p><ul><li>Strings — These are simple fields where a vehicle has a single value (or null). Examples include Model (e.g. “4Runner” or “Civic”) and Car Size (e.g. “Compact” or “Midsize”).</li><li>Collections — These fields are also strings, but a single vehicle could have multiple values. Examples include Features (e.g. “Rear Seat Heaters” or “Parking Sensors”) and Body Type (e.g. “Convertibles” or “Sedans”).</li><li>Numbers — These fields could be integers or doubles. Like strings, each vehicle has a single value (or null). Examples include Price (e.g. $22,998) and MPG Highway (e.g. 31). For numbers, instead of searching for specific values, customers generally want to search with a maximum and/or minimum value.</li></ul><p>Our search is powered by <a href="https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search">Azure Search</a>. Each of the above items is used in Filters to limit the set of vehicles returned to those that the customer is most interested in. They are also used in Facets. Facets allow us to show a count of documents (vehicles) that match each value. For example, of the 100 cars we are showing you right now, 37 are “Compact”, 52 are “Midsize” and 11 are “Large”. The customer knows that if they choose to add a filter down to Compact cars, they will have 37 results to consider. For number fields, these Facets are able to work on intervals of a consistent size. For example, we use an interval of 1000 for price, so you can get counts of cars with a price of $20,000–20,999, a price of $21,000-$21,999, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KUeVqdqtx_a36wSDMD8LeA.png" /></figure><h3><strong>Missing Data</strong></h3><p>This works well for us most of the time. But it turns out that our data isn’t always perfect. Sometimes we have missing data points. So you might see that, of the 100 cars we are showing you right now, 35 are “Compact”, 49 are “Midsize” and 8 are “Large”. This only adds up to 92 cars, with 8 that have no assigned Size value at all. Once you start filtering by size, there’s no way to see those 8 vehicles. It seems a little strange, but for certain filters, if you check off every possible option in the filter list to include them all, the number of results you see will decrease from where you started. Because those vehicles with null values are no longer returned.</p><p>Some fields we know won’t be populated on all cars. For example, Battery Range is applicable to Electric Vehicles and Plug-in Hybrids, but not to Gas and Diesel powered vehicles. Cab Style will only be populated for Pickup Trucks.</p><p>For many data points, every car does have a valid value in the real world, but sometimes we don’t know what that value is. This is obviously not ideal. Our first choice is to fill those gaps, and have every data point populated on every vehicle for which it is relevant. We are always working towards improving our data quality and making sure we give the customer all the information they need to make the right purchase for them.</p><p>Existing data gaps are a problem for several reasons. When a customer is looking at a specific car, we can’t give them all of the information about that car to help inform their decision. They can visit the store or reach out by phone or chat to get questions answered, but this is not very convenient.</p><p>When a customer is using our search page to filter down to the cars they are interested in, some cars that are actually relevant to their search may not show. In our previous example, say a customer filtered their results down to those 35 Compact vehicles. It may be the case that a couple of the 8 vehicles missing values are also Compact, but the user won’t see those.</p><h3><strong>Finding Vehicles With Missing Data</strong></h3><p>Some customers only want to see cars that they know have exactly what they want. That’s the way search on CarMax has worked for years. If you search for a car with All Wheel Drive, you don’t want to see any results that are not All Wheel Drive. Other customers are more concerned with making sure they don’t miss cars, especially if they are looking for something uncommon. For these customers, it’s more important to make sure that every single car with All Wheel Drive is returned for their search, even if it means a couple other cars slip in too.</p><p>We recently decided that we need to enable both scenarios. We want to leave the default behavior as it always has been — we only return cars that we know are a match. But also enable customers to opt-in to seeing cars where we aren’t sure. We are using the term “unspecified” for this data. And it’s specific to individual properties on a vehicle. A car might have most of its data complete, but have an unspecified value for Drivetrain, for example.</p><p>What we want to provide is another option in the filter that tells the user the number of vehicles that have an unspecified value that otherwise match their current search, and a way for them to choose to include those vehicles in search results. Azure Search didn’t make this easy.</p><h4><strong>Idea Number 1 — Just look for nulls</strong></h4><p>The first thing we considered was just leaving the data as it currently exists, with null values in the fields we don’t know. This is the most natural way to store things. Unfortunately it makes it hard to query.</p><p>When searching for a list of values using something like the search.in function, values must be real strings. It isn’t possible to include ‘null’ as one of the values in the list, so you have to add an extra clause to your search. Our searches are already a complicated mess of clauses ANDed and ORed together in various combinations, so this would only add to the mess.</p><p>And Facet counts don’t include null values. To get the count of vehicles with an unspecified value, we’d have to do a separate query for each field. We really didn’t want to do that — performance of the search page is important to us. This performance issue, far more than the complicated query syntax, led us to discard this idea before we even got to implementation.</p><h4><strong>Idea Number 2 — Special Unspecified Values</strong></h4><p>The next idea we tried was putting magic values in place of null. This would have actually been a fine solution if we were only dealing with string properties. We could store the string “unspecified”, and show it in the list of values alongside all the others. It doesn’t work as simply for numbers, but it was possible for us to build something functional.</p><p>For strings, we’d store “unspecified”; for collections of strings, we’d store [“unspecified”]; and for numbers, we’d store -99999. All of our numerical fields happen to only allow non-negative values, so this -99999 could never be confused for a real value.</p><p>In some ways this worked great. It was easy to search for this value alongside other values we were querying for, since it was the same type. It gets included in the Facet counts we get back from search. It provided just what we needed.</p><p>But it also required us to be really careful. Every single query we did on either vehicle data or Facets needed to have its results filtered so that these values weren’t shown to customers, or used in any backend processes. We obviously don’t want them showing up in the list of regular Facet values to choose from, or in the data we list about a car. (An MPG Highway of -99999 somehow wouldn’t look quite right.) And we had to be careful when filtering numbers by only a maximum value. If a customer is looking for cars with Mileage &lt; 50,000, we don’t want to always include unspecified results even though -99999 &lt; 50000.</p><p>And it’s not just the interface for customers — we have a lot of other support processes that use the same search indexes. We don’t want these values fed into our search typeahead suggestions, caught in text search processing, counted in certain monitoring jobs we run, etc.</p><p>We did implement this approach, but ultimately decided it was too risky and the values were going to leak out here and there, leading to unexpected and undesirable behavior.</p><h4><strong>Idea Number 3 — Track Unspecified Data in a New Field</strong></h4><p>As often seems to happen, our third idea turned out to be a winner. We ultimately added a new property to vehicles, a collection of strings, that holds the name of each field that is unspecified. So if a vehicle has null values for MpgHighway and Drivetrain, then the UnspecifiedFieldNames field would be [“MpgHighway”, “Drivetrain”]. This takes care of all of the issues we had with previous ideas.</p><ol><li>It keeps the Unspecified data management in its own new space, separate from all the existing data, so we are confident it won’t leak through where we don’t want it.</li><li>It’s easy to filter for vehicles with Unspecified values for any field, regardless of the data type of that field, by simply looking for the field name in this collection. We can set up the filter the same way, whether the base field was a number, a string, or a collection of strings.</li><li>When getting Facet counts, we now always get counts for this field as well. That tells us how many vehicles have unspecified values for each field that may be in the list without any additional queries or work necessary.</li></ol><p>This is a recent addition, and the UI to support it is still being built. But we are feeling great about having the functionality in the API in a way that will be easy to sustain in the long term. And this solution isn’t specific to vehicles in any way, it can be applied to any domain where users are filtering and faceting on a set of fields that may not always be populated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c393d0bc25f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/how-to-search-for-data-we-dont-have-6c393d0bc25f">How to Search for Data We Don’t Have</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Doing On-Call the Right Way]]></title>
            <link>https://medium.com/carmax-engineering-blog/doing-on-call-the-right-way-0a31e591152c?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a31e591152c</guid>
            <category><![CDATA[burnout]]></category>
            <category><![CDATA[alerts]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[on-call]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Wed, 29 May 2024 13:16:47 GMT</pubDate>
            <atom:updated>2024-05-29T13:16:47.210Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Improving Quality of Life for Engineers and Quality of Product for Customers</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6SCPvcpUXCN2KkfWie1QwA.png" /></figure><p>I’ve worked in software development for many years now. I’ve worked on a lot of different teams in a handful of different companies. Most of these jobs have involved on-call work in some form or another. I’ve had a couple of jobs where the on-call completely burned me out. For a while after that, I would only take jobs with no on-call required, even if it meant taking a step backwards in my career. But in recent years, I’ve found that on-call doesn’t have to be a nightmare.</p><p>My role at CarMax currently involves an on-call rotation. I am on call 24/7 one out of every four weeks. And past Maria might be surprised to hear that I really don’t mind it at all. It turns out that it’s not on-call that was hard, but poorly managed on-call. I want to share some strategies that make this responsibility better for the engineers, while also producing a better product for customers.</p><h4>I’m Only On Call for Things I Work On</h4><p>At CarMax, we used to have a large on-call rotation with all the engineers in the Product Organization. At first, this seemed great — there were so many of us that I was only on call about one week out of every eight months. But I quickly learned that this system wasn’t so great.</p><p>First of all, when I was on call, most of the alerts were for things I didn’t know anything about. Sometimes they were things I’d never even heard of. Often there was really nothing I was able to do to get things working again, other than call other people (who weren’t supposed to be on call) or wait till business hours. If I was able to resolve it myself, it took much longer than it should have, as I had to learn about the systems in the moment. And I got paged a lot, because I was responsible for so many different things that week.</p><p>Another big problem was that none of the underlying issues got fixed. If I get alerted about an intermittent problem in a system someone else owns, I’m not in a position to do anything about it. And there’s no visibility into how much of a problem it is. Only being on call once every eight months, I wouldn’t know if this comes up every week or only once a year. And the team that owned it wasn’t pushed to resolve it, because they too only got paged a couple times a year.</p><p>Now that I’m on call for only things that my own team works on, and on call much more often, things work a lot better. If something breaks, it’s something I am very familiar</p><p>with. This means I’m able to address it myself (or at least within my immediate team), and fairly quickly. If it’s an alert that is likely to go off again, I am able to get to the underlying cause and actually fix it. If I ever get paged a second time for the same thing, I know it wasn’t a fluke and should be cleaned up. This means I’m not only addressing the outage I was paged for, but also preventing future outages. As a result, my future on-calls are quieter. Another result is that the product is working better for our customers, because I’ve reduced the chance of another outage.</p><h4>We Are Empowered to Fix the Problems</h4><p>I’ve worked on teams where the product owners dictate the work that gets done, and what they care about is adding new features. Adding new features is a key part of my job, and it’s something we need to do to keep making our product better for our customers.</p><p>But we also need a solid foundation. At CarMax, the teams I have worked on have always empowered the engineers to do what is right. If there is an underlying bug, it can be prioritized to be fixed. If there’s an alert that is just noisier than it needs to be, it can be adjusted. Time can be made for these foundational issues, alongside new feature development and other product enhancements. The quicker these things get fixed, the less developer time is wasted dealing with the issues when they arise, and the less developer burnout happens, leaving the developer to focus more on those new features going forward.</p><h4>We are Thoughtful About Alerting Levels</h4><p>When incidents occur, our engineers don’t just think about addressing the issue, but also about how well the alerting worked. There are a few questions I ask myself whenever I’m responding to an alert or handling an outage:</p><ul><li>If I got an alert, was it appropriate? Was there action for me to take? If there was nothing to do, maybe I don’t need to be alerted. Reducing the noise of the alerts makes developers’ lives better, and ensures that we pay better attention when alerts do come in.</li><li>If we got an appropriate alert, was it fast enough? Could the problem have been detected earlier, so it could be resolved earlier?</li><li>Was the level of the alert correct? Our team has high-priority alerts that page the on-call immediately, and low-priority alerts that simply send an email for the on-call to see when they are at their desk. If an urgent issue triggers a low-priority alert, the issue could go unnoticed. (In the worst case, it might go off on a Friday evening and not be seen till Monday morning.) If a non-urgent issue triggers a high-priority alert, it’s creating extra noise. Noisy alerts can lead to developer burnout. Too many noisy alerts can also lead a developer to think alerts aren’t important, things that can be ignored for a while, potentially missing an alert that really is urgent.</li><li>Sometimes we find out about a problem with our service through other means. Maybe we stumble on it in the logs while looking for something else; maybe we run into the problem ourselves while using our services; in the worst case, someone else finds the problem first and brings it to our attention. In a perfect world, a customer (internal or external) should never know that our systems are having problems before we do. When there’s an issue we didn’t detect automatically, we always think about what alerts (or automated tests, if it’s a coding bug) we can add to catch the issue next time.</li><li>Did the alerts have the necessary information? Did they make it very clear what was wrong and where? If not, we can improve the alerts with more details.</li></ul><h4>We Make Alerts Easier to Resolve</h4><p>When an issue arises, the best-case scenario is that it resolves itself, and no developers need to be woken up. Things like auto-scale settings that react to increasing load on your resources are an example of this. You also might build your own framework to detect an issue and take the appropriate action — try replaying messages that show up on a DLQ, reboot a misbehaving server, etc.</p><p>The next best-case scenario is that the resolution is automated, but developers need to trigger it. For example, if we need to restart our services, we have Azure DevOps Pipelines that can manage this for us. Our service runs in three regions, so the pipelines will remove one region from Traffic Manager, wait for existing traffic to drain, restart the service, then add it back to Traffic Manager. This ensures that services can be restarted safely on a rolling basis while other instances are still serving requests. By automating this process , we ensure that a sleepy developer doesn’t miss a step, click on the wrong thing, or move too fast. Even a developer who is brand new to the team can use these tools safely. We don’t have an automated way to detect that these need to be run and trigger them. But once a developer decides it’s the right thing to do, it’s a single button click to make it all happen.</p><p>If nothing else, documentation is helpful. It’s great to have a document for each alert that might come through, including information on how to investigate and resolve it (possibly referencing those automated resolution tools I just mentioned). If an alert comes in that doesn’t have a documented resolution, adding that documentation is a good action item to be completed during normal working hours. As the lead engineer on my team, this is also great protection for my time. Just because I know how to handle some of these situations doesn’t mean everyone on the team, down to the most junior engineer, knows how. If another engineer is dealing with an outage scenario they aren’t familiar with, and there is no documentation, they might reach out to me. Having good</p><p>documentation means that my phone stays completely quiet during the weeks I’m not on-call.</p><p>Another thing that helps is having great observability. When something goes weird, we’ve got lots of logs, metrics, and dashboards to help us dig into what is going on, and what else might be related to it. If we don’t find the info that we need, or it was hard to get to and make sense of, this is something else we will do. We might add more logging, track more metrics, or enhance our dashboards to highlight the important information and show correlations that matter.</p><h4>A Common Thread: Continuous Improvement</h4><p>I am a huge believer in continuous improvement. I’m sure I’ll have more posts that talk about how we continuously improve at work. Any time an issue comes up, I like to think about “how could this have gone better,” and then take the necessary steps to make it better for the future. This really works. My on-calls are incredibly quiet, and it’s not unusual for months to go by where I don’t receive any high-priority alerts, and don’t ever have to log on outside my regular working hours. This makes me a lot happier, and a lot more productive at work. It’s also a sign that our product is working well and not causing issues for the customers who are trying to use it. When you set up a good on-call system, and continually iterate to make it better, everybody wins!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a31e591152c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carmax-engineering-blog/doing-on-call-the-right-way-0a31e591152c">Doing On-Call the Right Way</a> was originally published in <a href="https://medium.com/carmax-engineering-blog">CarMax Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Call — Saturday Night]]></title>
            <link>https://medium.com/shedev/on-call-saturday-night-31d204d6a383?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/31d204d6a383</guid>
            <category><![CDATA[production-support]]></category>
            <category><![CDATA[on-call]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Thu, 04 Jan 2018 15:01:04 GMT</pubDate>
            <atom:updated>2018-01-04T15:01:04.070Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*c73U5jWQJX9aOeV6XsZnjg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31d204d6a383" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shedev/on-call-saturday-night-31d204d6a383">On Call — Saturday Night</a> was originally published in <a href="https://medium.com/shedev">SheDev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Call — Thursday Night]]></title>
            <link>https://medium.com/shedev/on-call-thursday-night-6155746e2b21?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/6155746e2b21</guid>
            <category><![CDATA[production-support]]></category>
            <category><![CDATA[on-call]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Thu, 28 Dec 2017 15:01:02 GMT</pubDate>
            <atom:updated>2017-12-28T15:01:02.634Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*BxLS6mmAcj2PH5yd0CGHwQ.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6155746e2b21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shedev/on-call-thursday-night-6155746e2b21">On Call — Thursday Night</a> was originally published in <a href="https://medium.com/shedev">SheDev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Call — Wednesday Night]]></title>
            <link>https://medium.com/shedev/on-call-wednesday-night-ca8d644afbeb?source=rss-86bcc9f00470------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca8d644afbeb</guid>
            <category><![CDATA[on-call]]></category>
            <category><![CDATA[production-support]]></category>
            <dc:creator><![CDATA[Maria Gullickson]]></dc:creator>
            <pubDate>Thu, 21 Dec 2017 15:01:03 GMT</pubDate>
            <atom:updated>2017-12-21T15:01:03.979Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*DIgYbbK7TuIa6nt2oERVrw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca8d644afbeb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shedev/on-call-wednesday-night-ca8d644afbeb">On Call — Wednesday Night</a> was originally published in <a href="https://medium.com/shedev">SheDev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>