<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Luiz Scheidegger on Medium]]></title>
        <description><![CDATA[Stories by Luiz Scheidegger on Medium]]></description>
        <link>https://medium.com/@lfscheidegger?source=rss-91d7dcc1cd0b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*E20CXH7KPXzcabHa0c8ksA.jpeg</url>
            <title>Stories by Luiz Scheidegger on Medium</title>
            <link>https://medium.com/@lfscheidegger?source=rss-91d7dcc1cd0b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:49:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lfscheidegger/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Launch Control at Nextdoor]]></title>
            <link>https://engblog.nextdoor.com/launch-control-at-nextdoor-24f42c36b651?source=rss-91d7dcc1cd0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/24f42c36b651</guid>
            <dc:creator><![CDATA[Luiz Scheidegger]]></dc:creator>
            <pubDate>Tue, 15 Mar 2022 18:02:30 GMT</pubDate>
            <atom:updated>2022-03-15T18:02:30.784Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>How engineers configure and deploy A/B tests and feature flags</strong></p><p>In this article, we share our experience building Launch Control, Nextdoor’s combined feature flagging and experiment configuration tool. One of Nextdoor’s core values is “Experiment and Learn Quickly”, and one of our engineering principles is “<a href="https://engblog.nextdoor.com/engineering-principles-at-nextdoor-e82743b2ef2f">Move Fast — Build Iteratively</a>”. We believe fast iteration on our products and features is a great way to bring better value to our neighbors around the world. Teams at Nextdoor routinely use data from countless experiments to inform product improvements. Moreover, in an environment where it’s impractical to ship native mobile apps more than about once per week, we also make frequent use of feature flags as a way to safely and gradually release new products to our neighbors. Both of these needs — experimentation and feature flagging — require robust internal tools and strong developer education to be used at scale.</p><p>One of the most unique things about Launch Control is how it was built as a strong and ongoing cross-functional collaboration between engineers from all different teams. Its creation came about when a backend product engineer identified opportunities to make the legacy AB and Feature Config tools better. Although they didn’t officially work on internal tools, we strongly believe in ownership and empowerment at Nextdoor, so we ensured they had the space and support to quickly iterate on a prototype. Once this prototype had enough features to get adoption, we made more and more room for that engineer to contribute, and made sure to recognize their impact to Nextdoor engineering.</p><p>After this, Launch Control grew into a shared labor of love across Nextdoor — it has key contributions from many of our best engineers across different teams and stacks, from its core backend components, through a delightful React-powered user interface, all the way to its APIs that integrate experiments and feature flags into our Android and iOS mobile apps. A recurring ritual in Launch Control development is to empower and support engineers who identify ongoing feature improvements to make those improvements themselves. This generates a strong sense of camaraderie across teams and helps spread technical knowledge of how the Launch Control stack works.</p><p><strong>Evolving two internal tools into one</strong></p><p>Before Launch Control, we used two separate tools for experimentation and feature flagging, creatively named Feature Config and AB. These tools suffered from a number of technical limitations. For example, Feature Config allowed engineers to use a rich set of user features, such as their Nextdoor neighborhood, country, or app version, but only produced a binary true/false decision, making it impractical for experiments which need multiple treatment groups. In contrast, the AB tool could output custom treatment groups, but its targeting capabilities were limited to basic percentage-based rollouts.</p><p>Additionally, both tools had sparse and uninviting user interfaces, making it difficult for non-technical users to read, or contribute to, experiments and flags. Launch Control was designed to supersede these tools, with a friendly user interface, and a rich set of targeting capabilities, as well as support for arbitrary treatment groups.</p><p><strong>Launch Control: easy to understand user targeting</strong></p><p>At its core, each individual Launch Control experiment represents a function whose inputs are a set of parameters about a Nextdoor neighbor, such as their id, city, mobile platform, etc., and whose output is a specific <em>treatment group</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVk6RxmoPnUZ28opHLdZGw.png" /><figcaption><em>Each Launch Control experiment encodes a mapping between parameters about a Nextdoor neighbor and a specific treatment group for that experiment.</em></figcaption></figure><p>When deciding how to encode these functions, we tried to strike a balance between simplicity and expressive power. Our legacy Feature Config tool, for example, allows arbitrarily nested boolean and/or clauses which check for things such as country allow-lists and percentage rollouts. Unfortunately, that tool made it difficult to express complex relationships in a readable way, leading us to occasionally mistarget rollouts, and discouraging non-technical employees from participating in experimentation.</p><p>For Launch Control, we found that a non-nesting linear sequence of <em>Condition Blocks</em>, each of which is a combination of targeting features, is a great balance between readability and targeting power. When evaluating a particular Launch Control, the algorithm iterates over an experiment’s Condition Blocks, stopping at the first block which successfully captures a user. That block then assigns a specific treatment group to the user, and evaluation ends. In the case that no Condition Block captures a user, Launch Control automatically returns the reserved “untreated” treatment group for that user, indicating that they are not part of the A/B test, or should not get the feature flag in question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bTEuymNRH9K0LxUYfvoB3A.png" /><figcaption><em>Condition Blocks are and-combinations of individual targeting features, evaluated from top to bottom. Launch Control assigns a treatment group based on the first Condition Block that “captures” a user.</em></figcaption></figure><p><strong>Condition Blocks: the heart of Launch Control</strong></p><p>Launch Control supports many different types of Condition Blocks, which allow for targeting based on a variety of user features:</p><ul><li><strong>Individual allow-lists:</strong> These blocks allow us to target individual Nextdoor users. This is particularly useful in the early stages of developing a new feature, as we can target internal team members for dogfooding and testing long before the feature is ready for prime-time.</li><li><strong>Percentage-based rollouts: </strong>Percentage rollouts allow us to encode things like standard A/B tests, where we introduce a new feature to a small subset of our users, comparing those users’ key metrics with a similar-sized control group. In addition, percentage rollouts also give us the ability to gradually release improvements to our users, a few percentage points at a time.</li><li><strong>Geo-targeting: </strong>Some Condition Blocks allow us to target users based on their neighborhood’s city, state, and country. This allows us to iterate on features we decide to launch at different times for different markets (e.g., if we want to iterate on copy for international markets separately from the US).</li><li><strong>Delegation:</strong> It’s possible to configure a Condition Block that delegates its decision to another Launch Control experiment. This allows teams to build a hierarchy of flags, providing an easy-to-use mechanism for feature switches that are progressively broader.</li></ul><p>There is some interesting nuance in how a stochastic, percentage-based Condition Block needs to work. For most features, we expect the same user to <em>always be either in, or out</em>, of an experiment. It would be unreasonable to make a random selection every time a user is evaluated in a condition block, as users would be surprised when they occasionally find themselves jumping between having and not having a new product experience! Launch Control ensures this random-but-deterministic behavior by using a hash of users’ ids to resolve membership in probabilistic blocks. Briefly, that computation looks like this:</p><pre>hashed_value = <strong>int(</strong>sha1<strong>(</strong>user_id<strong>)</strong>.hexdigest<strong>())<br></strong>hashed_probability = <strong>(</strong>hashed_value <strong>%</strong> 10000<strong>)</strong> <strong>/</strong> 10000<br><strong>return</strong> hashed_probability <strong>&lt;</strong> rollout_percentage</pre><p>The method above ensures users always get consistent treatment groups from each Condition Block, without the need to store an explicit mapping between users and treatment groups on a database table or in memory at any time. For completeness, Launch Control does also offer a “Dice” block, which is a truly random determination, although its use cases are relatively rare.</p><p>However, even with a hash-based approach, there is still an additional trap we need to avoid: although the same user should get the same treatment group for <em>a particular</em> Condition Block, we expect users to get different treatment groups for <em>different</em> Condition Blocks. At any time, there may be dozens of different features, all configured as, e.g., 50/50 splits. If we rely on user ids alone for hashing, we would run into users that end up either in <em>all</em> features, or <em>no features at all</em>! Launch Control avoids this by concatenating a per-Condition Block salt to user ids before hashing them. This guarantees each user always gets the same result from a <em>particular</em> Condition Block, while getting potentially different results from <em>different</em> Condition Blocks. When we incorporate a salt into the evaluation, the previous algorithm looks like this:</p><pre>salted_string = f&#39;{condition_block_salt}{user_id}&#39;<br>hashed_value = <strong>int(</strong>sha1<strong>(</strong>salted_string<strong>)</strong>.hexdigest<strong>())<br></strong>hashed_probability = <strong>(</strong>hashed_value <strong>%</strong> 10000<strong>)</strong> <strong>/</strong> 10000<br><strong>return</strong> hashed_probability <strong>&lt;</strong> rollout_percentage</pre><p>The targeting engine described above is only one component of Launch Control. We also provide performant, easy-to-use APIs for engineers to query experiments across our tech. stack, in our application backend as well as our React frontend and native Android and iOS clients.</p><p><strong>Backend vs. Frontend Launch Control APIs</strong></p><p>Nextdoor engineers query Launch Control experiments in many places in our codebase: in our backend as we serve web requests to our clients, as well as in our desktop and mobile-web front ends, and finally in our native Android and iOS applications. In all cases, we provide APIs with two specific goals in mind: ease of use and fast, reliable performance. It is imperative that evaluating an experiment take no more than a few hundred microseconds, as complex user features like our main feed requires dozens of individual flags and experiments to render.</p><p>Launch Control experiments are stored internally as <a href="https://engblog.nextdoor.com/sitevars%C2%B9-at-nextdoor-how-engineers-quickly-push-configuration-changes-in-production-d40fd4f6ba5d">Nextdoor Sitevars</a>. Because of this, our backend containers automatically have access to cached, in-host payloads for all relevant experiment definitions. This makes it relatively easy to have the backend Launch Control APIs be performant. Evaluation typically involves a small amount of CPU operations, and thanks to the Sitevars cache, requires no RPCs or network calls. Even despite this, however, Launch Control also maintains a per-web request cache of evaluated experiments, ensuring that each experiment is evaluated at most once, per user and per request.</p><p>On the other hand, the frontend and mobile APIs present more of a challenge. Since this code runs far from our servers, it’s impractical to cache and constantly synchronize all experiment definitions. It would also be prohibitively expensive to make each individual experiment API call involve a network operation. Because of this, we provide two flavors of Launch Control APIs on our clients.</p><p>The most commonly used flavor is a <em>local</em> API, which relies on the client having a known, pre-fetched list of experiments available in memory at all times. Clients run a single network request which fetches all experiment evaluations in this list at key application lifecycle moments (such as user login, session refresh, etc.), and those results are then available for code to query against in a synchronous fashion. We also provide an <em>asynchronous</em> API, which does allow engineers to make individual network requests for each experiment. This API only exposes non-blocking components, however (such as <a href="https://reactivex.io/documentation/observable.html">Observables</a> on Android and <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promises</a> on web), to make its asynchronous nature clear and self-enforcing. While the local API is used for the most common experiments users are exposed to, the asynchronous flavor helps prevent rarer use experiments from excessively bloating the prefetch list. Finally, while it is out of scope for this article, Launch Control also provides specific APIs for engineers to run experiments in special cases, such as login and sign up screens, where we do not yet have a particular user available.</p><p><strong>Other Features</strong></p><p>The most important components of Launch Control are its targeting algorithms and query APIs, as described above. However, it also has a number of convenience and ease-of-use features, designed to make experimentation and feature flagging at Nextdoor a delightful experience. We strongly believe that engineers deserve tools that are as good as the products we ship to our neighbors around the world. Some of the other important Launch Control features include:</p><ul><li><strong>Contextual editing UI:</strong> The UI for each Condition Block type is unique and has deep knowledge of that block’s context. For example, allow-lists for individual users expose a generic search and typeahead. This allows us to add users by their name, email, or other traits, eliminating the need for people to memorize ids, and empowering team XFN partners such as Product Managers and Designers to add themselves to experiments directly.</li><li><strong>Versioning and fast reverts:</strong> Launch Control experiments are stored internally as Nextdoor Sitevars, which have built-in support for versioning. Whenever a Launch Control is updated, we store a new version of it, with useful metadata such as edit time and author. This allows us to easily navigate through the full history of an experiment, eliminating the need to maintain redundant documentation on when a particular rollout changed. In addition, storing every version of an experiment also allows us to quickly revert features to known-good states in case anything goes unexpectedly sideways with a release.</li><li><strong>Built-in observability and update subscriptions:</strong> Every Launch Control automatically publishes useful real time statistics whenever it’s evaluated, for all users. This allows teams to verify, in real time, that their experiments are going out to the expected number of users, and in the right proportions. In addition, employees can also subscribe to Launch Control experiments, so they get automatically notified when an experiment is updated.</li></ul><p><strong>Conclusion</strong></p><p>Launch Control experiments drive many of our recent Nextdoor product improvements, such as changes to <a href="https://engblog.nextdoor.com/nextdoor-notifications-how-we-use-ml-to-keep-neighbors-informed-57d8f707aab0">Notifications</a>, <a href="https://engblog.nextdoor.com/modernizing-our-search-stack-6a56ab87db4e">Search</a>, and our <a href="https://engblog.nextdoor.com/metablocks-our-journey-to-an-agglomerated-navigation-logic-beb6a2afcaa3">Business Experience</a>. By striking a balance between expressive power, simplicity, and usability, we were able to collaboratively build a tool that experienced widespread internal adoption.</p><p>Have you worked on, or used, feature targeting and A/B testing tools in your career? Let us know in the comments below what you’ve learned works and what doesn’t, and <strong>if you’re excited about collaborating with other great engineers on impactful work like this, check out our </strong><a href="https://boards.greenhouse.io/embed/job_board?for=nextdoor&amp;b=https%3A%2F%2Fabout.nextdoor.com%2Fcareers%2F#51115"><strong>careers page</strong></a><strong>!</strong> We have open opportunities across different teams and functions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24f42c36b651" width="1" height="1" alt=""><hr><p><a href="https://engblog.nextdoor.com/launch-control-at-nextdoor-24f42c36b651">Launch Control at Nextdoor</a> was originally published in <a href="https://engblog.nextdoor.com/">Nextdoor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sitevars¹ at Nextdoor — How engineers quickly push configuration changes in production]]></title>
            <link>https://engblog.nextdoor.com/sitevars%C2%B9-at-nextdoor-how-engineers-quickly-push-configuration-changes-in-production-d40fd4f6ba5d?source=rss-91d7dcc1cd0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d40fd4f6ba5d</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[configuration-management]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Luiz Scheidegger]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 21:42:08 GMT</pubDate>
            <atom:updated>2020-04-28T21:42:08.422Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Sitevars¹ at Nextdoor — How engineers quickly push configuration changes in production</strong></h3><p>Nextdoor engineers are always looking for ways to move faster. Making changes quickly and safely allows us to deliver more value to our members in neighborhoods around the globe, as well as to minimize the impact of bugs and disruptions we may come across. In this post, we’re going to share our experience building Sitevars, an internal system at Nextdoor which empowers engineers to push configuration payloads to all our servers across multiple regions in seconds.</p><p>Before Sitevars, engineers at Nextdoor had to go through a full write-commit-deploy cycle to update configuration values. This meant that the iteration cycle rarely took less than an hour — too long for fast product development.</p><p><strong>The problem</strong></p><p>As we iterate on our product, we come across many areas where features can benefit from having an easily-editable configuration payload with domain-specific parameters. Some examples of this include:</p><ul><li>ML parameters — we use <a href="https://engblog.nextdoor.com/the-kindness-reminder-868252995140">machine learning models</a> extensively at Nextdoor to help moderate content and improve our members’ experience. As we iterate on these models, it’s very helpful to be able to quickly experiment with different threshold values for classifiers, as well as weights for model features, etc.</li><li>Global kill switches — we use circuit-breaker style switches to protect new features in production. If anything goes wrong with a new feature, we can quickly turn it off before it causes problems for our members.</li><li>Logging configuration — we use Sitevars to quickly update dynamic logging levels for different parts of our codebase without performing a new deploy to production, or even making code changes.</li></ul><p>These examples all share a common need for a system that can propagate configuration changes — typically small JSON payloads — to our server fleet <strong>quickly and safely.</strong> When an engineer updates a configuration, that change should propagate to our servers across the world within seconds. We rely heavily on monitoring and automated alerts to notify engineers when a Sitevar change breaks something, so it can be quickly reverted to a healthy value.</p><p>In addition, the Sitevars service needs to be robust — a transient failure shouldn’t prevent a request from being served successfully. Finally, Sitevars payloads must be cheap to read. Some of our requests access dozens of Sitevars, so reads should take no more than a few microseconds.</p><p><strong>Sitevars: fast-propagating, versioned JSON</strong></p><p>Sitevars consist of two main parts: a Go service which manages the backing store for Sitevars payloads, and a set of client APIs in our Django application which handle service failure and local caching for improved performance. In addition, we built an internal tool that gives engineers a simple UI to create, search, and edit any Sitevars payload.</p><p><strong>The Sitevars Service</strong></p><p>The service component of Sitevars is a Go application which provides an API for callers to create, update, and fetch Sitevars payloads. Each payload contains a small piece of JSON (we currently limit this to 16KB in size), as well as common metadata fields — last author, update time, version, etc. Sitevars objects are stored in a globally-replicated <a href="https://aws.amazon.com/dynamodb/">DynamoDB</a> table. The schema for this table is very simple — its hash key consists of the unique name of a configuration, and its range key is a version number:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDwV-IyyFqs7eOeFlHOg7g.png" /><figcaption><em>The DynamoDB schema for Sitevars consists of a namespaced hash key, a version range key, and a JSON payload for the Sitevars value.</em></figcaption></figure><p>When a Sitevars object is updated, we insert a new row into the table, with the latest value of the payload and a new version number. <strong>This ensures that updates are non-destructive, and that reverting to a known-good configuration in case of problems is trivial.</strong></p><p>Single-row fetches from DynamoDB typically take a few milliseconds to complete. While that cost isn’t too high for a single Sitevar, many of our endpoints fetch dozens of configurations, so minimizing this latency is critical. To accomplish this, the Sitevars service keeps an in-memory cache with the latest version of each Sitevar. As the working set is relatively small (each Sitevar is limited to 16KB, and we have a few hundred configs to date), the service can easily hold all Sitevars in memory. Because of this cache, the majority of fetches never make a roundtrip to DynamoDB. Another advantage of a small working set is that it allows us to trivially refresh the entire cache at a set interval. At the moment, this is done every 60 seconds.</p><p>Finally, we use a <a href="https://redis.io/topics/pubsub">Redis pub-sub channel</a> to notify the service when a Sitevar is updated, allowing it to more quickly invalidate a single entry in the cache. This serves a key iteration requirement — <strong>it allows engineers to observe their change in the product almost immediately</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*62XmodvE5A5vgxfiBnZsyw.png" /><figcaption><em>The Sitevars service manages an in-memory cache with a short TTL, such that almost no Sitevar fetches require a roundtrip to DynamoDB. The service also publishes Sitevars updates to a Redis pub-sub channel to quickly propagate value changes.</em></figcaption></figure><p>Fast access to a Sitevars payload in the service is only half of the equation to ensure Sitevars fetches are efficient. Communication between our Django containers and the Sitevars service must also be as fast as possible. We address this in two ways: we deploy the Sitevars container as a sidecar to our Django application, and we use <a href="https://grpc.io/">gRPC</a> as a transport mechanism. Deploying the container as a sidecar ensures that calls between Django and Sitevars never leave a single host, and using gRPC (instead of, e.g., JSON over HTTP) reduces the p50 latency for requests from about 3–5ms to about 800µs. Since we built the gRPC server using <a href="https://github.com/grpc-ecosystem/grpc-gateway">grpc-gateway</a>, that change was trivial to implement. We were quite surprised to find such a big performance improvement!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HI4BMAHuNAkVuyGqTpsE2Q.png" /><figcaption><em>Deploying the Sitevars container as a sidecar (in the same host) along with the Django container ensures that the RPC between the two never leaves a single host.</em></figcaption></figure><p><strong>Client-side APIs</strong></p><p>The second main component of Sitevars is the set of APIs for developers in our Django codebase. These APIs provide a convenient, type-safe way to access Sitevars in our web application. One of their key features is that they require developers to provide, in code, a default value for their Sitevars. This serves two purposes: it allows engineers to write and commit code ahead of creating a specific Sitevar, and it provides a last-resort fallback in case the Sitevars service goes down for any reason.</p><p>However, using the last-resort code fallback for a Sitevar can be risky, especially as it becomes stale for configurations that have been heavily edited over time. Even if the API returns the default successfully, there’s a large chance the calling code is no longer factored to handle the default value without crashing. To mitigate this scenario, we maintain hourly S3 snapshots of all Sitevars, and bake them into our Django container. <strong>This allows us to ensure that, in the event the Sitevars service goes down, our web application can still use Sitevars that are no more than about one hour old.</strong></p><p>When discussing the Sitevars service above, we talked about a caching and transport strategy that brought down the cost of fetching a configuration to just under a millisecond. However, we have one more trick up our sleeve to make this number even smaller: we maintain a request-scoped cache of any fetched Sitevars in our web application. This means that any Sitevar payload is never fetched into Django more than once per request. Any subsequent fetch of the same configuration is only a Python dictionary access away, at the cost of a few microseconds. This is especially useful for configurations that are fetched frequently, such as ones used to drive core pieces of our web infrastructure. When all of these strategies are put together, latency for fetching Sitevars falls into a bimodal distribution, where about half of all configuration fetches takes less than 100µs to complete (when they hit the per-request cache), while the other half takes between 500µs and 800µs (when they require an RPC to the Sitevars service).</p><p><strong>Lessons Learned</strong></p><p>As we developed and shipped Sitevars, we capitalized on the value of a few important lessons. First, simplicity is a key factor in the success of an internal tool. Even though Sitevars is a service aimed at engineers with a good understanding of our tech stack, its focus is still on a simple, predictable API and an easy to use editing tool. This encourages engineers across Nextdoor to use Sitevars in many more places than we had originally envisioned. Our use of DynamoDB global replication also automatically ensures that the values of Sitevars are unified across all AWS regions in which we operate. Finally, we were really surprised by the latency difference between HTTP+JSON and gRPC. When we made that change, we expected to see performance benefits, but were delighted by how significant the gains were.</p><p>Going forward, we plan to expand Sitevars functionality beyond our Django application, to support it across our other services as well.</p><p><strong>Conclusion</strong></p><p>Sitevars are deployed at scale at Nextdoor, where they help power and customize many different features. With the caching and transport architecture we shared here, we brought down the cost of fetching a single configuration from 5–7ms to less than 100µs. Sitevars serve close to 100k QPS during peak times, without impacting site performance or stability. If working on practical, performant infrastructure like this gets you excited, <a href="https://about.nextdoor.com/careers/">we’re hiring — come join us</a>!</p><p>[1] The name “Sitevars”, as well as the inspiration for the benefits of this system, come from previous work in the industry. If you’re curious to learn more, check out this <a href="http://sigops.org/s/conferences/sosp/2015/current/2015-Monterey/printable/008-tang.pdf">great paper</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d40fd4f6ba5d" width="1" height="1" alt=""><hr><p><a href="https://engblog.nextdoor.com/sitevars%C2%B9-at-nextdoor-how-engineers-quickly-push-configuration-changes-in-production-d40fd4f6ba5d">Sitevars¹ at Nextdoor — How engineers quickly push configuration changes in production</a> was originally published in <a href="https://engblog.nextdoor.com/">Nextdoor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Method-local state in Typescript]]></title>
            <link>https://medium.com/@lfscheidegger/method-local-state-in-typescript-ebb41bae02cd?source=rss-91d7dcc1cd0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebb41bae02cd</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[webgl]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Luiz Scheidegger]]></dc:creator>
            <pubDate>Tue, 09 Oct 2018 21:12:37 GMT</pubDate>
            <atom:updated>2018-10-09T21:12:37.360Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been playing around with WebGL on a side project lately, and came across <a href="http://glmatrix.net/">glMatrix</a>, a library for all vector and matrix operations typical of graphics programming. One of the challenges of graphics programming is that the hot path (the handful of functions that gets called each frame) needs to be extremely fast. Ideally, this means few, if any, memory allocations, to minimize garbage collection churn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hVgO7RDO8iV5Eup0kNcGRg.jpeg" /></figure><p>On the other hand, typical graphics code is riddled with small, short-lived objects, such as x/y pixel coordinates and 3d points in space. glMatrix avoids constantly allocating small objects by adopting a convention in which the result of a function is stored in a special out parameter, instead of being new’d and returned. So instead of:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc97d6e5688eaac5758d35a0b4f8c5bc/href">https://medium.com/media/cc97d6e5688eaac5758d35a0b4f8c5bc/href</a></iframe><p>glMatrix does the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11ae50e05289b1b1dc7513166ca97a07/href">https://medium.com/media/11ae50e05289b1b1dc7513166ca97a07/href</a></iframe><p>This transfers the responsibility of managing out&#39;s scope to the caller, which can maintain it somewhere long-lived, thus avoiding an allocation every time add is called.</p><p>However, this presents a new problem: <em>where</em> is the best place to put these long lived objects? One approach is to store them as private class variables:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2c11cc4f53d78b9aac53925c2331c92/href">https://medium.com/media/f2c11cc4f53d78b9aac53925c2331c92/href</a></iframe><p>This solves the initial allocation problem — tempForAdding is allocated only once, and exists for the lifetime of a GLRenderer instance. However, as more and more temporary allocations come into play, the code for GLRenderer becomes messy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/651c2b952295a2e900aa8e63de783f98/href">https://medium.com/media/651c2b952295a2e900aa8e63de783f98/href</a></iframe><p>In addition, this solution has encapsulation problems. Since these fields must be mutable, their scope should be as small as possible. Otherwise this code becomes error-prone, as multiple methods can inadvertently mutate each other’s temporary objects. Thus I need a way to define objects that are allocated only once per instance, and are also private to the methods in which they’re used.</p><p>It turns out the solution to this problem is quite simple, and exploits Typescript’s easy-to-define <a href="https://en.wikipedia.org/wiki/Higher-order_function">higher-order functions</a>, as well as exploiting <a href="https://en.wikipedia.org/wiki/Closure_(computer_programming)">closures</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d44f23ee9c0fafc613ca7c2029f8c089/href">https://medium.com/media/d44f23ee9c0fafc613ca7c2029f8c089/href</a></iframe><p>I like this pattern for a few reasons:</p><ul><li>It keeps tempForAdding&#39;s declaration, and any other helper objects, <em>close to the method that uses them</em>. This makes the helpers’ intent clear, and prevents complicating the top of the class for other readers of your code.</li><li>The first line, renderGL = (() =&gt; {, is so different from a typical method definition that it is obvious something different is going on. Readers can quickly scan the code and see which methods have method-local state and which methods do not.</li><li>The ‘internal’ renderGL function can be written exactly as it would have been without this technique. Due to <a href="https://hacks.mozilla.org/2015/06/es6-in-depth-arrow-functions/">fat-arrow binding</a>, the value of this in calls to renderGL won’t be unexpected (there’s a subtlety here about how this is correctly resolved to the class instance, but that would be the subject of an entirely separate post).</li></ul><p>Once I realized the convenience of this pattern, I found myself cleaning a lot of class state that really should only be visible to a single method. A second use case for this came up when dealing with mouse events. When users drag the cursor across the screen, I need to process the delta-x and delta-y intervals between successive mouse events in a drag. To accomplish this, I had code that looked like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bb16dac82ddfc6414fe61d211e7c600/href">https://medium.com/media/4bb16dac82ddfc6414fe61d211e7c600/href</a></iframe><p>As before, having previousEvent as a class property is not ideal. It pollutes the class namespace and confuses readers with an object that really only belongs to a single method. Using method-local state, the code becomes much more readable:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69717ac8cd4cc2ea977a67f680731659/href">https://medium.com/media/69717ac8cd4cc2ea977a67f680731659/href</a></iframe><p>Finally, I also wanted to verify that these state objects are, in fact, per-instance, rather than static to the entire application. To do that, I wrote a small test class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccb3f1cec41fbd184b02e577e91f985a/href">https://medium.com/media/ccb3f1cec41fbd184b02e577e91f985a/href</a></iframe><p>As expected, this prints 0 twice, demonstrating that a fresh testValue is allocated for each instance of Test.</p><p>Of course, not all class state can be encapsulated this way. There are certainly fields that need to be visible to multiple methods in a class (and some fields need to be public, which this approach doesn’t support). Still, method-local variables are a convenient way to hide unnecessary complexity when all that’s needed are long-lived helper objects for individual methods.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebb41bae02cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A semi-technical description of Meltdown]]></title>
            <link>https://medium.com/@lfscheidegger/a-semi-technical-description-of-meltdown-e96c74394e55?source=rss-91d7dcc1cd0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e96c74394e55</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Luiz Scheidegger]]></dc:creator>
            <pubDate>Thu, 04 Jan 2018 20:40:52 GMT</pubDate>
            <atom:updated>2018-01-04T20:40:52.887Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the holiday break, <a href="http://pythonsweetness.tumblr.com/post/169166980422/the-mysterious-case-of-the-linux-page-table/amp">murmurs</a> surfaced about a serious hardware vulnerability affecting (what was then suspected to be) pretty much all modern Intel CPUs. Today, the media released <a href="https://www.theverge.com/2018/1/3/16846840/intel-arm-processor-flaw-chipocalypse-windows-macos-linux">more details</a>, and Intel issued an <a href="https://newsroom.intel.com/news/intel-responds-to-security-research-findings/">official statement</a>. The rumored vulnerability turned out to be two —<a href="https://meltdownattack.com/"> Meltdown and Spectre</a>. Much talk centered on how quickly, by Linux standards, the community pushed through <a href="https://lwn.net/Articles/738975/">a new set of security patches</a> to the kernel. Now it’s clear why — these patches (called the KAISER patch series) focus on a crucial part of what makes Meltdown possible (we’ll get to that in a bit).</p><p>I want to describe Meltdown in enough detail that any fairly technical person should be able to fully grok the attack, even if (like me) any code running outside of a modern browser gives you the heebie jeebies. I’m basing this explanation directly on the <a href="https://meltdownattack.com/meltdown.pdf">published paper</a>, and I’m focusing on Meltdown because, honestly, it’s almost bedtime and I haven’t had time to read the <a href="https://spectreattack.com/spectre.pdf">paper for Spectre</a> yet.</p><p>For starters, let’s discuss what can be done with Meltdown. It allows malicious code to read the entire contents of a computer’s memory. If that doesn’t scare your pants off, it should. For instance, if you’re running Chrome right now (like I am), code executing the attack can read your passwords in Chrome’s password manager; <strong>even if Chrome encrypts them when saving them to disk</strong>. If the machine is a web server, the attack can reveal private keys used to secure data transmission over https, private API keys, and all sorts of sensitive information. No matter how careful developers are about encrypting sensitive data, that data will be out in the open in memory at some point (otherwise it’s pretty much useless) and that makes it vulnerable to Meltdown. <strong>This is a big deal.</strong></p><p>Normally, different processes on the same machine can’t read each other’s memory. This is done by design; it is intended to prevent the kinds of issues described above. This isolation comes primarily from the idea of <a href="https://en.wikipedia.org/wiki/Virtual_memory"><em>virtual memory</em></a>. This mechanism ensures that each process in the system has its own dedicated address space. When two different processes try to access the same memory address, e.g., 0xdeadbeef, they actually end up reading two different physical memory locations, preventing them from seeing each other’s data. In fact, what you get when a process tries to read memory outside of its allotted addresses is the infamous Segmentation Fault. The memory management unit in a CPU maintains an important data structure (called a <a href="https://en.wikipedia.org/wiki/Page_table"><em>page table</em></a>, or <em>translation table</em>), which maps the running process’ virtual addresses into physical addresses in memory. Whenever the running process is switched by the operating system, this table needs to be replaced — <strong>this is an expensive operation</strong>, and we’ll see why that’s a big problem below.</p><p>In theory, the OS kernel could have its own page table too. This would ensure that user processes can’t ever see kernel data. However, since user processes make system calls to the kernel all the time, having to replace a process’ page table by the kernel’s page table every time a system call is made would be too slow. Instead, the Linux OS (I’m focusing on Linux because, although Meltdown can be exploited on Windows too, the mechanism for memory mapping on Windows kernels is different) <strong>keeps the kernel memory</strong> mapped in the virtual memory space for every running process. In 32-bit systems, for instance, this memory is at the last 1GB of virtual space. Naïvely, this would mean that a user process can try to read location 0xc0000000 (that’s the first byte of the last 1GB of memory) to start getting kernel data. How, then, does the OS prevent this from happening? It turns out that modern CPUs can label certain memory addresses as privileged. If any user process tries to access those addresses, the CPU triggers a trap, and the kernel instantly terminates the process. The kernel can still access that memory because it executes in a special mode, controlled by the CPU, which bypasses those checks. Meltdown allows user processes to extract the contents of those privileged pieces of memory, <strong>even though they can’t read them directly.</strong></p><p>Modern CPUs have many tricks up their sleeves to improve performance, including things like <a href="https://en.wikipedia.org/wiki/Instruction_pipelining">instruction pipelines</a>, multiple levels of memory <a href="https://en.wikipedia.org/wiki/Cache_(computing)">caches</a>, and the <a href="https://en.wikipedia.org/wiki/Out-of-order_execution">ability to execute instructions out of order</a> or ahead of time. It turns out that the latter can be used by malicious code to <strong>leak information about protected areas in memory.</strong> This is the Meltdown attack in a nutshell.</p><p>Suppose we run the pseudo-code below:</p><pre>const array = new ByteArray(256);<br>const byteICantRead = readFromMemoryAt(0xc0000000);<br>print array[byteICantRead];</pre><p>This code does 3 things: it (1) allocates an array of 256 bytes, (2) tries to read a single byte in a forbidden location, and (3) tries to access the index within the array that is also the <strong>value</strong> in the forbidden location.</p><p>Since this code is in a user process, the second line triggers a CPU trap (as it’s trying to read from protected memory), and the process is killed. However, out of order execution on the CPU means that the instructions for the third line begin executing, optimistically, before the CPU trap can kill the process. At this point, the CPU has already placed byteICantRead in a temporary register, and begins to load the requested position in array. Now, this would be harmless (as the process is about to die anyway), except for a crucial side-effect: loading that specific array value also <strong>warms up the CPU cache</strong> in a very predictable way. We can then reveal the forbidden information by timing how long it takes to read each entry in array. Since array[byteICantRead] is now in the cache, the position that reads fastest compared to all the others corresponds to the value we were not supposed to see!</p><pre>for (int i in 0..256) {<br>  const now = time();<br>  print array[i];<br>  const timeDelta = time() - now;<br>  if (timeDelta &lt;= CACHE_HIT_TIME) {<br>    print &quot;And the secret behind the kernel door is... &quot; + i + &quot;!&quot;;<br>  }<br>}</pre><p>This pseudo-code is obviously an oversimplification — in reality the array must be padded to account for the cache bringing in much more than just the requested memory address in a single load, and the timing mechanism needs to be more sophisticated than checking against a constant. But the key insight for the attack is this simple.</p><p>There’s only one thing left to resolve: how can we run the timing code above, if our process dies as soon as we try to read the forbidden byte? Who cares that the CPU left some fingerprints in the cache if we’re not around to check that anyway? It turns out that there are two ways for the user process to keep running after the CPU trap (the <a href="https://meltdownattack.com/meltdown.pdf">Meltdown paper</a> goes into some more detail about how each of them works): (1) the process can either register itself as a handler for the trap interrupt or (2) it can use a special feature of Intel processors called <a href="https://software.intel.com/en-us/node/524022">TSX</a>. TSX essentially wraps a set of instructions in a transaction such that if any instruction fails, they are all rolled back, allowing the rest of the code to execute. As you might have suspected, TSX crucially <strong>does not</strong> roll back any side-effects the failed instructions might have caused to the cache. Sad times.</p><p>The example above only reveals a single forbidden byte (located at the start of the kernel space, in 0xc0000000), but it’s easy to come up with a variation that iterates through any range in memory space and extracts the data. Once a user process can read into kernel memory, <strong>it can then read into all other processes’ memory too!</strong> You see, the kernel needs to keep all running processes’ page tables in its own memory, so it can swap them in when the time is right. So once kernel memory is compromised, it really is game over.</p><p>As a parting thought, let’s consider why the KAISER patches can mitigate this vulnerability; and why they might have considerable performance implications once adopted. Recall that the reason why a user process was able to peek into the Kernel memory is because the Kernel memory is mapped to the user process’ memory space (the top 1GB business above). The KAISER patches, simply put, invalidate that. They were originally designed to address (pun intended) different, although related, vulnerabilities. In effect, the patches bite the bullet and force a new page table to be loaded every time a system call is made, with all the performance implications I hinted at above. With the user process no longer able to directly see Kernel memory, the Meltdown attack no longer has a viable target.</p><p>As to the Spectre attack, it seems to also involve information inadvertently leaked by out-of-order execution on the CPU, but I don’t actually know how it works in more detail, so I’ll stop here for now. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e96c74394e55" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>