<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Juzheng Li on Medium]]></title>
        <description><![CDATA[Stories by Juzheng Li on Medium]]></description>
        <link>https://medium.com/@juzheng.li?source=rss-6f7fa21af202------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ETT6NbPVkTvsXZOT</url>
            <title>Stories by Juzheng Li on Medium</title>
            <link>https://medium.com/@juzheng.li?source=rss-6f7fa21af202------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:08:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@juzheng.li/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Phoenix — Tinder’s Testing Platform — Part II]]></title>
            <link>https://medium.com/tinder/phoenix-tinders-testing-platform-part-ii-e99fbc72af55?source=rss-6f7fa21af202------2</link>
            <guid isPermaLink="false">https://medium.com/p/e99fbc72af55</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Juzheng Li]]></dc:creator>
            <pubDate>Wed, 04 Dec 2019 20:37:27 GMT</pubDate>
            <atom:updated>2019-12-04T20:37:27.134Z</atom:updated>
            <content:encoded><![CDATA[<h3>Phoenix — Tinder’s Testing Platform — Part II</h3><p><strong>Written by:</strong></p><p><strong>Connor Wybranowski</strong>, Software Engineer <strong>|</strong> <strong>Siggi Jonsson</strong>, Staff Software Engineer <strong>|</strong> <strong>Keith McKnight</strong>, Staff Software Engineer <strong>|</strong> <strong>Juzheng Li</strong>, Engineering Manager</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i_FaB608lJK4lpME5JDZcQ.png" /></figure><p>In this section, we’ll discuss how we address our specific needs around remote configuration and decoupling experience delivery from app releases. We suggest reading our <a href="https://medium.com/tinder-engineering/phoenix-tinders-testing-platform-part-i-8e51c19ca977">first article</a> to gain a holistic overview of our experimentation platform before continuing here. Once you’re ready, let’s dive in!</p><p>A lot of work can go into creating a new feature for Tinder. Product Managers and Designers need to figure out what the feature should do and how it should look and feel. Engineers need to write the code that makes it work. Quality Assurance needs to test things to make sure that everything is working properly and that the feature doesn’t create trouble in other parts of the app. Binaries need to be compiled and sent to app stores for approval.</p><p>Even after all of that, there can still be surprises. Maybe we noticed a bug that we didn’t catch during QA while rolling out the new app version. Maybe a sudden flash of inspiration required tweaking some details to make the experience that much better. Maybe everything is good to go, but we want to give the feature a splashy launch at a specific time. Any change can require going through the entire process again, and that can really throw things off.</p><p>The ability to decouple experience delivery from app releases on the mobile clients is a critical part of our workflow at Tinder. After all, there are a number of reasons why feature releases and app releases don’t always align on a consistent schedule.</p><h3>Remote Configuration</h3><p>Remote configuration is a straightforward concept with powerful implications. Rather than baking parameters and feature flags directly into the binary, we can allow the app to request the configuration from our servers. It enables the configuration of the Tinder app from outside the app itself, allowing us to change how the app behaves outside of the release cycle. We can even automate it by scheduling configuration changes.</p><p>Although this approach can be used on any type of client independent of the release cadence, remote configuration really shines on mobile apps, where we cannot simply revert and redeploy if something goes wrong.</p><h3>Levers</h3><p>We call the fundamental unit of our remote configuration system a “Lever.” Each Lever consists of an identifier, a type (Boolean, String, or Number), and a value. This limited complexity is flexible enough to describe fairly robust configurations while still being straightforward to implement consistently across multiple platforms.</p><h3>Designing Configuration with Levers</h3><p>The configuration described by Levers encourages granular feature control that is platform-agnostic and product-focused. We can reason about parts of the feature that might make good candidates for an experiment before writing any code, which can in turn help us to make better design and architecture decisions.</p><p>One common pattern is to use separate Levers to control the primary gating of the feature itself as well as any parameters of the feature. For example, a new flow might use String Levers for the member-facing copy and button color, another Number Lever to control the delay before a notification pops up, while still maintaining a Boolean Lever that acts as the primary control for whether the member sees the new feature at all.</p><p>The API response might include something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ee72d2e13f04dd68a092a8481fd9c20/href">https://medium.com/media/7ee72d2e13f04dd68a092a8481fd9c20/href</a></iframe><h3>Managing Levers with Config Service</h3><p>The backend is responsible for constructing the payload of Levers when the client requests it. On its own, this payload is purely a delivery mechanism for the remote configuration. On the client-side, we created a “Config Service” on each platform, which is an SDK to process, persist, and vend new values from our Phoenix system. While each Config Service implementation addresses concerns that may be specific to a certain platform, we relied on cross-platform collaboration to achieve feature parity across the iOS, Android, and Web clients.</p><p>One “gotcha” about decoupling the configuration from the client is that the backend might be out of sync with the client. Fortunately, the nature of integrating the Lever into the feature leads us to make some assertions. It wouldn’t make sense for the client to want to interpret the same Lever as two different types, therefore the client must inherently know the type of Lever it is using. This also implies that the client knows all the Levers that it might read.</p><p>We created a manifest to declare all the Levers and their corresponding types, which allows us to validate the values that the backend sends and ignore anything that would violate the type of Lever the client expects. We also declare fallback values in case the server doesn’t send a particular Lever, such as what might happen on an older client after we’ve deprecated a Lever on the backend.</p><p>To add more flexibility, we defined different locking policies (unlocked, session-locked, etc), which regulate when a Lever would take effect after its receipt from backend.</p><h3>Experimentation</h3><p>So far, we haven’t said much about experimentation, so let’s put this into context. Just like remote configuration provides a mechanism for adjusting the app experience, experiments provide a mechanism for adjusting the configuration. Experiments allow us to propagate remote configuration to clients based on targeting criteria, quickly determine if new products are working as intended for our members and measure their impact.</p><p>The Config Service itself doesn’t really have a concept of an “experiment.” It doesn’t make any assumptions about <strong><em>why</em></strong> the configuration is a certain way; the configuration just <strong><em>is</em></strong>. Each Lever comes with an identifier that tells the backend whether the member’s experience is associated with an experiment. The identifier has no particular meaning to the client, but it is included with any key metrics that the feature tracks. This helps us understand the impact of new features powered by Levers while maintaining a separation of concerns.</p><h3>Metrics</h3><p>In the past, we’ve found that while ensuring that our experiment data is accurate and valid is of critical importance, it’s really easy to make mistakes when instrumenting analytics from scratch. We thought that developers should get robust reporting for free by using Levers and Config Service, so we made tracking new values on Lever reads a core feature. This improves the quality of our analytics and reduces the burden of each team to report experiment variants and assignment.</p><p>In our next blog post, we will go into greater detail about how we collect metrics to produce valuable and actionable insights based on experimentation.</p><h3>So, How Did We Do?</h3><p>Config Service and Levers came out of a need for a robust experimentation platform and it ended up offering us more than that. Decoupling feature configuration from the app release cycle gives us an extra measure of safety. It’s neither trivial nor fast to roll back an app release after there is already a problem. However, rolling back a configuration change can be done in a matter of seconds.</p><p>It allows us to take a more nimble, iterative approach to feature development. We want to avoid the case where new work lands late in the release cycle without sufficient time to identify potential bugs. We can land new work on primary development branches early and harden the implementation without fear of shipping an incomplete feature into production.</p><p>We love what we’ve been able to accomplish with this, and we hope you will too.</p><p>Interested in joining Tinder? Click to explore <a href="https://tinde.rs/350rEU5">open opportunities</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e99fbc72af55" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tinder/phoenix-tinders-testing-platform-part-ii-e99fbc72af55">Phoenix — Tinder’s Testing Platform — Part II</a> was originally published in <a href="https://medium.com/tinder">Tinder Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Phoenix — Tinder’s Testing Platform, Part — I]]></title>
            <link>https://medium.com/tinder/phoenix-tinders-testing-platform-part-i-8e51c19ca977?source=rss-6f7fa21af202------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e51c19ca977</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[ab-test]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Juzheng Li]]></dc:creator>
            <pubDate>Wed, 30 Oct 2019 18:00:31 GMT</pubDate>
            <atom:updated>2020-07-16T19:11:38.907Z</atom:updated>
            <content:encoded><![CDATA[<h3>Phoenix — Tinder’s Testing Platform, Part — I</h3><p><strong>Written by: Kenneth Yeh</strong> <strong>|</strong> <strong>Jinqiang Han</strong> <strong>|</strong> <strong>Juzheng Li</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZEXQMN-PttixQQeF.png" /></figure><p>Data influences all of the decisions we make at Tinder. We are often faced with decisions focused on which updates to make within the Tinder experience and how we present features to our members. Experimentation plays a vital role in making these decisions.</p><p>At Tinder, we’ve built an in-house tool, Phoenix, to manage and optimize all of our experiments. We’re excited to share how we’ve done so, and this is the first part of our three-part series highlighting our Phoenix experimentation platform. First, we will cover the key concepts and design considerations. Second, we will cover Levers, our treasured mechanism for decoupling experiments from app release cycle. And last, we will discuss our Metrics System, which helps in deriving data insights and concluding experiments.</p><h3>An Overview</h3><p>From a very high level, our end-to-end experimentation platform, Phoenix, is made up of four components, illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*_gTuUWabCe4Q6z-m" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/0*fG6PUnxnGKF_aLeE" /></figure><ul><li><em>Ground Control</em> serves as the foundation of the platform. It manages the life cycle of an experiment — from creation, to update, all the way to its eventual rollout and termination.</li><li><em>Assignment</em> is the module that determines which treatment each user will be receiving based on the experiment definition.</li><li><em>The Levers System</em> is a client integration SDK that handles, processes and renders the treatment applied to users via assignment on the client side. We will do a deep dive focused on this in our next blog post.</li><li><em>The Metrics System</em> is the heart of the platform. Throughout the experiment’s life, we collect data points across both the client and server sides. All of those events are ingested by Metrics System and are further processed and analyzed daily by a data crunching pipeline. A more detailed post focused on the Metrics System will complete this series.</li></ul><p>Now, let’s take a good look at the setting up of an experiment.</p><h3>The Many Knobs</h3><p>There are many different types of experiments. When the time comes and you need to set up an experiment, you will be faced with many knobs to turn.</p><h3>Which Side of Auth?</h3><p>You need to decide on which side of the auth wall you’d like to conduct your experiment. Pre-auth experiments are done mostly when a user is not logged in yet. If you are testing your registration flow, for instance, that’s the type of experiment you’d employ. The experiment unit or the experiment key field to use in these cases would be a device ID.</p><p>Once there is a registered user, you can deploy post-auth experiments. The experiment unit or key field would be a user ID in this category. An inherent advantage of using a user ID is that, even if the same user switches devices, they will still be served the same experience.</p><h3>Which Side of the Network?</h3><p>Another decision you need to make is whether to conduct a client-side experiment or a server-side experiment. As their names imply, they inherently differ in certain key aspects. If you’re working to optimize user action flow, it would naturally be a client-side experiment; and if you’re tuning the recommendation algorithm, it would be a solid server-side experiment.</p><p>It’s also significant to note that, traditionally, every time you want to conduct a client-side experiment, you will need to pair that with an app release cycle — which may take days or sometimes weeks. Comparatively, most engineering teams would be able to handle a server-side module release and carry out a new server-side experiment in a more timely fashion. Thanks to Phoenix Levers System, we are able to close that gap in timing. We will share more about that in our next post.</p><h3>Sticky</h3><p>When setting up an experiment, it’s also important to decide whether the assignment should be sticky — or remain active when conditions change — or not. This matters when the user changes some of their attributes, like switching devices or traveling to different regions. Making an experiment sticky will maintain the experiment treatment even if factors like these change.</p><p>Being sticky comes in handy for some use cases when it is important to have a consistent user experience. If an experience is geo-fenced, though, a user’s qualification should be tightly guarded by real-time attributes and you should leave the sticky box unchecked.</p><p>It is interesting to note that all those above settings are orthogonal: you can have a non-sticky client-side pre-auth experiment <em>and</em> you can have a sticky server-side post-auth experiment.</p><h3>We are Family</h3><p>It is complicated to set up your experiment. And it gets even more complicated when there are multiple experiments — especially when two or more experiments may have competing effects. This could pose a grave danger of rendering experiment analysis ineffective. For example, what if the same user is exposed to two competing experiments? How will you attribute the gain or drop of some key metrics?</p><p>To guarantee that no user is exposed to two competing experiments, Phoenix offers a concept called an experiment family. This gathers competing experiments into the same experiment family. And within the same experiment family, we employ a tie-breaker when two competing experiments are targeting the same user. As a result, the user is only assigned to one of them. At Tinder, we often use priority as a tie-breaker.</p><h3>Hashing your assignments</h3><p>When all the dust is settled, you may be left wondering how the treatments received by each user are ultimately decided. The key mechanism here is hashing. We hash the user identifier, whether that is the device ID or user ID, and use the hash and modded number to decide the specific treatment allocation.</p><p>Solely using a user identifier means that the same user will have the same hashed and modded numbers, and may be assigned to the control treatment for many different experiments (it is common for most experiments to have a control and it is often the first treatment out of the many possibilities). To solve this issue, we throw in the experiment ID as a salt, which is derived from a combination of the experiment family and the experiment itself.</p><p>Another potential problem with this combined salt and identifier approach is that, if we want to shuffle the treatment allocation for any reason (e.g.: a bug, re-start, or second version of the experiment), we won’t be able to do it. To solve this issue, we added a shuffle ID.</p><h3>Experiment or Feature Gating</h3><p>Lastly, quite often people will use an experiment to achieve the goal of gating a feature. There are some fundamental differences between the two: feature gating serves as more of a <em>CONTROL</em> and experimentation serves as more of a <em>COMPARE</em>.</p><p>Often, we only release a feature to certain users — like those in a list of selected test countries. We call this feature gating. If you are in a qualified country, you should have that experience; if not, however, you should not see it. When releasing a feature behind a gate, the focus should be the targeting rules that establish whether a user qualifies to see the feature. Based on this, you CONTROL the experience.</p><p>Experimentation, however, operates on a different level with a different focus. The main idea of conducting an experiment is that you want to COMPARE things. You want to see how A is different from B; how <em>enabled</em> is better / worse than <em>control</em>. How <em>new</em> is different from <em>old</em>. With experimentation, you can add targeting rules (just like you did for feature gating), but users that qualify the using these rules will be further split into different treatments for the sake of comparison.</p><h3>Till Next Time</h3><p>Tinder is about creating a new spark. Experimentation is about iteration. Phoenix is the magical creature born and reborn into a better self through fire. We hope that through rapid experimentation, Phoenix can help Tinder users achieve stronger matches, engage more effectively, and create meaningful relationships, at a truly global scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e51c19ca977" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tinder/phoenix-tinders-testing-platform-part-i-8e51c19ca977">Phoenix — Tinder’s Testing Platform, Part — I</a> was originally published in <a href="https://medium.com/tinder">Tinder Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>