<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Josh Bradt on Medium]]></title>
        <description><![CDATA[Stories by Josh Bradt on Medium]]></description>
        <link>https://medium.com/@josh.bradt?source=rss-ea4b7084c1ac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hBOQQK9D7QDU8sCrRs7OAA.jpeg</url>
            <title>Stories by Josh Bradt on Medium</title>
            <link>https://medium.com/@josh.bradt?source=rss-ea4b7084c1ac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:06:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@josh.bradt/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Designing the Segmentation Improvement Project]]></title>
            <link>https://klaviyo.tech/designing-the-segmentation-improvement-project-d7f4e5c522fa?source=rss-ea4b7084c1ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7f4e5c522fa</guid>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Josh Bradt]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 13:54:53 GMT</pubDate>
            <atom:updated>2024-06-03T13:54:53.589Z</atom:updated>
            <content:encoded><![CDATA[<p>What’s the safest and most effective way to speed up a core component of an application without interrupting customers’ work? The best answer is usually building on the current system and squeezing out improvements with small, well-tested changes that iterate on the design. This gets gradual improvements in front of customers as soon as possible, and it limits risk.</p><p>But what if you need to speed the system up 10x or more? What if what used to take an hour should now be done in 5 minutes or less?</p><p>That’s a big project, the kind of thing that makes project managers start drawing Gantt charts and writing down lists of dependencies. But if you’ve got a Gantt chart, that means you’re doing waterfall project management, and that can lead to the trap of thinking you can create a flawless system design and schedule at the start of the project, when you know the least about what you’re building.</p><p>A better plan is to do something in between. That is, plan ahead, but only <em>just enough</em>.</p><p>That was the target we aimed for with the Segmentation Improvement Project, which we introduced in an <a href="https://medium.com/klaviyo-engineering/taking-the-first-sip-an-overview-of-klaviyos-segmentation-improvement-project-7db997f36b39">earlier blog post</a>. We were tasked with improving segmentation performance enough that even our largest customers could run segments filtering millions of profiles having hundreds of millions of events in 5 minutes or less, something that could take over an hour before. We’d already iterated on the existing segmentation engine for years, and we’d run out of clever hacks to squeeze out more performance. It was time for a rewrite, and we wanted to find a way to maintain some agility while also planning out the project to reduce risk. In the rest of this post, I’ll talk about the approach we took to balance these two extremes of project management philosophy.</p><h3>Planning philosophy</h3><p>For past projects, we’ve generally tried to balance two schools of thought. On the one hand, we could follow a traditional “waterfall” process where we design everything at the start, make robust staffing plans, and set deadlines for each deliverable and phase of the project. This is appealing if you want to stick an end date on the project and fit it into a roadmap, but waterfall processes have gotten a bad reputation since it turns out it’s quite difficult to accurately predict the future. Even with our best due diligence, full system designs can miss edge cases, requirements can change, and people take well-deserved time off. Deadlines slip and rework is required. This isn’t anyone’s idea of a good time.</p><p>At the other end of the spectrum is an agile process. I’m using a lower-case “a” here deliberately since I’m talking about working iteratively, delivering value gradually, and designing on the fly regardless of whether we’re following a formal, capital-A Agile process with sprints, retros, and other ceremonies. In this kind of process, we’d do very little to no design up front, and we’d build and refactor as we go. In a weird way, this cuts the risk of incorrect designs and rework because it <em>assumes</em> those things will happen and builds it into the timelines. It’s also appealing because the beginning of the project is when we know the <em>least</em> about what we’re building, so it’s hard to make a flawless technical design at that point like a waterfall process would require. That said, it’s even harder to predict a finish date in a process like this, and while a moderate level of refactoring is good and healthy, if we don’t give the design any thought up front, we risk backing ourselves into technical corners that are expensive and time-consuming to get out of. (“Oops, should we have used a queue here instead of sprinkling synchronous calls everywhere?”)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WhwxWSmXgEMKHjM_1JlawQ.png" /><figcaption>It’s better to make changes early in the project’s lifecycle, before other decisions have been based upon what you’re changing. The longer you wait, the more rework you need to to. (Image borrowed from <a href="https://www.riskupfront.com/">Risk Up Front</a>, Chapter 1.)</figcaption></figure><p>For this project, we tried to strike a balance between the two. <em>Waterfall</em> isn’t necessarily a dirty word, and there’s also nothing wrong with iterating a bit! Taking inspiration from <a href="https://www.riskupfront.com/"><em>Risk Up Front</em></a>, we started by identifying the most important, highest risk design decisions, and then we worked on reducing those risks by researching, testing, and prototyping solutions until we felt confident that the system would work. Then we designed the interfaces for each component of the system and wrote up a draft of how each component should work. From that point, the process became more iterative as we built out the components and worked to make them meet their requirements, a process that was made practical by building behind feature flags in production. Iterating in production sounds scary, but it’s actually quite safe and effective when feature flags are used to carefully hide the incomplete systems from customers. This let us double-write real customer workloads onto the new system to see how it behaved, which is much more accurate than trying to simulate load in a separate environment.</p><p>The risk inherent in this project management approach is that you might not identify the most important risks. No amount of planning can eliminate the unknown unknowns, though we also missed our fair share of <em>probably</em>-should-have-known unknowns. The biggest one of these was that we didn’t load test our usage of message buses in a proof of concept. I’ll talk more about how we recovered from this in a later section below.</p><h3>Design phases</h3><p>The design process was broken out into roughly three phases. First, we did research, solidified requirements, prototyped critical components of the proposed system, and chose an overall path. Next, we prepared an architecture document describing the system at a high level. Finally, we described the design of each component in detail.</p><h4>Research and requirements gathering</h4><p>We took care of this first phase of the project with a very small group of people including me and another engineer, a product manager, the team’s engineering manager, and a select group of other key stakeholders who could help make decisions about the project’s scope, eventual staffing, and requirements. The first goal at this phase was to figure out what exactly we were doing. We knew segments were too slow in large accounts, but why was that the case? How fast did we want to make them? And, crucially, what scope of work were we willing to take on?</p><p>We needed more information to answer any of these questions, so we started a free-form engineering research notes Google Doc and dug in. We documented the behavior and interfaces of the existing system in detail and gathered metrics from our <a href="https://klaviyo.tech/scaling-graphite-to-millions-of-metrics-543b830877f7">stats stack</a> that would help us understand the timing of various parts of segment evaluation. At this point, we concluded we’d probably need to involve an OLAP database in the system somewhere, so we also started researching and prototyping various options and writing out their pros and cons in this doc. Though the doc grew quite long, it ended up being easiest to keep all this information in one place, and it’s a pattern we’ve used again in later projects.</p><p>After doing our research, we came up with four very rough designs with slightly different properties. We thought it would be best to have a few alternatives of varying scope to choose from since the concrete requirements were still being narrowed down at this point, and we weren’t sure yet exactly how much engineering time we were willing to commit to. We brought these alternatives to the weekly sync meeting we were having with the other stakeholders, and after discussing the tradeoffs of each option, we collectively made a decision to move on with the most ambitious but also the most impactful of the four proposals.</p><h4>High-level design phase</h4><p>Like any project, we wanted to parallelize work as much as possible to make things go faster, but we concluded it would make the most sense to have one person get the whole system thought out in their head first. With one person defining the interfaces between the components and thinking about the system holistically, we’d have a better chance of the components working together correctly the first time. So, I sat down and started drawing pictures.</p><p>The first (and easiest) step was to identify the major components of the system. Where would we store the data? How was the data going to get there? Where were queries going to be run? The answers to these questions let us break the system down into subsystems and components. Then we had to figure out the interfaces these components would use to talk to each other and what schemas we’d use to store data in our databases. Figuring out what data would be passed around was fairly straightforward, but estimating the throughput on each interface was a bit of an art. Luckily, we already had a fairly robust set of timers and metrics on the existing code we were replacing, and those were a decent starting point.</p><h4>Detailed designs</h4><p>After sharing the architecture doc to get feedback, we moved on to writing up detailed technical designs for each major component of the system. At this point, the work defining the interfaces paid off since we could now split up the detailed design work across several engineers to get it done in parallel.</p><p>The detailed design documents covered the major components in more detail than the high-level design. This included things like the specific technologies we’d use to implement the component, how the component would work, and any database schemas we hadn’t already defined in the high-level design. We didn’t go as far as to outline the class structure of the code, though in retrospect, it would have been helpful to do that to some reasonable level to avoid a few design mistakes we made later in the implementation phase.</p><p>Once the detailed designs were written up and shared with other impacted teams for feedback, we were ready to start building.</p><h3>Iterating on our designs</h3><p>Even the best plans rarely survive contact with reality completely unscathed, and our designs for SIP were no exception. Despite our best planning, there were a few parts of the system that we had to rethink several times before we got to a working solution.</p><p>One component we had a bit of trouble with was what we called the Membership Log Watcher. This component queries the segmentation system’s central ClickHouse database to extract segment results, and it persists these results to the databases used by the rest of the Klaviyo app to say who’s in a segment and who’s not. Our original design had this component publish a message to Klaviyo’s internal message bus for each segment membership change, which was an elegant design that decoupled the segmentation microservice from the rest of the Klaviyo app monolith. When we built this and started testing it, though, we quickly discovered that we could neither publish nor consume messages fast enough to keep up with the peak of over 20 million membership changes per second we were generating. Given that, we had to iterate to find a solution that <em>would</em> work. We went through a few cycles of efficiency improvements, message deduplication, and batching of individual updates together in application code, and eventually we decided to scrap the queue entirely and just write directly to the databases from our microservice until we could come up with a more scalable way to store these results.</p><p>The lesson we took from this was that it was ok to iterate on a solution, but it would have been cheaper and easier if we had gotten that out of the way earlier in the process. We should have done more load testing on the message bus to prove that we could publish at the rate we needed, or maybe we could have built a proof-of-concept model of this component to test the design further before committing to writing a production-grade version of it. Being able to safely iterate in production was generally a big benefit for this project since it let us test with real production traffic, but it also had the cost that everything had to be production-quality and meet things like coding and test coverage standards, unlike a proof of concept outside the production environment where quick and dirty code is the expectation.</p><h3>Takeaways</h3><p>Though we’ve been fans of planning and designing things ahead of time for a while, the segmentation improvement project was the first time we really leaned into the idea and did a lot of up-front design. Here are a few things we took away from this process:</p><ol><li><strong>Planning ahead is still a good thing!</strong> It’s way faster and less expensive to iterate on paper and in a prototyping context during planning than a full production environment after implementation has started.</li><li><strong>Trust but verify.</strong> Build a proof of concept. Test it. Don’t assume things will work just because your rates are within a published SLO.</li><li><strong>Minimize single-threaded design work, but it’s ok to have some of it</strong>. It was very helpful to parallelize the design work, but that was only made possible by the initial, overarching design work that was less parallelizable.</li><li><strong>Make sure you get the interfaces right before building anything</strong>. Specifying the interfaces was a pain, and not the most fun, but when a half dozen components all written in parallel by different engineers all started up and talked to each other the first time, it was all worth it.</li></ol><p>We’re already applying these ideas to our upcoming projects over the next few months, and hopefully the planning for those will go even better than this did!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7f4e5c522fa" width="1" height="1" alt=""><hr><p><a href="https://klaviyo.tech/designing-the-segmentation-improvement-project-d7f4e5c522fa">Designing the Segmentation Improvement Project</a> was originally published in <a href="https://klaviyo.tech/">Klaviyo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking the first SIP: An overview of Klaviyo’s Segmentation Improvement Project]]></title>
            <link>https://klaviyo.tech/taking-the-first-sip-an-overview-of-klaviyos-segmentation-improvement-project-7db997f36b39?source=rss-ea4b7084c1ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/7db997f36b39</guid>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[olap]]></category>
            <category><![CDATA[customer-segmentation]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <dc:creator><![CDATA[Josh Bradt]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 17:39:05 GMT</pubDate>
            <atom:updated>2023-07-10T17:39:05.784Z</atom:updated>
            <content:encoded><![CDATA[<h3>Taking the first SIP: An overview of Klaviyo’s Segmentation Improvement Project, or how we learned to stop worrying and love OLAP databases</h3><p><em>This post was co-authored by </em><a href="https://medium.com/u/ea4b7084c1ac"><em>Josh Bradt</em></a><em> and </em><a href="https://medium.com/u/fc44a6d538b5"><em>Nicholas Hoffmann</em></a><em>.</em></p><p>One of the most interesting parts of working on the engineering team at Klaviyo is watching how huge systems respond to increasing scale as the company grows. We try to build systems that will scale up to meet increasing demand and incoming data rates for at least the next 3–5 years. As the end of a system’s useful life approaches, though, it’s fascinating to watch what we thought was our best work gradually start to fall behind.</p><p>Over the last year, we took on a big project to replace Klaviyo’s segmentation engine, which we called the Segmentation Improvement Project (SIP). Segmentation lets our customers create dynamically updating audiences from the profiles in their Klaviyo accounts based on extremely flexible user-defined rules. This system has to respond to billions of changes to data points across the Klaviyo platform each day to keep millions of segments up to date in real time. The existing system was starting to reach the limits of what we could do to scale it further, so over the last year, we designed and implemented a new, more scalable replacement segmentation engine. In this blog post, we’ll talk about what segmentation is, how the old system worked, and the design of our new and improved segmentation engine.</p><h3>What is a segment?</h3><p>Though the Klaviyo app is flexible and is used for many different use cases, our typical user is a direct-to-customer ecommerce business. These businesses integrate Klaviyo with their online storefront provider to track things like which products their customers are viewing, customers’ purchase histories, and properties about their customers. Businesses can use this data to send well-targeted, personalized email and SMS communications to their customers with the goal of sending people marketing messages they’re actually interested in seeing.</p><p>At the core of this system is the profile, which represents a person who can be communicated with. Profiles have both identifying properties (like an email address or phone number) and other attributes (like a mailing address, or whatever custom properties a business needs).</p><p>Segments are dynamic groupings of profiles based on their attributes. Customers define a set of rules that they want to use to include profiles in a segment. The segmentation engine then compares all the profiles in the customer’s account to the segment definition and records which profiles belong in the segment. Segments are also automatically updated as new data enters Klaviyo. For example, if the definition says that the profile must have looked at jeans on the web site in the last ten days, and someone just did that, they will be immediately included in the segment.</p><p>Segment definitions are composed of a set of criteria that determine the segment’s membership. There are many different types of criteria, and they draw data from across the Klaviyo application, making segments a very flexible tool. Some example criteria include:</p><ul><li>People who have placed an order at least once in the last 30 days</li><li>People who have clicked a link in a specific email campaign</li><li>People who live within 10 miles of the ZIP code 02110 in Boston, MA</li><li>People who have consented to receive SMS messages</li><li>People whose expected date of next order, based on their activity history, is in the next two weeks</li></ul><p>These criteria can be combined with AND and OR to make arbitrarily complex logic.</p><h3>The original segmentation engine</h3><p>At a basic level, the approach of our original segmentation engine was to pull data from the various underlying data stores powering the Klaviyo app, check the data against the segment definition, and record which profiles qualified for the segment.</p><p>Segmentation operates in three different modes:</p><ol><li>When a new segment is created, the system needs to calculate the initial set of memberships from all profiles in an account. This processing mode is also used if the definition of an existing segment is updated. We call this <strong>manual segmentation</strong> since it’s triggered on-demand by the Klaviyo customer.</li><li>Memberships in existing segments are kept up to date by a real-time processing mode as events and other data about each profile are streamed into Klaviyo. Customers rely on their segments being up to date and ready to use for things like email campaign sends and segment-triggered flows, so it’s critical that <strong>real-time segmentation</strong> keeps to a tight SLO.</li><li>Finally, a <strong>nightly</strong> <strong>relative time check</strong> is an extension of the real-time processing mode that accounts for membership changes that should happen due to the passage of time. As an example, if there is a criterion that checks if zero orders were placed in the last 30 days, there is no “non-event” that happens on day 31 that tells the system to update any relevant memberships. The relative time check will evaluate that criterion periodically and make sure everything is kept up to date.</li></ol><p>The original segmentation engine worked by directly querying the many OLTP databases that store the data needed to calculate who belonged in a segment and then combining this data in Python code. The latest incarnation of this was a modular system with standard interfaces for fetching and caching data from the source of truth data stores, formatting it for use with the segmentation engine, and evaluating segment criteria against it. This model was flexible and easy to extend since the query and evaluation logic needed for each type of segment criterion was nicely encapsulated in one place.</p><p>The catch was that each backing datastore needed to be able to tolerate the segmentation engine’s scale. While some data was read from previously computed aggregates stored in columnar stores like Cassandra, in many cases we were effectively performing analytical queries against OLTP databases — typically Amazon Aurora MySQL clusters — and that’s to say nothing of the sheer volume of data we were sending over the wire. As a result, the segmentation system had knocked over its fair share of data stores over the years, earning it the internal nickname “the ray gun of death.” It was particularly effective at degrading the Cassandra nodes storing pre-aggregated event data, which was unfortunate since event-based criteria are the second most commonly used criterion type, being evaluated roughly 100 million times per minute on a typical day.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3K0D1Dkqh9MOmLSjzW%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExamZvdjhoZDUwd2RodGI1dmp6eXd1eGlhb2ZiaGxyNzBseWp1NDAwcyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2F3K0D1Dkqh9MOmLSjzW%2Fgiphy.mp4&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F3K0D1Dkqh9MOmLSjzW%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ada60d22428205555ef0ebbf2572dd2/href">https://medium.com/media/1ada60d22428205555ef0ebbf2572dd2/href</a></iframe><p>These Cassandra clusters with pre-aggregated counters for events were one key optimization in the old system. Others included careful batching of work to allow Cassandra to read continuous ranges of data from disk, tuning queries to rely on indexes in MySQL databases, some ingenious task spawning tricks that relied on knowing implementation details of our profile schema to accelerate how fast we could create new segments, and an elaborate Redis-based system for deduplicating real-time segment update triggers within sliding time windows.</p><p>By the start of 2022, though, it had become clear that we were running out of tricks. The segmentation system was successfully handling more data than ever, with 5.19 billion real-time segment updates being triggered on Black Friday alone in 2021, but we were having to run an unsustainable amount of hardware to support the workload. With hundreds of nodes in total, the Cassandra clusters that held our event counters in particular were getting prohibitively expensive with each round of scaling out. As Klaviyo added larger and larger customers, we also wanted to make sure that they got the same speedy performance that everyone had come to expect from the segmentation engine. That realization led us to consider a deeper overhaul of the segmentation engine and think about what an alternative might look like.</p><h3>Requirements for the new system</h3><p>The team had worked with the segmentation engine for several years at this point, so we had a good idea of where the gaps and shortcomings were. In addition to the constant desire to be faster, we had built up a backlog of feature requests and improvements that were difficult or impossible to build in the old system. We also knew that we wanted the new system to cost significantly less to operate than the old one. Ideally it would be easier to operate as well since our Cassandra clusters were often at the heart of some of our longest and most challenging incidents.</p><p>Using all of that as a starting point, we worked with our product managers to come up with a set of primary requirements for the project. After a bit of back and forth, we chose an arbitrary performance target that would be a great experience for customers, and which we suspected was within the realm of technical possibility with good engineering. Choosing a target brought focus to our discussions and allowed the project to move forward. A lot of our early design work then went into validating this target as something that was feasible before we went on to build the rest of the system.</p><p>By the end of Q1 of 2022, we had a set of primary requirements that set a clear destination for our desired end state. We also had several high-level design options for engineering to pursue further. The requirements we ended with were:</p><ol><li>Create a segment in less than 5 minutes at p99 for an account with 60M profiles</li><li>Be able to keep up to 18M segments up to date in real time</li><li>Support all existing functionality from the current segmentation engine</li><li>Support known future feature development</li><li>Operating costs no more than 50% of the cost of the existing system</li></ol><h3>Overview of the new system</h3><p>After completing our research and testing prototypes, we concluded that the only way to hit our performance target was to turn the segmentation model on its head: instead of bringing the data to the queries, we’d bring the queries to the data. The right tool for this job was an OLAP database.</p><p>We’d actually tossed around the idea of running segmentation on a giant OLAP database for years, but we’d dismissed the idea before since it would require duplicating all of the data we base segments on and keeping that duplicate copy in sync with the source of truth, which is a substantial task. However, after a few years of rapid growth, the engineering team had already invested in defining robust domain boundaries around each Klaviyo product area with all updates being funneled through service classes. This earlier work — along with the increased capacity of the team — now made this previously formidable project something we could actually consider.</p><p>We also had the benefit of running an OLAP database in another part of Klaviyo. Another team that we work closely with was using <a href="https://clickhouse.com/">ClickHouse</a> to power the latest iteration of our account-level event and reporting functionality. Choosing technologies that other teams at Klaviyo had already used successfully was a big part of balancing risk and innovation for this project. We wanted to use the best tools available and stay aligned with the broader technical direction of the company, but we didn’t want to get too far out ahead of everyone else. We did spend a few more “<a href="https://mcfunley.com/choose-boring-technology">innovation tokens</a>” by running the new segmentation engine on Kubernetes and using Python’s <a href="https://docs.python.org/3/library/asyncio.html">asyncio</a> framework, both of which were newer to the team, but here again we were able to follow a path blazed by others. We compensated for this relative novelty in other areas by choosing “<a href="https://mcfunley.com/choose-boring-technology">boring</a>” technologies like Python and Django that we were experts in to help manage risk.</p><p>The new segmentation engine has a 192-node ClickHouse cluster at its core. We considered a few alternatives for the database, but we landed on ClickHouse because, as mentioned earlier, we already had in-house experience using it for event analytics, and it also supported all of the features we wanted for our long-term roadmap.</p><p>The ClickHouse cluster is kept up to date by custom change capture pipelines that funnel in 1.5 billion profile trait updates per day and nearly 2 billion events per day from the rest of the Klaviyo app. Every time something about a profile changes — including things like their contact details, custom properties about them, their subscription status, or predicted attributes about them from our data science team — a change notification message is sent into a topic in our internal messaging service. Change notification handlers respond to those messages by reading the current profile trait values from the source-of-truth databases we used to query in the old segmentation system, serializing the result into a format close to what we’ll insert into ClickHouse, and sending the result into a second messaging service topic. Finally, these trait change messages are grouped by ClickHouse shard and inserted into ClickHouse in large batches.</p><p>With all of the data colocated on our one giant ClickHouse cluster, the next task was to reimplement all of our segmentation logic in SQL. This was quite complicated since a core benefit of the segmentation engine is its flexibility: we allow our customers to combine as many criteria as they need to get the job done. Additionally, we needed to be able to process many segments at once in large batches since ClickHouse works best with a low rate of more complex queries. These requirements meant that we couldn’t dynamically build SQL queries in code based on the customer’s segment definition (to say nothing of the security risks that would entail), so we created a way of encoding segments as sets of filtering rules that we could store in the ClickHouse cluster and join to our data at runtime. The query that supports this weighs in at over 1300 lines of heavily optimized SQL. The result of this query is the list of segment members, which is written to a table in ClickHouse to be read out asynchronously by another process that syncs those back to the main Klaviyo app.</p><p>The full system took us a little over a year to design, implement, tune, and release, and the results were worth it. We met all of our project requirements. Most end users will have noticed no difference (which is good!), but for users working with large segments, the speed improvements are striking:</p><figure><img alt="A histogram showing the difference between the old and new segmentation engine’s segment creation times. The old engine had a long tail of segments that took up to 90+ minutes to process, but the maximum processing time in the new engine is less than 15 minutes." src="https://cdn-images-1.medium.com/max/1000/0*Jmwmy8hQq5kMREkI" /><figcaption><em>Histogram of segment creation time, exponential scale. Even segments with 20+ criteria for accounts with tens of millions of profiles can be created in under fifteen minutes during peak system load, which was not the case with the old system.</em></figcaption></figure><h3>Conclusion</h3><p>The Segmentation Improvement Project is the largest project that the Segmentation Team has undertaken to date and is among the largest projects that any team at Klaviyo has ever done. The project was an opportunity for the whole team to do some of the best and most engaging engineering work of our careers so far. This summary blog post presents the project as a straightforward process of requirements, design, build — but in fact, the project was filled with fascinating challenges. We learned important lessons around technical design, leveraging the best parts of the technologies we chose, and managing such a large project. We’re planning on discussing all of those topics in future posts, and we’ll link them here as they are published.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7db997f36b39" width="1" height="1" alt=""><hr><p><a href="https://klaviyo.tech/taking-the-first-sip-an-overview-of-klaviyos-segmentation-improvement-project-7db997f36b39">Taking the first SIP: An overview of Klaviyo’s Segmentation Improvement Project</a> was originally published in <a href="https://klaviyo.tech/">Klaviyo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>