<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Faire Data and Engineering Team on Medium]]></title>
        <description><![CDATA[Stories by Faire Data and Engineering Team on Medium]]></description>
        <link>https://medium.com/@faire_data_and_eng_team?source=rss-782380e4dcad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QqyXKlEULOta2V2X9wxO4Q.png</url>
            <title>Stories by Faire Data and Engineering Team on Medium</title>
            <link>https://medium.com/@faire_data_and_eng_team?source=rss-782380e4dcad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:40:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@faire_data_and_eng_team/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Conductor: Faire’s platform for orchestrating in-app messages]]></title>
            <link>https://craft.faire.com/conductor-faires-platform-for-orchestrating-in-app-messages-ceb5197972f9?source=rss-782380e4dcad------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceb5197972f9</guid>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[sdui]]></category>
            <category><![CDATA[in-app-messaging]]></category>
            <category><![CDATA[mobile-design]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Faire Data and Engineering Team]]></dc:creator>
            <pubDate>Tue, 20 Jan 2026 20:34:25 GMT</pubDate>
            <atom:updated>2026-01-20T20:34:25.160Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we show the right message to the right user at the right time</h4><p>Written by <a href="https://www.linkedin.com/in/divyasudhakar/">Divya Sudhakar</a> and <a href="https://www.linkedin.com/in/julian-s-pettit/">Julian Pettit</a></p><figure><img alt="An abstract illustration of a laptop displaying interconnected UI components, representing Conductor orchestrating in-app messages." src="https://cdn-images-1.medium.com/max/1024/1*jZ-oGMnB6iCULsEaDi2F5A.png" /></figure><p>How do you communicate the right message to the right users at the right time? At <a href="https://www.faire.com/">Faire</a>, the Engineering team grappled with this challenge as our product and company grew. Different teams were adding banners, nudges, modals, and alerts throughout the app to promote and upsell their various programs. Without a unified system, the result was often fragmented with duplicated logic and inconsistent user experiences. On some of our highest impact pages like the checkout page, a retailer might have encountered up to five different banners at once. These were controlled by disparate client code paths and often looked entirely different, with no common design language. In short, we were <a href="https://en.wikipedia.org/wiki/Conway%27s_law">shipping our org chart</a>.</p><p>In addition to our product looking chaotic, this made it arduous for engineers to add or modify messages since the display rules were scattered across codebases. We needed a better way to coordinate customer-facing messages that would streamline development and ensure a polished experience for our users.</p><p>Our answer was <strong>Conductor</strong>, an internal platform that now serves as a one-stop shop for creating and managing any in-app message we show to retailers. Conductor lets us define what message to show and when to show it using a low-code UI, and takes care of delivering that message to all eligible users across web and mobile platforms. Having all our promotional messages flow through Conductor also lets us prioritize which messages to show and create hyper-personalized content to ensure that users are seeing the most relevant and engaging messages.</p><figure><img alt="A diagram showing Conductor coordinating messages from Faire’s backend services and delivering targeted in-app message components to web and mobile clients." src="https://cdn-images-1.medium.com/max/1024/1*yTrowG5Zz4aRRisxpZZmnQ.png" /><figcaption><em>Conductor sits in between the messages our Faire teams want to show and our users.</em></figcaption></figure><p>In this post, we’ll share how Conductor works, why we built it, and how it successfully powered a major refactor of our checkout page banners. We’ll discuss the technical approach and use of server-driven UI components, and the impact we’ve seen: dramatic drops in bugs and a boost in developer velocity.</p><figure><img alt="A screenshot showing Conductor’s self-serve low-code tool displaying a list of campaigns." src="https://cdn-images-1.medium.com/max/1024/1*i0ot3-HnR5piCcM25N8zYA.png" /><figcaption><em>A list of Conductor “campaigns” in our self-serve, low code tool.</em></figcaption></figure><h3>The challenge and our approach</h3><p>By late 2024, our approach to in-product messaging was hitting its limits. Each product team would implement their own banners or modals, often hard-coded into the frontend (more on this later). These banners were not consistently organized; an audit of the checkout surface found this code spread across multiple frontend packages, with no single source of truth for when each should appear. This led to several pain points:</p><ul><li><strong>Engineering complexity</strong>: Adding or updating a banner required digging through various code paths to find the right place to insert logic. Removing an outdated banner was equally slow, since it was not immediately clear what other components might interact with it.</li><li><strong>An outdated mobile experience:</strong> This client-centric engineering approach meant teams required one or more frontend, Android, and iOS engineers to build any given feature. In practice, if all the client engineers were not available simultaneously, the feature would launch on the web first. Mobile updates would often be delayed. The mobile experience tended to lag behind the web experience resulting in fragmentation and lack of cohesion for clients who use the apps and the web.</li><li><strong>No global coordination</strong>: Because each banner’s logic was isolated, nothing prevented five different teams’ banners from showing all at once. We saw instances of many banners stacking on the same page, cluttering the UI and potentially confusing users.</li><li><strong>Inconsistent design and UX</strong>: With each team building banners independently, styling and placement varied. Some banners used templated designs, others were one-offs, leading to a jarring experience and more work for our design and engineering teams to review each time we implemented them.</li></ul><figure><img alt="A screenshot showing a wizard in Conductor’s self-serve admin tool for creating a modal in a campaign." src="https://cdn-images-1.medium.com/max/1024/1*YZ0HDPOevXy8tSN_7Z5pBQ.png" /><figcaption><em>The form for creating a new modal in the self-serve tool.</em></figcaption></figure><p>We built Conductor to address these challenges with a focus on:</p><ul><li><strong>Self-serve configuration</strong>: Messages are configured via a low-code admin UI, with no code deployment needed for new or modified messages.</li><li><strong>Standardized and opinionated presentation</strong>: All messages use pre-built server-driven UI (SDUI) components from our internal design system.</li><li><strong>Centralized coordination</strong>: With full knowledge of all the messages that can be shown to a user, Conductor determines what message to show. Indeed, this aspect of “conducting” the messages was so important that we named the platform after this capability.</li></ul><p>This turned our previously scattered banner logic into a coordinated, declarative system that improved both developer velocity and user experience.</p><h3>Architecture and terminology overview</h3><figure><img alt="Table defining key Conductor terms: Campaign as a collection of related messaging components, Component as a server-defined UI element such as a banner or modal, and Trigger as an event or condition that shows a campaign’s components to a user." src="https://cdn-images-1.medium.com/max/1024/1*Msx1LmHW2MZ2MUSJsA8M5w.png" /><figcaption><em>An overview of Conductor terminology.</em></figcaption></figure><p>Conductor runs as a backend service that delivers <a href="https://craft.faire.com/transitioning-to-server-driven-ui-a76b216ed408">server-driven UI (SDUI)</a> <strong>components</strong> (for example: banners and modals) to eligible users.</p><p>A <strong>campaign</strong> defines a messaging theme or goal — such as a promotional push or a failed payment alert — and can include multiple <strong>components</strong> across different parts of the app. Each campaign encapsulates targeting rules and priority. Components use standard UI templates from our design system, Slate.</p><p>Users are enrolled in these campaigns through async events we call <strong>triggers</strong><em>.</em> For example, a trigger might fire when a user receives a promotion, enrolling them in the corresponding Conductor campaign to display relevant messages. Campaigns can also target all users globally, as we often do during our biannual Market events.</p><figure><img alt="System architecture diagram showing Conductor as a backend service that receives client requests, evaluates campaign eligibility and priority, and delivers server-driven UI components to web and mobile clients while logging analytics." src="https://cdn-images-1.medium.com/max/1024/1*KscahRSDQKXq5GtoEwy3YA.jpeg" /><figcaption><em>The Conductor System Architecture.</em></figcaption></figure><h4><strong>Runtime flow</strong></h4><ul><li><strong>Client request</strong>: When a page loads (e.g., checkout), it calls Conductor with user context and surface (e.g., CHECKOUT_PAGE_TOP).</li><li><strong>Eligibility evaluation</strong>: Conductor filters active campaigns by audience, triggers, and enrollment status to determine whether or not a user is eligible to receive messaging.</li><li><strong>Prioritization</strong>: If multiple campaigns qualify, it returns only the highest-priority ones within configured limits. Prioritization ensures that only a specified number of campaigns display per surface.</li><li><strong>Rendering</strong>: The client renders messages using our SDUI framework, which can be displayed directly as provided.</li><li><strong>Analytics</strong>: Conductor natively logs every impression and interaction with components. This gives us high quality data on what messages users are receiving and which ones are resonating with them we can use to debug issues and train our model. These guardrails allow product teams to experiment and iterate quickly without compromising user experience.</li></ul><p>This system keeps client code simple. Campaign logic, targeting, and presentation live on the backend, while clients just render the result.</p><h3>Showing the right message at the right time</h3><p>While Phase 1 of Conductor focused on improving developer velocity and design cohesion, Phase 2 centered on “intelligence.” As more campaigns migrated to Conductor, <strong>prioritization</strong> became one of its most valuable features. We also saw growing concern about “banner blindness,” prompting requests for <strong>guardrails</strong> to prevent showing the same message repeatedly to users who weren’t engaging with them. A central repository for messages — with a system to govern it — enabled us to build intelligence and capabilities that individual teams couldn’t achieve on their own, especially when banner development was just incidental work for them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQYxCBUO-4FaxhnF0KEwOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGQGZK63W-Dbbe3SKuaYLA.png" /><figcaption><em>Page “Hero” Banners advertising our Insider program for new users vs paused users. We rely on Conductor to pick the appropriate button text.</em></figcaption></figure><p>Closely related is <strong>personalization</strong>. We’ve always recognized the value of tailoring extremely personal experiences for our users. But having one central system made it easier to plug in all the data we have on our retailers and use it to pick the right copy and creative to meet retailers where they are.</p><figure><img alt="xamples of promotional modals with different background images and copy, demonstrating how Conductor personalizes content based on retailer store type." src="https://cdn-images-1.medium.com/max/1024/1*uNzQKdymxGjH-xCaVa_-yQ.png" /><figcaption><em>Another example of Conductor-driven personalization. Our new retailers see one of 16 different modals with different background images and text depending on their store type.</em></figcaption></figure><p>Together, prioritization and personalization transform Conductor from a delivery mechanism into an intelligence layer for in‑product messaging, ensuring that retailers see fewer — but more relevant and engaging — messages.</p><h3>Case study: migrating checkout banners with Conductor</h3><p>Checkout was a prime candidate for Conductor: a critical surface filled with conflicting, hard-coded banners. We migrated these banners to Conductor with the following steps:</p><ul><li><strong>Audit and standardization</strong>: We consolidated 11 banner types into only two standardized components, improving our visual consistency and adherence to Faire’s design principles.</li><li><strong>Triggering logic</strong>: Legacy conditions governing when and where components should be displayed were moved into Conductor campaigns. The triggering events automatically enroll users into the right campaigns when conditions are met to receive messaging.</li><li><strong>Determining priority</strong>: We limited checkout to show at most one banner at a time. Each campaign was assigned a priority which is enforced by Conductor at runtime so the most prominent message is always being shown to retailers.</li><li><strong>Integration and rollout</strong>: We phased out our client-side banners, replacing them with calls by checkout services to Conductor as part of page generation. Once the rollout completed we were able to clean up and greatly simplify the frontend checkout messaging code.</li></ul><figure><img alt="Stack of banner examples from the Orders page with different headlines and calls to action, representing variants selected by Conductor using contextual bandits." src="https://cdn-images-1.medium.com/max/996/1*kmY5BvAVtyiSyFxXiulRwQ.png" /><figcaption><em>The banner variants you might see on the Orders page. These banners are driven by Conductor using Contextual bandits.</em></figcaption></figure><h4><strong>Results</strong></h4><p>Conductor delivered strong results across multiple dimensions:</p><ul><li><strong>Velocity</strong>: Campaigns that previously took days to design, implement, and deploy to production now launch in minutes, saving days of effort and delay. The self-serve model and low-code tooling enabled faster feedback loops during the checkout banner migration, letting engineers and QA adjust configurations without writing new code. It’s also reduced engineering overhead by an estimated 30–40 hours per month.</li><li><strong>Quality</strong>: Checkout banner bugs dropped 43% the quarter after the migration, reducing engineering maintenance effort. The largest class of defects related to incorrect web banner interactions was eliminated altogether.</li><li><strong>User experience:</strong> Users see fewer, more relevant messages. Banners no longer stack or compete unintentionally, and design is no longer made more complex on surfaces that already feature many potential messages.</li><li><strong>Performance</strong>: Conductor API calls are fast (~25ms) and run concurrently with page loads, keeping the user experience snappy and responsive.</li></ul><figure><img alt="Checkout page showing a blocking banner warning about overdue invoices, placed above order and payment details and served through Conductor." src="https://cdn-images-1.medium.com/max/1024/1*96QrwDSh5fcpDuVbYfVTlA.png" /><figcaption><em>A checkout-blocking banner served using Conductor.</em></figcaption></figure><h3>Scaling a platform: adoption lessons</h3><p>Driving Conductor adoption has looked very different at different stages.</p><h4><strong>Early (provisional) stage</strong></h4><p>Adoption was relatively straightforward. We treated Conductor like a product and approached adoption as a search for <a href="https://www.firstround.com/levels">product-market fit</a>. We partnered closely with a small set of sophisticated application teams, designing the platform alongside them and iterating based on candid feedback. These teams were willing to invest in us, and our role was to stay just ahead of their roadmaps — delivering clear value without slowing them down.</p><h4><strong>Operational stage</strong></h4><p>As the platform matured and adoption shifted from <em>Erratic</em> to <em>Extrinsic Push</em> (from the <a href="https://tag-app-delivery.cncf.io/whitepapers/platform-eng-maturity-model/">Platform Engineering Maturity Model</a>), the nature of the problem changed. We were no longer working with a handful of aligned early adopters. Instead, we were asking many teams across the company to change how they built, shipped, and operated messaging. Each team had different incentives, constraints, and success metrics.</p><p>Driving adoption at this stage was messy and required a multi‑pronged approach. We had to rethink both the problem we were solving and the promise we were making, meeting teams where they were while still guiding the organization toward a shared platform. A single mandate or narrative would have been ineffective and, in some cases, counterproductive. Conductor’s strength was that it offered value along multiple dimensions — velocity, design consistency, prioritization, personalization, and long‑term maintainability — which gave us flexibility in how we framed that value. In parallel, we invested in identifying advocates who could speak to Conductor’s impact from different perspectives and act as internal partners, creating pockets of high adoption.</p><p>These advocates weren’t just engineers. Designers were especially helpful in surfacing gaps and unblock adoption early, motivated by a shared design language and cohesive user experience.</p><p>Ultimately, platform adoption isn’t just a technical problem. It’s a socio‑technical one — sitting at the intersection of product strategy, incentives, workflows, and culture — and it requires patience, empathy, and a shared commitment to craft.</p><h3>What’s next</h3><p>Looking ahead, we see Conductor as the execution layer in a larger ecosystem.</p><p>By integrating Conductor with our customer data platform — powered by a knowledge graph — we’re working to unlock deeper personalization and smarter prioritization. This sets the stage for messaging that is increasingly contextual, adaptive, and relevant to each retailer.</p><p>As Conductor continues to lower the cost of launching campaigns, we’re investing in stronger guardrails and cooldowns to ensure retailers aren’t bombarded with messages. In parallel, we’re continuing to improve Conductor’s tooling as adoption expands and we undertake what is effectively a large rewrite of messaging across our site and apps.</p><p>Conductor started as a way to ship faster. Now, it’s becoming a foundation for consistent design, intelligent messaging, and scalable growth across Faire.</p><h3>Acknowledgments</h3><p>Thanks to <a href="https://www.linkedin.com/in/jake-buller-631b8432/">Jake Buller</a>, <a href="https://www.linkedin.com/in/jli0423/">Justin Li</a>, <a href="https://www.linkedin.com/in/praviin-premsankar/">Praviin Premsankar</a>, <a href="https://www.linkedin.com/in/emilywcai/">Emily Cai</a>, <a href="https://www.linkedin.com/in/dingxizheng/">Eric Ding</a>, <a href="https://www.linkedin.com/in/zacharysweigart/">Zachary Sweigart</a>, <a href="https://www.linkedin.com/in/zacharyjradford/">Zachary Radford</a>, <a href="https://www.linkedin.com/in/pedro-almeida-bba747197/">Pedro Almeida</a>, <a href="https://www.linkedin.com/in/gordon-winch/">Gordon Winch</a>, <a href="https://www.linkedin.com/in/birolsenturk/">Birol Senturk</a>, and <a href="https://www.linkedin.com/in/emily-thompson-576634122/">Emily Thompson</a> for their contributions in building Conductor.</p><p>Thanks to <a href="https://www.linkedin.com/in/emilycoleary/">Emily O’Leary</a>, <a href="https://www.linkedin.com/in/sumit-somani-/">Sumit Somani</a>, <a href="https://www.linkedin.com/in/madcido/">Fabio Carmo</a>, <a href="https://www.linkedin.com/in/brandon-dang-899553221/">Brandon Dang</a>, <a href="https://www.linkedin.com/in/yao-w-11772a163/">Yao Wang</a>, and <a href="https://www.linkedin.com/in/degleeson/">Derek Gleeson</a> for their contributions migrating the Checkout page banners onto Conductor and also to our many many client teams for their work and patience.</p><p>Thanks to <a href="https://www.linkedin.com/in/luthfur-chowdhury-27651b6/">Luthfur Chowdhury</a>, <a href="https://www.linkedin.com/in/alexa-weiser-b8532283/">Alexa Weiser</a>, <a href="https://www.linkedin.com/in/jeffhodnett/">Jeff Hodnett</a>, <a href="https://www.linkedin.com/in/jessicakarle/">Jessica Karle</a>, <a href="https://www.linkedin.com/in/amytalus/">Amy Talus</a>, and <a href="https://www.linkedin.com/in/taichi-hoshino/">Taichi Hoshino</a> for their leadership and support throughout the last year of building Conductor and driving adoption.</p><p><strong><em>Interested in the kinds of problems we tackle at Faire?</em></strong><em> We’re hiring engineers and data scientists who want to solve unique marketplace challenges and combine cutting-edge tech with real-world impact.</em> <a href="https://www.faire.com/careers">Join us</a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceb5197972f9" width="1" height="1" alt=""><hr><p><a href="https://craft.faire.com/conductor-faires-platform-for-orchestrating-in-app-messages-ceb5197972f9">Conductor: Faire’s platform for orchestrating in-app messages</a> was originally published in <a href="https://craft.faire.com/">The Craft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling machine learning at Faire: the grouped feature store revolution]]></title>
            <link>https://craft.faire.com/scaling-machine-learning-at-faire-the-grouped-feature-store-revolution-441028eaed06?source=rss-782380e4dcad------2</link>
            <guid isPermaLink="false">https://medium.com/p/441028eaed06</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-and-ml-applications]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-store]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Faire Data and Engineering Team]]></dc:creator>
            <pubDate>Thu, 11 Sep 2025 17:26:21 GMT</pubDate>
            <atom:updated>2025-09-11T17:46:53.282Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we built a modular architecture for faster iteration, lower costs, and greater scale</h4><p>Written by <a href="https://www.linkedin.com/in/eason6wang/">Eason Wang</a> and <a href="https://www.linkedin.com/in/benbates30/">Ben Bates</a></p><figure><img alt="Illustration showing an abstract representation of the grouped feature store." src="https://cdn-images-1.medium.com/max/1024/1*o3PxRndAhs_-t_ofwntphQ.jpeg" /></figure><p>Machine learning is at the heart of how <a href="https://www.faire.com/">Faire</a> empowers small businesses to thrive. It drives personalized recommendations, enhances search relevance, and enables data-driven decision-making for brands and retailers across our marketplace. During our early days, ML growth faced a bottleneck: feature engineering. Teams built custom SQL pipelines and ad hoc scripts, leading to fragmented solutions and minimal reuse. <a href="https://craft.faire.com/real-time-ranking-at-faire-part-2-the-feature-store-3f1013d3fe5d">Our first feature store post</a> shared how we solved this by centralizing the system, scaling us from startup chaos to a standardized platform.</p><p>As Faire’s marketplace expanded, so did the demands on our machine learning systems in both scale and complexity. Our feature store scaled by 300% year-over-year, closely tracking the growth of our business. Each new product surface, recommendation system, and risk model brought distinct feature requirements, driving a rapid increase in both the number and diversity of features we managed.</p><p>Our original architecture wasn’t built for the volume and diversity of features we have today. As the system scaled, performance degraded and infrastructure costs climbed. We needed a fundamental redesign that could scale not just with our business, but with the relentless pace of ML innovation itself. Enter the grouped feature store — a radical rethinking of how we compute, store, and serve features, built from the ground up for the next phase of Faire’s ML journey.</p><h3>The monolithic feature store</h3><figure><img alt="Architecture diagram of the original feature store. Features flow from a centralized registry through batch computation, dispersal, and into a unified serving layer. Only float features and embeddings are supported." src="https://cdn-images-1.medium.com/max/1024/1*BYIhDBDjCkMpSUgrybwvNQ.png" /><figcaption>Our original design computed, stored, and served all features together.</figcaption></figure><p>Our original feature store served us well for several years, and we continuously made incremental improvements. As part of this migration, we began referring to this design as the “monolithic feature store” because it required all features to be computed, dispersed, and served together.</p><p>As ML use cases at Faire have continued to proliferate, we realized that there were fundamental design limitations that would eventually prevent effective scale:</p><ul><li><strong>Slow dispersal: </strong>All features computed/dispersed in one pipeline → SLA breaches as data volume grew. Also, the unnecessary dependencies between irrelevant features blocked parallel processing.</li><li><strong>Wasteful reads: </strong>Since all features were stored together in the online in-memory database, each feature request would fetch all features that existed for a given entity (e.g. brand, retailer, product, etc.). This meant that even if we only needed a single float feature, we were forced to read kilobytes of data, wasting network bandwidth and memory, and adding unnecessary latency.</li><li><strong>Data type restrictions: </strong>The system was designed to only support float and embedding features. Adding support for new data types such as string, string array, and other complex data types within the original architecture would require major code refactoring.</li><li><strong>No storage flexibility: </strong>All features live in online in-memory database, even non-latency-critical ones. This blocked efforts to optimize for cost or performance trade-offs.</li></ul><h3>Grouped feature store</h3><p>Our redesigned system, the grouped feature store, tackles these limitations head-on.</p><figure><img alt="Diagram showing the grouped feature store. The system involves a singular feature registry supporting all data types. This is followed by an arbitrary number of feature computation groups feeding into feature serving groups, each with autogenerated DAGs for ease of use. Serving groups upload features online with a choice of storage patterns, from which clients can fetch features." src="https://cdn-images-1.medium.com/max/1024/1*CADKkX4mIHnImCspO1YbsA.png" /><figcaption>The architectural design of the grouped feature store.</figcaption></figure><p>We named this architecture the grouped feature store because it centers around two new core abstractions: feature compute groups and feature serving groups. In the following sections, we’ll walk through each major component of the system, the problems we set out to solve, and why these changes were essential to evolving our feature store.</p><h4>Unified feature registry</h4><p>The original feature registry was rigid by design, supporting only two feature types: numerical floats and vector embeddings. As our ML applications diversified, this became a bottleneck since teams increasingly needed to work with more complex data types such as string arrays and other specialized formats. Without native support, engineers were forced to implement awkward workarounds, like encoding strings as pseudo-embeddings or storing metadata in side tables. These hacked solutions created development and maintenance headaches and made features harder to discover or reuse.</p><p>The core of the problem lay in our registry architecture. We had built completely separate systems for floats and embeddings — different Python classes for definition, separate Airflow DAGs for computation, and divergent Snowflake schemas for metadata storage. This duplication meant every new feature type would require its own parallel infrastructure. This duplicated approach clearly wouldn’t scale to additional data types.</p><p>Our solution was to design a unified feature registry from the ground up. We created a flexible schema that could represent any feature type through a common interface while preserving each type’s unique characteristics. The new system abstracted away the differences between data types behind standardized APIs for registration, computation, and storage. Under the hood, we implemented shared infrastructure components that all features could use, while still allowing type-specific optimizations where needed. This architecture immediately supported our existing float and embedding features while seamlessly accommodating any new complex data types.</p><p>The unified registry delivered benefits far beyond just supporting more data types. It gave us a single source of truth for all feature metadata, eliminated redundant code, and established consistent patterns for feature lifecycle management.</p><h4>Feature compute groups</h4><p>At the heart of our redesigned system lies the concept of feature compute groups — logical collections of features that are processed together on a shared schedule. Unlike rigid architectural mandates, we designed this system to flexibly accommodate how data scientists naturally organize their work. In practice, we see teams grouping features by:</p><ul><li>Entity type (all product features computed together)</li><li>Use case (features for a new recommendation experiment)</li><li>Data source (<a href="https://craft.faire.com/embedding-based-retrieval-our-journey-and-learnings-around-semantic-search-at-faire-2aa44f969994">deep text embeddings</a> from our NLP pipeline)</li></ul><p>The power of compute groups comes from their simplicity. A data scientist can define a new group with just a few lines of configuration.</p><pre>brand_feature_compute_group = OfflineFeatureComputeGroup(<br>    group=OfflineFeatureComputeGroupName.BRAND_FEATURE_COMPUTE_GROUP,<br>    dag_config=FeatureComputeDagConfig(<br>        schedule_interval=brand_feature_group_schedule_interval,<br>    )<br>)</pre><p>Behind this clean interface, the system handles all the complexity. Each compute group automatically gets its own dedicated Airflow DAG, built from our standardized templates.</p><p>The results speak for themselves. What used to require days of coordination to modify the monolithic computation pipeline now takes minutes. Teams simply define their compute group and get an isolated, production-ready data pipeline. This modular approach has been particularly valuable for:</p><ul><li>Experimentation: New features can be tested without impacting stable ones.</li><li>Performance tuning: Compute-heavy groups get dedicated resources.</li><li>Debugging: Issues are naturally contained within their group.</li></ul><h4>Feature serving groups</h4><p>The second pillar of our redesign introduces feature serving groups — collections of features that share the same storage, dispersal, and serving characteristics. Where compute groups organize how features are calculated, serving groups optimize how they’re delivered. This separation gives teams fine-grained control over their operational tradeoffs. For example, latency-sensitive features can leverage a high performance database like Redis, while others may consider cheaper database options. Also, experimental features can isolate their impact with dedicated infrastructure.</p><pre>monolith_feature_serving_group = OfflineFeatureServingGroup(<br>    serving_group=OfflineFeatureServingGroupName.MONOLITH_FEATURE_SERVING_GROUP,<br>    dag_config=FeatureDispersalDagConfig(<br>        schedule_interval=monolith_feature_serving_group_schedule_interval,<br>    )<br>) </pre><p>The power of serving groups comes from their declarative configuration. When defining a group, engineers specify the storage backend and dispersal schedule, and which compute groups feed into it. Behind the scenes, our system automatically generates a customized Airflow DAG for each serving group that handles all dispersal operations. Crucially, these DAGs intelligently manage dependencies — a serving group configured to pull features from multiple compute groups will automatically wait for all upstream computations to complete before beginning dispersal. This ensures atomic updates where either all features disperse successfully or none do, preventing partial updates that could compromise model performance.</p><p>Our serving group implementation launched with multiple storage options, each designed for specific performance and cost profiles:</p><ol><li>Redis persistent with DynamoDB backup — the pattern used by the monolithic feature store.</li><li>DynamoDB persistent with Redis caching. This makes a tradeoff of higher latency for lower cost.</li><li>Several experimental solutions with other NoSQL databases.</li></ol><p>This approach fundamentally changed how teams interact with the feature store. By abstracting away storage complexities behind a unified interface, we empowered data scientists and ML engineers — the experts who best understand their features’ requirements — to select the optimal storage pattern through simple configuration. A search team needing ultra-low latency can specify Redis persistence, while a risk modeling team might choose the cost-optimized DynamoDB option. The system handles all underlying implementation details automatically.</p><p>The flexibility extends beyond our initial offerings. When evaluating new database technologies, we can now create experimental serving groups that shadow production traffic, comparing performance against existing systems without risky all-or-nothing migrations. This capability has dramatically accelerated our ability to innovate with new storage solutions while maintaining system stability.</p><p>Remarkably, these architectural improvements required no changes to feature consumption patterns. Client services continue using the same simple API, now with the optional ability to specify a serving group when needed. Behind the scenes, the system routes requests to the appropriate storage backend, whether that’s Redis, DynamoDB, or an experimental alternative.</p><pre>val features = featureStore.getFeatures(<br>    identifiers = identifiers,<br>    requestedGroup = MONOLITH_FEATURE_SERVING_GROUP,<br>    requestedFeatures = listOf(<br>        BRAND_FEATURE_1,<br>        BRAND_FEATURE_2,<br>        BRAND_FEATURE_3,<br>    ),<br>)</pre><p>Serving groups are both a technical win and a shift in team philosophy. By rejecting the one-size-fits-all mentality, we’ve created a system that matches infrastructure to actual business needs. Teams now make deliberate tradeoffs between cost and performance rather than working around system constraints. This flexibility has become increasingly valuable as our ML applications continue to diversify, ensuring our feature store can adapt to future requirements without another ground-up redesign.</p><h3>The migration journey</h3><p>Migrating from our monolithic feature store to the grouped architecture was a significant engineering challenge. We deliberately built the new system as a parallel implementation rather than modifying the existing one, allowing us to design without constraints but requiring a carefully orchestrated transition. Our guiding principle was to make the migration invisible to downstream services: every API call would return identical results whether routed to the old or new system while satisfying the same latency requirements. Here’s how we managed the migration.</p><p><strong>Migrating the feature registry</strong></p><p>The migration began with our feature registry, which contained thousands of features defined in code, with each feature definition referencing other files such as SQL files for computation. We developed automated migration scripts that parsed existing feature definitions and transformed them into the new unified schema. It continuously synced new features added during the migration period to ensure complete coverage, with validation checks confirming no feature was lost or corrupted in translation.</p><p><strong>Offline feature store migration</strong></p><p>With the feature registry in sync, the group-specific compute and dispersal DAGs were generated automatically by the DAG template system. We maintained dual-writes to both systems during transition to allow us to conduct comprehensive data validation checks at each stage.</p><p><strong>Online serving</strong></p><p>We designed a unified feature store interface that hides all the complexity of our underlying storage systems. We designed a single, robust get() API that maintains backward compatibility while unlocking the power of our new grouped architecture.</p><p><strong>Validation and rollout</strong></p><p>Achieving seamless migration required exhaustive validation at every layer, including:</p><ul><li><strong>Automated data parity:</strong> We automatically compared billions of feature values and achieved &gt;99.9% consistency. We explored the rare differences, and confirmed that they could all be attributed to computation workloads running at slightly different times.</li><li><strong>Full production shadow testing:</strong> We shadowed all production traffic and confirmed feature retrieval latency and model outputs matched the monolithic system with no meaningful performance differences.</li><li><strong>Phased rollout:</strong> We gradually increased traffic from 5% to 100% over four weeks, closely monitoring feature parity, model outputs, and system health at each step.</li></ul><h3>Results</h3><p>The migration to our grouped feature store architecture has fundamentally transformed how we develop and deploy machine learning features at Faire. By modularizing compute and serving, we’ve achieved breakthroughs in both technical performance and business impact.</p><figure><img alt="Table showing the difference between monolith and grouped feature store systems, specifically in reliability and speed, development flexibility, innovation potential and scalability." src="https://cdn-images-1.medium.com/max/1024/1*vyaenjZ1XxoPf8LVjbXbhg.png" /><figcaption>The differences between our monolith and grouped feature stores.</figcaption></figure><p><strong>Reliability meets velocity</strong></p><p>Our new parallelized pipelines reduced end-to-end feature freshness SLAs by 60% across all critical surfaces. This transformation came while maintaining 100% system availability and reliability. The modular design proved particularly valuable when we pinpointed and optimized a previously hidden bottleneck in our product feature pipeline.</p><p><strong>Empowering innovation</strong></p><p>The decoupled nature of our compute and serving groups has revolutionized how our teams approach feature development. Data scientists now enjoy complete freedom to experiment with novel features — including high-cardinality and computationally intensive ones — without risk of destabilizing production pipelines. This isolation has eliminated the traditional fear of “breaking things” or “slowing things down,” leading to a dramatic increase in experimental feature development and allowing teams to explore more ambitious ideas. Simultaneously, the platform team has capitalized on this same flexibility to safely evaluate next-generation databases in parallel with live production traffic, gathering real-world performance data without compromising system stability!</p><p><strong>Expanding the possibilities</strong></p><p>Our native support for additional data types has opened doors to transformative new capabilities that were previously impractical or impossible to implement. In just the past few months, we’ve seen numerous features leveraging new data types to enable complex recommendation logic for our personalized shopping experience. These advancements have directly contributed to measurable business outcomes, such as increasing GMV.</p><h3>What’s next</h3><p>With the grouped feature store, we’ve built a system that scales in every direction: more features, more teams, more use cases, and more infrastructure choices. It’s an adaptive foundation for ML at Faire — powering everything from visual search to next-best-action recommendations — with the reliability our business depends on. And as new technologies like generative AI emerge, this architecture ensures we’re ready to integrate them seamlessly and continue shipping the features that matter.</p><p>This work could not have been done without the contributions and input of various people including <a href="https://www.linkedin.com/in/namitkatariya/">Namit Katariya</a>, <a href="https://www.linkedin.com/in/olgeorge/">Georgii Oleinikov</a>, <a href="https://www.linkedin.com/in/hagarwa3/">Harshit Agarwal</a>, <a href="https://www.linkedin.com/in/edyakabosky/">Ed Yakabosky</a>, <a href="https://www.linkedin.com/in/yanwei-wayne-zhang-314597ab/">Wayne Zhang</a>, <a href="https://www.linkedin.com/in/jgulbronson/">Jeff Gulbronson</a>, and more.</p><p><em>Interested in joining a team working on innovative projects that serve small businesses around the world? Apply for open roles: </em><a href="https://www.faire.com/careers"><em>https://www.faire.com/careers</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=441028eaed06" width="1" height="1" alt=""><hr><p><a href="https://craft.faire.com/scaling-machine-learning-at-faire-the-grouped-feature-store-revolution-441028eaed06">Scaling machine learning at Faire: the grouped feature store revolution</a> was originally published in <a href="https://craft.faire.com/">The Craft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test]]></title>
            <link>https://medium.com/@faire_data_and_eng_team/test-385ddf22f0c6?source=rss-782380e4dcad------2</link>
            <guid isPermaLink="false">https://medium.com/p/385ddf22f0c6</guid>
            <dc:creator><![CDATA[Faire Data and Engineering Team]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 22:32:52 GMT</pubDate>
            <atom:updated>2025-09-09T22:32:52.022Z</atom:updated>
            <content:encoded><![CDATA[<p>Test</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=385ddf22f0c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advancing product categorization with vision language models: The power of fine-tuned LLaVA]]></title>
            <link>https://craft.faire.com/advancing-product-categorization-with-vision-language-models-the-power-of-fine-tuned-llava-2f4bf024a102?source=rss-782380e4dcad------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f4bf024a102</guid>
            <category><![CDATA[vision-language-model]]></category>
            <category><![CDATA[llava]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[multimodal]]></category>
            <dc:creator><![CDATA[Faire Data and Engineering Team]]></dc:creator>
            <pubDate>Wed, 17 Jul 2024 16:39:17 GMT</pubDate>
            <atom:updated>2024-07-17T16:39:17.529Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Title image: Advancing product categorization with vision language models" src="https://cdn-images-1.medium.com/max/1024/1*WfusNZYu8SPF5DzbxQxaGg.png" /></figure><p>Written by <a href="https://www.linkedin.com/in/yanwei-wayne-zhang-314597ab/">Wayne Zhang</a></p><p>As a global wholesale marketplace, <a href="https://www.faire.com/">Faire</a> connects hundreds of thousands of independent brands and retailers all over the world. In order for retailers to identify and find the products they’re looking for, accurate product type categorization is crucial. Yet this process is more complex than it might seem. When brands list their products on Faire, they must include various product information such as titles and images. Additionally, they need to specify the product type (e.g., kitchen tool or T-shirt) from approximately 3,000 predefined options.</p><p>In this blog post, we discuss an innovative approach to product categorization using a fine-tuned vision language model (VLM) based on <a href="https://github.com/haotian-liu/LLaVA">LLaVA</a>. This approach significantly improves the accuracy of product categorization, enables a streamlined user experience, and enhances operational efficiency through high levels of automation.</p><h4>Key results:</h4><ul><li>By leveraging both text and image data, the fine-tuned vision language model can categorize product types with an accuracy of 95%.</li><li>Images play a crucial role in understanding intricate and nuanced characteristics for product categorization and provide consistent visual cues in a multilingual context.</li><li>High accuracy in categorization reduces user onboarding time by approximately 20%, saves hundreds of thousands of dollars on manual reviews, and makes product discovery more accurate.</li><li>This approach is generalized to an instruction-following framework to handle many product-related tasks, such as recommending product titles and descriptions, and extracting product attributes like color, materials, weights, and dimensions.</li></ul><h3>The challenge of product categorization</h3><p>Choosing from approximately 3,000 product type options is not easy. Brands spend a significant amount of time selecting the best-fitting product type, which slows down the onboarding process. Internal staff assisting with brand onboarding also spend around 15% of their time on product categorization, leading to substantial operational costs.</p><figure><img alt="Screenshot of product type categories for a product name. Product types include 2-in-1 Hair &amp; Body Moisturizer, 2-in-1 Shampoo &amp; Body Wash, 2-in-1 Shampoo &amp; Body Wash — Baby, 2-in-1 Shampoo &amp; Body Wash — Mens, and more." src="https://cdn-images-1.medium.com/max/839/0*7wISwtJ63blor7ts" /><figcaption><em>Brands select product type out of ~3000 options when listing a product.</em></figcaption></figure><p>To accelerate this process, the Faire Data team previously developed a machine learning model based on KNN and <a href="https://arxiv.org/abs/2103.00020">CLIP embeddings</a> to recommend the most applicable product types based on product information. However, human audits revealed that over 15% of existing products were still mis-categorized. Our operations team had to conduct regular and large-scale manual reviews and corrections of existing products. While this effort significantly improved the accuracy of product categorization, it became a costly operation.</p><p>Several factors complicate product categorization.</p><ul><li>Certain product types are naturally hard to categorize, especially in apparel where different styles can blur category lines and distinctions are often subtle. For example, the product shown in image (a) below was tagged as a “Tunic” by human reviewers but should be a “Blouse” based on specific definitions:</li></ul><pre><strong>Tunic</strong>: Falls below the hips, regardless of sleeve length.<br><strong>Blouse</strong>: Made of lightweight fabric, features a loose/fitted design with a <br>        front opening and closures, more formal than a T-shirt, often worn <br>        to work or go out.</pre><ul><li>Inconsistent information from different product fields further complicates predictions. Product titles, descriptions, and images can contain critical information related to the product type but sometimes indicate different categories. For example, the product in image (b) below is described as a “Tank Top” but has an image that better fits the “Knit Top” category:</li></ul><pre><strong>Tank top</strong>: Sleeveless, casual; often cotton or lightweight fabric.<br><strong>Knit top</strong>: Made from a stretchy, knit fabric, typically with a simple, <br>          comfortable design; short or long sleeves, crew or V-necklines, <br>          and fitted or loose silhouettes; comfortable and versatile for <br>          everyday wear.  </pre><ul><li>Moreover, the presence of non-English products adds another layer of complexity. Products described in languages other than English, such as French, require robust multilingual understanding to categorize correctly, like the jewelry in image (c) below.</li></ul><figure><img alt="Examples of challenging product categorization, showing three products: a pink blouse, a black knit top, and bracelets." src="https://cdn-images-1.medium.com/max/1024/0*Ny-nSfgo7qqivSC5" /><figcaption><em>Product categorization is challenging. (a) This is a Blouse but was mis-categorized as a Tunic. (b) This is a Knit Top but mis-categorized as Tank Top because of the product description. (c) This is a Bangle Bracelet described in French.</em></figcaption></figure><h3>Why we categorize products using vision language models</h3><p>To tackle this product type categorization problem, we launched a vision language model fine-tuned on Faire’s own data. The vision language model fits well in this application for several reasons.</p><p>Unlike regular language models that rely solely on text data, or image-only models that depend exclusively on visual input, a vision language model leverages both text and image signals. This dual-modality approach ensures a comprehensive analysis and holistic understanding of the product, capturing nuances that might be overlooked when using single-modality models. For instance, a product’s title and description might provide critical context about its intended use or style, while the image can reveal design details.</p><p>One notable advantage of incorporating image data is the reduced reliance on multilingual understanding. Images are universally interpretable and can provide consistent visual cues about the product, such as the type of object and the style. This makes it easier for the model to classify non-English products accurately without needing extensive multilingual support, thus broadening the applicability and robustness of the system.</p><p>Furthermore, the use of a large vision language model trained on extensive datasets enables the handling of subtle patterns and inconsistencies in product information. As demonstrated in the examples above, product categories can often be defined by intricate and nuanced characteristics that are challenging to capture, and products could have inconsistent information across different fields, with titles, descriptions, and images sometimes providing conflicting signals. The large model structure from the vision language model, trained on diverse and large datasets, can learn these subtle distinctions effectively and synthesize and reconcile these discrepancies to make more accurate and reliable predictions.</p><h3>The LLaVA model</h3><p>To achieve better product categorization, the Faire Data team specifically fine-tuned the <a href="https://llava-vl.github.io/">LLaVA</a> model, an open-source vision language model with state-of-the-art performance in various benchmarks. The model was trained on millions of products that our operations team has manually audited in the past.</p><h4>Our fine-tuning setup</h4><p>The model was set up to generate the most likely product type based on the product title, product description, brand name, brand description, and the hero product image. Each field is optional, and the model can still function accurately when only text or image data is available. Below is an example of our fine-tuning setup:</p><figure><img alt="Image of LLaVA fine-tuning setup showing model input and output for a sports bra." src="https://cdn-images-1.medium.com/max/962/1*XupFzLcuaBNBCVt6E7yfEw.png" /><figcaption><em>LLaVA fine-tuning setup. Text prompt is constructed using product name, description, brand name and brand description. Both text prompt and image are used to generate the product type for a product.</em></figcaption></figure><h4>Comparing data and model compositions</h4><p>To assess the impact of data and model composition on model performance, we trained an array of models with different datasets and configurations.</p><ul><li><strong>Datasets:</strong> We compared models trained on unaudited data with those trained on manually-audited data. The unaudited data came from existing products, allowing us to use a large volume from the existing database. However, as previously discussed, about 15% of this data is mis-categorized, introducing noise into model training.</li><li><strong>Models:</strong> We tested both the LLaVA-7b and LLaVA-13b models.</li></ul><p>In each training instance, we split the data into training and evaluation sets based on predefined brand-level splits. This approach prevents intra-brand leakage into evaluations, as products from the same brand might share characteristics indicative of taxonomy prediction. Additionally, we randomly masked one of the product fields with a 5% probability to account for scenarios where some fields may not be populated at product creation time. This technique enhances the model’s robustness and improves its performance when only text or image data is available.</p><h4>Insights from fine-tuning LLaVa</h4><p>The table below summarizes the results of the different models. Key points include:</p><ul><li>All LLaVA-based models significantly outperformed the existing production model based on KNN and CLIP embeddings.</li><li>Models trained on audited data outperformed those trained on unaudited data, underscoring the importance of data labeling for establishing high-quality ground truth data.</li><li>Increasing the number of training samples and the model size both contributed to improved model performance. The best model, based on LLaVA-13b, achieved 95% accuracy.</li></ul><p>By leveraging the capabilities of fine-tuned LLaVA models, we have achieved a remarkable improvement in product categorization, paving the way for a more efficient product onboarding flow.</p><figure><img alt="Chart comparing performance of fine-tuned LLaVA models." src="https://cdn-images-1.medium.com/max/817/0*JwRiPC5kLLcnw3WO" /><figcaption><em>Performance of fine-tuned LLaVA models. Best performing model achieved 95% accuracy. In contrast, the existing production model has about 70% accuracy.</em></figcaption></figure><h4>Model serving</h4><p>We deployed the final model on SageMaker using <a href="https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/what-is-dlc.html">AWS Deep Learning Containers</a>, which created a horizontally scalable HTTP endpoint that clients can invoke. This deployment process aligns with our standard model deployment methods on this platform. For example, we implement the model_fn to load the model using a custom LlavaModelWrapper, and predict_fn to load the text prompt, image URLs, and generation configuration to generate the output. The wrapper implements generation using <a href="https://docs.vllm.ai/en/latest/getting_started/examples/llava_example.html">vLLM</a> which is about 3x faster than the native implementation.</p><pre>def model_fn(model_dir, context):<br>   &quot;&quot;&quot;Load model from HuggingFace directly&quot;&quot;&quot;<br>   import json, os   <br>   model_path = os.getenv(&#39;MODEL_PATH&#39;, model_dir)<br>   loading_config = json.loads(os.getenv(&#39;LOADING_CONFIG&#39;, &quot;{}&quot;))<br>   llava_model = LlavaModelWrapper.from_pretrained(<br>                    model_path, **loading_config)<br>   return llava_model<br><br><br>def predict_fn(data, model):<br>   &quot;&quot;&quot;Predict using the model&quot;&quot;&quot;<br>   image_urls = data.get(&quot;image_urls&quot;, [])<br>   prompt = data[&quot;prompt&quot;]<br>   config = data.get(&quot;config&quot;, {})  <br>   return model.generate(prompt, image_urls, **config)</pre><h4>Fallback for generating valid product types</h4><p>Since the model is generative, there is no guarantee that it will always generate a valid product type. We observed that about 1% of the generations produced a product type not included in the existing options. To address this, we implemented fallback logic until a valid product type is generated:</p><ul><li>Run generation by sending other images from the product.</li><li>Run generation with only the image.</li><li>Run generation with only the text.</li><li>Run generation with multiple outputs using beam search.</li></ul><h3>Product implications for using LLaVA for product categorization</h3><p>We have launched our LLaVA model in selected surfaces for product creation and are in the process of rolling it out to all product creation flows. The model has largely maintained its high accuracy in production. So far, our experience has been very positive, with significant downstream applications, including:</p><ul><li>With highly accurate automated product categorization, we have streamlined and enhanced the product creation process, improving onboarding time by approximately 20%.</li><li>This has reduced the need for correcting product types through manual review by the operations team, significantly lowering operational costs.</li><li>More accurate product types have enhanced product discovery, unblocked new product launches that depend on high-quality product type data, and indirectly improved machine learning models that rely on product type data.</li></ul><h3>Next: product understanding beyond product categorization</h3><p>In this story, we discussed the problem of product categorization using vision language models. In order to extend the impact of this approach, we have built an initial multi-task instruction-following LLaVA model that has shown promising results in various product understanding tasks.</p><p>Specifically, with this approach, we formulate the product categorization task as masking the product type field of the product and predicting it using the other unmasked product fields. A product can have many additional fields, including:</p><ul><li>Product attributes: e.g., color, material, size, weight, dimension, etc.</li><li>Product alt text caption</li><li>Translated product title and description</li><li>Product image quality: e.g., whether padding was added, whether there are multiple objects, etc.</li></ul><p>We can mask any field and predict it using the other available fields. With this formulation, we are able to extend the model to a multi-task setting capable of handling many critical tasks related to product understanding, such as:</p><ul><li>Recommending a product title that better aligns with Faire’s product listing requirements</li><li>Recommending a product description when it is missing</li><li>Extracting key product attributes</li><li>Generating alt text for SEO</li><li>Translating product title and description for a new locale</li><li>Identifying image qualities in product listings</li><li>Tagging whether a product is relevant to a given search query</li></ul><p>We are excited to continue this work and will share our findings in a future update.</p><p>This work could not have been done without the contribution and input of various people including <a href="https://www.linkedin.com/in/weizhouml/">Wei Zhou</a>, <a href="https://www.linkedin.com/in/jack-bubes/">Jack Bubes</a>, <a href="https://www.linkedin.com/in/erin-wall-hunter-a60b48105/">Erin Hunter</a>, <a href="https://www.linkedin.com/in/clairecolberg/">Claire Mai Colberg</a>, <a href="https://www.linkedin.com/in/max-barltrop/">Max Barltrop</a>, <a href="https://www.linkedin.com/in/hongyi-yang-b941654b/">Hong Yang</a>, <a href="https://www.linkedin.com/in/btara-truhandarien/">Btara Truhandarien</a> and more!</p><p><em>Interested in joining a team exploring possible applications of vision language models like LLaVA? </em><a href="https://www.faire.com/careers"><em>Apply for open roles</em></a><em> with the Faire Data team.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f4bf024a102" width="1" height="1" alt=""><hr><p><a href="https://craft.faire.com/advancing-product-categorization-with-vision-language-models-the-power-of-fine-tuned-llava-2f4bf024a102">Advancing product categorization with vision language models: The power of fine-tuned LLaVA</a> was originally published in <a href="https://craft.faire.com/">The Craft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-tuning Llama3 to measure semantic relevance in search]]></title>
            <link>https://craft.faire.com/fine-tuning-llama3-to-measure-semantic-relevance-in-search-86a7b13c24ea?source=rss-782380e4dcad------2</link>
            <guid isPermaLink="false">https://medium.com/p/86a7b13c24ea</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[llama-3]]></category>
            <dc:creator><![CDATA[Faire Data and Engineering Team]]></dc:creator>
            <pubDate>Mon, 29 Apr 2024 16:01:37 GMT</pubDate>
            <atom:updated>2024-05-14T18:13:19.044Z</atom:updated>
            <content:encoded><![CDATA[<p>Written by: <a href="https://www.linkedin.com/in/qqhsu">Quentin Hsu</a>, <a href="https://www.linkedin.com/in/yanwei-wayne-zhang-314597ab">Wayne Zhang</a></p><figure><img alt="Blog title image: Fine-tuning Llama3 to measure semantic relevance in search" src="https://cdn-images-1.medium.com/max/1024/1*4Odo-p2bhQ8xGX6bi7TaTQ.png" /></figure><h3>Introduction</h3><p>As a global wholesale marketplace, <a href="https://www.faire.com/">Faire</a> connects hundreds of thousands of independent brands and retailers all over the world. Within such a large marketplace, search plays a primary role in helping retailers discover and purchase products for their stores. However, search results that return products that are irrelevant to the query can be more than just an annoyance that makes it harder for retailers to find what they’re looking for — it can reduce trust in Faire’s capabilities to connect retailers with relevant brands.</p><figure><img alt="Search results image. Search for “pita chips” showing tortilla chips and kettle chips in the search results." src="https://cdn-images-1.medium.com/max/1024/0*inUZeGFrG5P0aPCN" /><figcaption><em>We see other types of chips that are not “pita chips.”</em></figcaption></figure><p>As such, improving the relevance of our search algorithm at Faire is important and ongoing work to ensure we live up to our mission of <a href="https://news.faire.com/2021/11/04/what-defines-faire/">helping our customers succeed</a>. Evaluating the relevance of search results has historically been a manual process that is hard to scale, but recent AI advancements and increased availability of Large Language Models (LLMs) allow us to leverage their strong natural language understanding capabilities to scale out the labeling of semantic search relevance.</p><p>In this blog post, we’ll describe our journey in defining semantic relevance, building visibility of relevance in our search ecosystem, and making relevance a critical and actionable dimension to consider in search.</p><p><em>Note: Unless otherwise specified all metric improvements are in relative terms.</em></p><p><strong>Our key learnings include:</strong></p><ul><li>Fine-tuned open-source LLMs demonstrate strong performance. In our case, the best fine-tuned <a href="https://llama.meta.com/llama3/">Llama3–8b</a> model improves search relevance prediction accuracy by 28% compared to the existing fine-tuned <a href="https://en.wikipedia.org/wiki/Generative_pre-trained_transformer">GPT</a> model from a leading LLM provider.</li><li>We developed in-house solutions to fine-tune and serve LLMs. This approach considerably lowers training and inference cost compared to external LLM providers. Effective strategies such as quantization, batching, employing a rapid serving framework like <a href="https://github.com/microsoft/DeepSpeed">DeepSpeed</a>, and horizontal scaling are crucial for achieving high-throughput inference capable of handling tens of millions of predictions daily.</li><li>Prompt engineering alone cannot fully capture semantic search relevance on Faire. LLM fine-tuning is needed for models to understand our definition of relevance.</li><li>Successful application of a fine-tuned LLM still requires clear problem definition, high-quality labeled data, and model iterations similar to a standard machine learning product integration. In our experimentation, the volume, quality, and composition of the labeled data play the most significant role in enhancing model performance.</li><li>In our application, the fine-tuned Llama3–8b model consistently outperforms the fine-tuned Llama2–7b model, with gains ranging from 1.4% to 8%. Additionally, the fine-tuned Llama3–8b model achieves performance on par with the larger Llama2–13b model.</li></ul><h3>Defining semantic relevance</h3><h4>ESCI framework</h4><p>We need a clear definition for relevance that can be objectively agreed upon amongst different people before we even begin trying to model the problem. We adopt the ESCI breakdown of relevance used in the <a href="https://www.aicrowd.com/challenges/esci-challenge-for-improving-product-search">Amazon KDD Cup 2022</a> competition for improving product search.</p><p>From a high level, we break down relevance into four tiers (quoted from the competition):</p><ol><li><strong>Exact</strong> <strong>(E)</strong>: the item is relevant for the query, and satisfies all the query specifications (e.g., water bottle matching all attributes of a query “plastic water bottle 24oz”, such as material and size)</li><li><strong>Substitute</strong> <strong>(S)</strong>: the item is somewhat relevant: it fails to fulfill some aspects of the query but the item can be used as a functional substitute (e.g., fleece for a “sweater” query)</li><li><strong>Complement</strong> <strong>(C)</strong>: the item does not fulfill the query, but could be used in combination with an exact item (e.g., track pants for a “running shoe” query)</li><li><strong>Irrelevant</strong> <strong>(I)</strong>: the item is irrelevant, or it fails to fulfill a central aspect of the query (e.g., pillow for a “kimono” query)</li></ol><figure><img alt="Example query and product pairs for each category of ESCI. For Exact, sticky notes are shown for the query “sticky notes”. For Substitute, flip flops are shown for “sandals”. For Complement, cheese knives are shown for “cheese board”. For Irrelevant, a sunflower pillow is shown for “sunflower kimono”." src="https://cdn-images-1.medium.com/max/920/0*gFMBZ5JX6qiqM0IJ" /><figcaption><em>Examples for each category of ESCI from Faire search results.</em></figcaption></figure><p>Having multiple tiers of relevance provides flexibility in using the relevance labels based on downstream application needs. For applications like search engine optimization, where accurate product matching is critical, only exact matches can be chosen to ensure high precision. In contrast, applications in retrieval and ranking could focus on removing only irrelevant matches to prioritize broader recall to help users discover related products.</p><h3>Measuring for semantic relevance is challenging</h3><p>Measuring and modeling for semantic relevance is challenging due to the inherent subjectivity of the problem and the lack of ground truth for relevance from user behavior.</p><h4>Search queries are often vague and open to interpretation</h4><p>Search queries are often short and vague, leading to many queries having multiple interpretations that could vary depending on context and retailer. The query “bat” could be searching for products in the shape of the mammal or it could be looking for baseball bats. The likeliness of whether a retailer is searching for one or the other could depend on whether the retailer is a sticker shop vs a sports shop.</p><p>We need to develop guidelines to align on these types of edge cases. Models that we use also need to be able to pick up our definitions of what is relevant.</p><h4>Relevance lacks ground truth from user behavior</h4><p>Historical engagements such as clicks, carts, and orders are sparse and it’s difficult to infer what a lack of engagement means since many things are never seen or never engaged with. Engagement also tends to vary depending on search intent, and sometimes occur on irrelevant products that are highly desirable (e.g., high quality product) or personalized to the customer’s interests outside of the current search session. The final purchasing decision is influenced by a blend of semantic relevance, desirability, and personalization.</p><p>Being able to model for relevance would allow us to more comprehensively explore query-product pairs that have never been impressed or engaged.</p><h3>Our past approaches to measuring semantic relevance</h3><h4>Human labeling: a monthly relevance snapshot</h4><p>When we first started measuring relevance at Faire, we worked with a data annotation vendor to label a sample of query-product pairs every month for measuring the relevance of our search system. This gave us both an initial view into the value of having visibility into relevance and allowed us to iterate on the guidelines/definitions for relevance especially for edge cases. We built out several iterations of decision trees to reach &gt; 90% agreement amongst labelers and our quality audits. These labels served as ground truth for our definition of relevance.</p><figure><img alt="Partial view of the flowchart for Faire’s relevance labeling guidelines." src="https://cdn-images-1.medium.com/max/1024/0*z8vTK74rdRAjQikc" /><figcaption><em>Snippet of a version of our labeling guidelines for relevance.</em></figcaption></figure><p>Although human labeling was essential at the start, it was expensive and not immediately actionable due to the one month delay between our measurement and available labels. To make our relevance measurements more actionable, we needed to reduce the cost and lag time of our measurements. LLMs have a strong pretrained understanding of natural language and are particularly suitable in learning from our limited amount of labels to robustly predict for relevance.</p><h4>Scaling out with GPT: a daily relevance snapshot</h4><p>We framed the multi-class classification as a text completion problem and fine-tuned a leading large GPT model to agree with the labelers. In the prompt, we concatenated the search query text with product information (including the name and description of the product), the name of the brand offering the product, and the categorical type of product. The model was tasked to complete the text with one of the four ESCI labels to measure the relevance of the search query and the product.</p><figure><img alt="Sample prompt setup for query “fairy lights for tent” and product “Live Moss Terrarium Supply Kit” with product description, brand, and taxonomy type. Output shows that the completion for this prompt should be the word “Irrelevant”." src="https://cdn-images-1.medium.com/max/1024/0*5LObAKVj2F0ZDmrx" /><figcaption><em>Framing query-product pair relevance evaluation as a text completion problem.</em></figcaption></figure><p>We measured performance with <a href="https://en.wikipedia.org/wiki/Krippendorff%27s_alpha">Krippendorff’s Alpha</a>, which intuitively is the agreement between our fine-tuned model and the labels for all four relevance categories. The score typically ranged from 0 to 1 where 0 is random, and 1 is complete agreement. Negative scores can exist when the “disagreements are systematic and exceed what can be expected by chance.” <a href="https://www.statisticshowto.com/krippendorffs-alpha/#:~:text=Values%20range%20from%200%20to,are%20still%20acceptable%2C%20%CE%B1%20%E2%89%A5%20.">Krippendorff suggests</a>: “[I]t is customary to require α ≥ .800. Where tentative conclusions are still acceptable, α ≥ .667 is the lowest conceivable limit.”</p><p>This is a challenging metric since it penalizes all category mistakes equally even if some mistakes are not as harmful as others (e.g., predicting “substitute” when it was labeled as “exact” due to user preferences). We also referenced other metrics like <a href="https://en.wikipedia.org/wiki/F-score">F1 Score</a> for exact vs not exact to gauge whether our models were applicable for measurement.</p><p>We reached 0.56 Krippendorff’s Alpha and could label ~300k query product pairs per hour at the time we deployed this to production. This allowed us to start measuring relevance on a daily basis from limited samples.</p><p>Our fine-tuned GPT solution made our relevance problem more actionable, but the amount of labels we could obtain was still primarily limited by the costs of using the API offered by the LLM provider. As our search system evolved, our use of the relevance labels for analysis and training increased. In particular, we had begun introducing personalized retrieval sources, which increased the variation of query-product pairs being shown to different retailers and made our existing measurement solutions blind to the aggregate effect of personalization on the search ecosystem.</p><h3>Increasing performance and reducing costs with Llama</h3><p>We needed a way to increase the throughput of relevance labeling at a reduced cost and at equal or better performance. We turned to open-source LLMs that have state-of-the-art performances in various benchmarks. These models often come in different sizes, which makes it possible to choose smaller models tailored for specific tasks, potentially yielding improvements in performance, cost, and operational efficiency. Our tests have focused on <a href="https://llama.meta.com/">Meta’s Llama</a> family of models because of its performance and licensing for commercial use.</p><h4>Technical approach for fine-tuning Llama</h4><p>A key hypothesis we had was that despite the nuances of search relevance understanding, it is a specific language understanding problem that may not require a huge model with hundreds of billions of parameters.</p><p>Below we highlight some key considerations in our model setup and training:</p><ul><li>Our fine-tuning has been centered around the smaller base models Llama2–7b, Llama2–13b, and Llama3–8b. One benefit of working with these models is that they fit into the memory of an A100 GPU card and are fast to prototype and iterate on.</li><li>We froze the weights from the underlying model and employed <a href="https://huggingface.co/blog/peft">Parameter Efficient Fine-Tuning</a> using the <a href="https://arxiv.org/abs/2106.09685">LoRA adapter</a>. This significantly reduced the amount of trainable parameters, resulting in lower memory usage and far faster training speed. In the fine-tuning, the training only optimized about 4% of the base model’s parameters.</li><li>We batched training samples, padded short sequences with the <em>&lt;eos&gt;</em> token, and only computed the cross-entropy loss for tokens in the completion text.</li><li>The model was trained using <a href="https://www.deepspeed.ai/getting-started/">DeepSpeed</a> on 8 A100 GPUs with data and tensor <a href="https://huggingface.co/docs/transformers/v4.17.0/en/parallelism">parallelization</a>.</li><li>To further reduce GPU memory consumption, we used <a href="https://huggingface.co/docs/transformers/v4.18.0/en/performance#gradient-checkpointing">gradient checkpointing</a> which recomputed some nodes. We found this speed and memory tradeoff made the training more stable and less susceptible to OOM issues.</li></ul><p>We trained a multitude of models with different model types, training data sizes, hyper-parameters to achieve better performance. Training leveraged existing GPUs procured for general deep learning development. As a result, iteration of all these parameters is also much quicker and does not come at incremental fine-tuning costs.</p><p>Specifically, we tested Llama2–7b, Llama2–13b, and Llama3–8b. We also tested three datasets with different sizes: Small (11k), Medium (50k) and Large (250k). Our existing production model was a large GPT model fine-tuned on the Small dataset. New fine-tuned open-source models are trained on the Medium and Large datasets for two epochs, and all models are evaluated on a hold-out dataset of ~5k records for comparison. The largest model Llama2–13b took about five hours to complete training on the Large dataset.</p><h4>Performance with Llama3</h4><p>In short, our best performing model, Llama3–8b trained on the Large dataset, is able to achieve 28% improvement on Krippendorff’s Alpha from our existing production model. Details of the model results are reported in Table 1. Based on these metrics, we note the following:</p><ul><li>We briefly explored prompt engineering and found that basic prompt engineering with zero-shot prediction was not performant in predicting our definition of semantic search relevance. The fine-tuned GPT model has almost 2x accuracy compared to prompt engineering.</li><li>Increasing the size and composition of the labeled dataset is the most important factor in improving the performance. Models fine-tuned on the Large dataset outperform their counterparts on the Small or Medium dataset.</li><li>The performance differences between different base models decreased as we trained with more ground truth data. For example, on the same medium sized dataset, the fine-tuned GPT and Llama2–7b models reach performance parity, while the Llama3–8b model improves the performance by ~8%. Llama3–8b has similar performance to Llama2–13b. Llama3–8b trained on the Large dataset achieved the best performance, although the difference with Llama2–7b is reduced to about 1.4%.</li></ul><figure><img alt="Table of performance scores including GPT-based and Llama models." src="https://cdn-images-1.medium.com/max/788/0*9S8rpeZp9SYlOVGY" /><figcaption><em>Table 1 — Performance scores of different models on relevance classification.</em></figcaption></figure><h4>Self-hosted inference</h4><p>The selected Llama3–8b model is hosted on our GPU cluster to create batch predictions of relevance for new search sessions. Our initial application is to leverage the model predicted relevance to measure the performance of our search algorithms on all retailer search sessions. This requires high throughput to score tens of millions of product and query pairs daily. Thus, we set up inference to maximize throughput in the following way:</p><ul><li>Quantize the model to 8 bit</li><li>Run batches on a single A100 GPU</li><li>Improve inference speed using <a href="https://github.com/microsoft/DeepSpeed">DeepSpeed</a></li><li>Horizontally scale the number of GPU instances</li></ul><p>By doing so, we were able to reach 70 million predictions per day using 16 GPUs when we backfilled our labels at scale.</p><h3>Conclusion and future ideas</h3><p>In summary, successful application of fine-tuned LLMs still required clear problem definition, high-quality labeled data, and model iterations similar to a standard machine learning product integration. We defined semantic relevance and developed guidelines to achieve high agreement when labeling for relevance. We then leveraged LLMs to greatly scale out predictions, reducing the lag time of obtaining relevance labels, and making relevance a far more measurable and actionable dimension in our search system. These scaled out predictions unlocked many downstream use cases including offline retrieval analysis, measurement of personalization, measurable contribution of experiments towards relevance, Pareto frontier exploration between engagement and relevance in the ranker, and more.</p><p>The current use of relevance is mainly offline. Deploying the model in a real-time setting or distilling information with smaller models will provide opportunities to change the relevance of search results shown to users. This will require a low-cost and low-latency inference solution.</p><p>Going forward, we plan on exploring areas where our fine-tuned LLMs have room to improve, such as missing domain context (e.g., understanding of the product lines or style of a brand) and the limitations with text only context information (e.g., no images). We can explore retrieval-augmented generation (<a href="https://arxiv.org/pdf/2005.11401.pdf">RAG</a>) techniques to improve missing domain context and are in the process of exploring multimodal LLMs like <a href="https://llava-vl.github.io/">LLaVA</a> to help parse out the rich image information. As part of the exploration, we also want to use LLMs to explain why they labeled something as relevant or irrelevant. Having a more granular reasoning for relevance can help us understand difficult search cases and potentially use <a href="https://arxiv.org/abs/2201.11903">chain of thought</a> to help improve performance.</p><p>Stay tuned for more updates about how we’re improving search at Faire!</p><p><em>See past articles on </em><a href="https://craft.faire.com/building-faires-new-marketplace-ranking-infrastructure-a53bf938aba0"><em>ranking</em></a><em>, </em><a href="https://craft.faire.com/real-time-ranking-at-faire-part-2-the-feature-store-3f1013d3fe5d"><em>feature store</em></a><em>, and </em><a href="https://craft.faire.com/how-we-use-engagement-based-embeddings-to-improve-search-and-recommendation-on-faire-912277de4e6d"><em>embeddings</em></a><em> to learn more about how search works at Faire.</em></p><h3>Shoutouts</h3><p>This work could not have been done without the contribution and input of various people on the Search Algo and Machine Learning Platform teams — including <a href="https://www.linkedin.com/in/hagarwa3/">Harshit Agarwal</a>, <a href="https://www.linkedin.com/in/sam-kenny/">Sam Kenny</a>, <a href="https://www.linkedin.com/in/xiaomeng-hu-381a78170/?originalSubdomain=ca">Xiaomeng Hu</a>, <a href="https://www.linkedin.com/in/minhtripham">Minh Pham</a>, <a href="https://www.linkedin.com/in/tdugan3">Tom Dugan</a>, <a href="https://www.linkedin.com/in/owenwenhao/">Wenhao Liu</a> and more!</p><p>We appreciate <a href="http://fireworks.ai/">Fireworks.ai</a> for their valuable collaboration and initial exploration in the early phase of this project.</p><p><em>Want to join a team working on complex challenges like this? We’re hiring multiple positions on the Data team to build ML/AI solutions for Faire. Apply for </em><a href="https://www.faire.com/careers?utm_source=blog&amp;utm_medium=blog&amp;utm_campaign=craft&amp;utm_content=article_01_17-23"><em>open roles</em></a><em>.</em></p><p><em>View all articles from the Faire Data Science team </em><a href="https://craft.faire.com/tagged/data-science"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86a7b13c24ea" width="1" height="1" alt=""><hr><p><a href="https://craft.faire.com/fine-tuning-llama3-to-measure-semantic-relevance-in-search-86a7b13c24ea">Fine-tuning Llama3 to measure semantic relevance in search</a> was originally published in <a href="https://craft.faire.com/">The Craft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>