<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by yotam feldman on Medium]]></title>
        <description><![CDATA[Stories by yotam feldman on Medium]]></description>
        <link>https://medium.com/@yotam56?source=rss-5eaaba1aaa97------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*YWc2KbvRxUlFz1Aa</url>
            <title>Stories by yotam feldman on Medium</title>
            <link>https://medium.com/@yotam56?source=rss-5eaaba1aaa97------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:23:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yotam56/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Hundreds of Pods to One Vector Index: Simplifying Model Inference with Elasticsearch]]></title>
            <link>https://medium.com/dynamicyield-techblog/from-hundreds-of-pods-to-one-vector-index-simplifying-model-inference-with-elasticsearch-cc91bbbd2801?source=rss-5eaaba1aaa97------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc91bbbd2801</guid>
            <category><![CDATA[ml-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[yotam feldman]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 14:18:05 GMT</pubDate>
            <atom:updated>2025-09-17T14:18:05.338Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://medium.com/u/d31ab1dd4f9d">binyamin greenberg</a>, <a href="https://medium.com/u/5eaaba1aaa97">yotam feldman</a></p><p><strong>TL;DR:</strong> We moved our NLP recommendation algorithm inference from per-customer Kubernetes pods backed by Annoy to precomputed embeddings served directly from Elasticsearch using HNSW KNN. The result: a drastically simpler footprint, lower operational overhead, and new product flexibility (vector similarity + classic filters in one query).</p><blockquote>Personal note:<em> There’s a special joy in refactoring by deletion — retiring multiple services and hundreds of pods in one go.</em></blockquote><h3>The problem we outgrew</h3><p>A few years back, our <strong>NextML</strong> algorithm predicted which product a user was most likely to interact with next, based on historical user interactions. To serve it, every customer (“section”) had a dedicated <strong>model-infer</strong> deployment — its own container image and its own Annoy index — running across environments (<a href="https://blog.dy.engineering/model-inference-at-scale-c4e3bc359bdc">you can read more about that setup here</a>). It worked, but scale exposed the cracks:</p><ul><li><strong>Operational drag:</strong> Hundreds of pods to build, ship, monitor, and secure.</li><li><strong>Scaling friction:</strong> CI/CD and autoscaling multiplied by customer count.</li><li><strong>Feature limits:</strong> Hard to combine semantic similarity with filters like price or category.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*06goMdt0y3bxpcburudSQg.png" /><figcaption>the old architecture (note the pod per section)</figcaption></figure><h3>The design goal</h3><p>We needed a system that would:</p><ul><li><strong>Scale with customers without multiplying pods.</strong></li><li><strong>Cut infra and licensing costs.</strong></li><li><strong>Unlock product features on top of vector data</strong> (e.g., “next items under $50 in Jackets”).</li></ul><h3>Why Elasticsearch</h3><p>Elasticsearch already powers our recommendation engine and holds <strong>per-customer product indices</strong>. We also used ES vector capabilities for other models (e.g., Visual Similarity). Leaning on a platform we operate in production allowed us to consolidate, rather than add, moving parts.</p><h3>The new architecture at a glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*mJ9Cez5UUGFtA0aFNRqJMg.png" /><figcaption>the new architecture</figcaption></figure><h3>Offline pipeline (batch)</h3><ul><li><strong>Embedding generation:</strong> We precompute product embeddings offline using the same training jobs in Amazon SageMaker.</li><li><strong>Data packaging:</strong> Airflow produces <strong>Parquet</strong> files with vectors per SKU.</li><li><strong>Indexing:</strong> Our rcom-worker jobs load vectors into <strong>our existing customer-scoped ES indices</strong>.</li></ul><p>Result: indexing is controlled, repeatable, and decoupled from real-time traffic.</p><h3>Online serving (query)</h3><ul><li><strong>Similarity engine:</strong> Elasticsearch’s <strong>HNSW-based KNN</strong> replaces Annoy.</li><li><strong>One query, many constraints:</strong> We combine vector similarity with standard filters/sorts.</li><li><strong>No inference pods:</strong> Traffic goes straight to ES; the pod farm disappears.</li></ul><p>A simplified query looks like this:</p><pre>POST /{customer_index}/_search<br>{<br>  &quot;knn&quot;: {<br>    &quot;field&quot;: &quot;embedding&quot;,<br>    &quot;query_vector&quot;: [/* embedding vector of the product the user is currently viewing */],<br>    &quot;k&quot;: 50,<br>    &quot;num_candidates&quot;: 1000<br>  },<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: [<br>        { &quot;term&quot;:   { &quot;category&quot;: &quot;jackets&quot; } },<br>        { &quot;range&quot;:  { &quot;price&quot;: { &quot;lte&quot;: 50 } } },<br>        { &quot;term&quot;:   { &quot;availability&quot;: &quot;in_stock&quot; } }<br>      ]<br>    }<br>  },<br>  &quot;sort&quot;: [{ &quot;score&quot;: &quot;desc&quot; }, { &quot;popularity&quot;: &quot;desc&quot; }]<br>}</pre><h3>What we gained</h3><ul><li><strong>Simpler infrastructure:</strong> Hundreds of per-customer inference pods → <strong>zero</strong>. We rely on our battle-tested ES clusters instead of maintaining a parallel serving tier.</li><li><strong>Lower costs:</strong> Less compute to autoscale and fewer moving services to maintain or license.</li><li><strong>Product flexibility:</strong> <strong>Similarity + filters</strong> becomes a single, composable ES query (e.g., “most similar under $50,” “exclude brand X,” “only in Women’s”). Crucially, those filters act as <strong>pre-filters</strong> during KNN retrieval, narrowing the candidate set <em>before</em> scoring — so we keep the most accurate, policy-compliant products instead of dropping results afterward with <strong>post-filters</strong>.</li><li><strong>Smarter ranking:</strong> Combine vector similarity scores with popularity scores. Top results are both relevant and in demand — not just sorted by popularity.</li><li><strong>Smaller security surface:</strong> Fewer services and images to patch, audit, and harden.</li></ul><h3>Rollout notes &amp; lessons learned</h3><p><strong>Dual-run for confidence.</strong> We ran <strong>model-infer</strong> and the ES-based path side-by-side, splitting traffic to compare outputs and latency before decommissioning the old service. We tracked:</p><ul><li><strong>Top-K overlap &amp; rank correlation</strong> for result parity.</li><li><strong>P95 latency</strong> under production load.</li><li><strong>ES index configurations.</strong></li></ul><h3>What’s next</h3><p><strong>Personalized Search?</strong> We may integrate with site search to rank results based on each user’s interactions and affinities, while honoring filters and constraints.</p><h3>Conclusion</h3><p>This was more than a cost exercise. By consolidating inference into Elasticsearch, we simplified our ML stack, aligned with systems we already operate at scale, and <strong>unlocked product capabilities</strong> that were awkward in a pod-per-customer world. It’s a cleaner foundation for the next generation of personalization at Dynamic Yield by Mastercard.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc91bbbd2801" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dynamicyield-techblog/from-hundreds-of-pods-to-one-vector-index-simplifying-model-inference-with-elasticsearch-cc91bbbd2801">From Hundreds of Pods to One Vector Index: Simplifying Model Inference with Elasticsearch</a> was originally published in <a href="https://medium.com/dynamicyield-techblog">DynamicYield Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Elasticsearch at Dynamic Yield: Cutting Latency by 67% and Enhancing Recall]]></title>
            <link>https://medium.com/dynamicyield-techblog/optimizing-elasticsearch-at-dynamic-yield-cutting-latency-by-67-and-enhancing-recall-580a53aefd44?source=rss-5eaaba1aaa97------2</link>
            <guid isPermaLink="false">https://medium.com/p/580a53aefd44</guid>
            <category><![CDATA[latency]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[knn]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[yotam feldman]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 14:17:26 GMT</pubDate>
            <atom:updated>2024-12-16T14:17:26.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2S61yhdy3x1xHvppbhRIA.jpeg" /></figure><p>At Dynamic Yield by Mastercard, we use Elasticsearch as a vector database to power our recommendation system. In this setup, we represent products as high-dimensional embedding vectors, allowing us to capture relationships between products.</p><p>It all started one morning when our dashboards lit up with red alerts — query latencies spiked as we rolled out from Early Access to General Availability (GA). We knew we couldn’t afford such delays in a real-time recommendation system, where every millisecond matters to the user experience. This challenge pushed us to dig deeper into our Elasticsearch setup and rethink our approach. Along the way, we also saw an opportunity to improve recall, ensuring our recommendations would be both fast and diverse.</p><p>This blog is dedicated to sharing the optimizations we implemented to overcome these challenges, providing you with practical insights and best practices to save time and improve the performance of your KNN queries.</p><h3>A Bookstore Analogy for Optimizing Elasticsearch Queries</h3><p>Imagine walking into a large bookstore, searching for a specific type of book. Each book in the store has been encoded into a unique vector — representing its content, genre, and other attributes. Using these vectors, our goal is to efficiently find the books most relevant to our search, but just like in a bookstore, we need a plan to make this process faster and more organized.</p><p>This analogy will guide us through the strategies to optimize Elasticsearch queries for kNN searches. We’ll explore techniques like <strong>pre-filters</strong>, <strong>field collapsing</strong>, and <strong>kNN queries</strong> within the context of the bookstore.</p><h3>Basic Parameters: Explained Through the Bookstore Analogy</h3><p>To optimize k-nearest neighbor (kNN) queries in Elasticsearch, it’s important to understand two key parameters: <strong>k</strong> and <strong>num_candidates</strong>. These settings define how Elasticsearch identifies and ranks the most relevant results. Let’s return to the bookstore analogy to break them down.</p><ul><li><strong>k (Nearest Neighbors to Return)</strong>: Imagine you ask a bookseller to recommend books and specify that you only want the <strong>5 best options</strong>. The number of books you want to see is equivalent to the k parameter — it defines how many nearest neighbors Elasticsearch should return in the search results.</li><li><strong>num_candidates (Potential Matches Evaluated)</strong>: Now think about how the bookseller (serving as an analogy to an Elasticsearch shard) decides which 5 books to recommend. Instead of searching the entire bookstore, they evaluate a subset of <strong>50 books</strong> that they think are the most promising candidates. This is analogous to the num_candidates parameter in Elasticsearch, which determines how many potential nearest neighbors each shard evaluates before selecting the top k results.</li></ul><h3>Balancing num_candidates and Performance</h3><ul><li>A <strong>higher num_candidates</strong> (like evaluating 100 books instead of 50) improves the chance of finding the best matches but increases the time it takes for the search.</li><li>A <strong>lower num_candidates</strong> reduces latency but risks missing some potentially better results.</li></ul><p>A typical top-level kNN query in Elasticsearch is structured as follows:</p><pre>{<br>  &quot;knn&quot;: {<br>    &quot;field&quot;: &quot;vector_field&quot;,<br>    &quot;query_vector&quot;: [0.1, 0.2, 0.3],<br>    &quot;k&quot;: 10,<br>    &quot;num_candidates&quot;: 100<br>  }<br>}</pre><p>After understanding the basic configurations, let’s dive into the tweaks that made the biggest impact and cut our latency by two-thirds on average!</p><p>The first and most fundamental is pre-filters.</p><h3><strong>Pre-Filters: Narrowing Down to a Specific Section</strong></h3><p>Instead of searching the entire bookstore, you focus only on the <strong>horror section</strong> since you’re looking for a spooky novel. This is the equivalent of using a <strong>pre-filter</strong> in Elasticsearch: narrowing the dataset before performing the computationally expensive kNN search. By applying a pre-filter, Elasticsearch doesn’t need to scan irrelevant sections, drastically improving efficiency.</p><p>Here’s how this might look in Elasticsearch:</p><pre>{<br>  &quot;knn&quot;: {<br>    &quot;field&quot;: &quot;vector_field&quot;,<br>    &quot;query_vector&quot;: [0.1, 0.2, 0.3],<br>    &quot;k&quot;: 10,<br>    &quot;num_candidates&quot;: 100,<br>    &quot;filter&quot;: [<br>      { &quot;term&quot;: { &quot;genre&quot;: &quot;horror&quot; } }<br>    ]<br>  }<br>}<br># This top-level KNN query includes a filter to narrow down the search to a specific genre (&quot;horror&quot;) before executing the k-nearest neighbor (KNN) search.</pre><h3><strong>Field Collapsing: Picking the Best Edition</strong></h3><p>In Elasticsearch, deduplication can be achieved using two options: <strong>aggregations</strong> and <strong>field collapsing</strong>. While aggregations offer flexibility for complex analytics, I chose field collapsing for its simplicity and efficiency in straightforward deduplication tasks.</p><p>Let’s use the bookstore analogy. Imagine you’re searching the horror section, and you find multiple editions of the same book. Instead of picking up every edition, you decide to select the latest one. This is exactly how field collapsing works in Elasticsearch — it ensures you retrieve only the top document for each unique field, like the most recent edition of a book, based on a specified sorting criterion.</p><p>Field collapsing is faster than aggregations because it operates directly during the search phase, avoiding the two-phase process of collection and reduction. It consumes fewer resources, uses less memory and CPU, and has a much simpler syntax.</p><p>Here’s an example query with field collapsing:</p><pre>{<br>  &quot;knn&quot;: {<br>    &quot;field&quot;: &quot;vector_field&quot;,<br>    &quot;query_vector&quot;: [0.1, 0.2, 0.3],<br>    &quot;k&quot;: 10,<br>    &quot;num_candidates&quot;: 100<br>  },<br>  &quot;collapse&quot;: {<br>    &quot;field&quot;: &quot;edition_date&quot;<br>  }<br>}<br># This example demonstrates a top-level KNN query with a collapse step to deduplicate results based on the &quot;edition_date&quot; field. Using field collapsing ensures that only the most recent edition is retrieved.</pre><p>For scenarios requiring simple deduplication, field collapsing is the optimal choice over aggregations.</p><h3>kNN Query (Not at the Top Level): A Better Recall</h3><p>One powerful optimization to consider is using a kNN query (kNN inside a query) instead of a top-level kNN search. The main difference lies in flexibility: a kNN query can be combined with other query types, such as bool, match, or range, making it highly versatile.</p><p>I want to focus on the <strong>advantage of the kNN query in achieving the desired k elements after reducing duplication</strong> through collapse or aggregation.</p><p>To illustrate, let’s return to the bookstore analogy. Imagine you’re looking to buy <strong>5 books</strong>. You ask several booksellers to help you find the best options. Each bookseller independently searches the horror section and selects their <strong>top 10 recommendations</strong>, filtering out duplicates within their own selections (e.g., different editions of the same book). After combining their results, you get <strong>exactly 5 diverse books</strong> from different authors and brands.</p><p>Now consider how a <strong>top-level kNN search</strong> would work. Here, each bookseller still brings their <strong>top 10 recommendations</strong>, but this time, duplicates (such as different editions of the same book) are not filtered at the individual level. Once all books are collected, the system selects the <strong>top 5 books globally</strong> and only then applies collapsing to remove duplicates. This could result in fewer than 5 unique books — for instance, if 2 of the top results are editions of the same book, collapsing will leave you with just <strong>4 unique books</strong> instead of the 5 you wanted. This lack of diversity is a key drawback of the top-level approach.</p><pre>{<br>  &quot;size&quot;: 10,<br>  &quot;query&quot;: {<br>    &quot;knn&quot;: {<br>      &quot;field&quot;: &quot;vector_field&quot;,<br>      &quot;query_vector&quot;: [0.1, 0.2, 0.3],<br>      &quot;num_candidates&quot;: 100<br>    }<br>  }<br>}<br># This example shows a kNN query that is not at the top level, retrieving the 10 nearest neighbors while evaluating 100 candidates. It retains the flexibility and capabilities of a typical Elasticsearch query structure.</pre><h3><strong>Last Example: Combining kNN Query, Filtering, and Field Collapsing</strong></h3><pre>{<br>  &quot;size&quot;: 10,<br>  &quot;query&quot;: {<br>    &quot;knn&quot;: {<br>      &quot;field&quot;: &quot;vector_field&quot;,<br>      &quot;query_vector&quot;: [0.1, 0.2, 0.3],<br>      &quot;num_candidates&quot;: 100,<br>      &quot;filter&quot;: {<br>        &quot;bool&quot;: {<br>          &quot;filter&quot;: [<br>            {<br>              &quot;term&quot;: {<br>                &quot;genre&quot;: &quot;horror&quot;<br>              }<br>            }<br>          ]<br>        }<br>      }<br>    }<br>  },<br>  &quot;collapse&quot;: {<br>    &quot;field&quot;: &quot;edition_date&quot;<br>  }<br>}<br># This example demonstrates a combined approach using a kNN query with a filter to limit results to the &quot;horror&quot; genre and field collapsing to retrieve only the most recent edition.</pre><h3><strong>Dynamic Yield Results: Latency Dropped, Recall Improved</strong></h3><p>At Dynamic Yield, we implemented the discussed optimizations across our Elasticsearch-powered recommendation system. To evaluate the impact, I tested the system on an Elasticsearch index containing <strong>21GB of data and 1.8 million documents</strong>. For the kNN query, I set the number of candidates (num_candidates) to <strong>600</strong> and requested the top <strong>10 products (k)</strong> per query.</p><h4>The Results: Latency Reduction</h4><p>The average query latency before applying the optimizations was <strong>31.36 ms</strong>. After implementing the optimizations — including pre-filters, kNN query, and field collapsing — the average query time dropped to just <strong>10.27 ms</strong>. That’s a reduction of nearly <strong>67%</strong> — a significant improvement for real-time applications! 🏆</p><h3><strong>Post-Deployment: Latency Reduction in Slowest Elasticsearch Queries</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fj6AfwAhw5Ws633_EiDY-g.jpeg" /><figcaption>This figure highlights the impact of our optimizations on Elasticsearch query performance, focusing on reducing latency in both the 90th and 50th percentiles of slow queries. The red line represents the 90th percentile, showing the slowest query times, while the yellow line represents the 50th percentile (median) query times. Following the deployment at around 11:45, both lines show a notable drop in latency, with a particularly significant reduction in the 90th percentile (red line). This clearly demonstrates the effectiveness of our optimizations in reducing high-latency outliers and improving overall system responsiveness.</figcaption></figure><h3>Recall Improved: Better Results with Reduced Latency</h3><p>In addition to the latency reduction, we also addressed a key issue with recall. Before optimization, deduplication and collapsing led to fewer results than requested, reducing the diversity of recommendations. By integrating shard-level collapsing and carefully configuring num_candidates, we ensured that each query returned the requested <strong>10 unique products</strong> with greater diversity and consistency.</p><p>I hope this blog has provided you with practical insights and actionable strategies to enhance your own Elasticsearch setups. Whether you’re working on a recommendation engine or another real-time application, these optimizations can help you achieve faster, more reliable performance.</p><p>Thank you for reading,</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=580a53aefd44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dynamicyield-techblog/optimizing-elasticsearch-at-dynamic-yield-cutting-latency-by-67-and-enhancing-recall-580a53aefd44">Optimizing Elasticsearch at Dynamic Yield: Cutting Latency by 67% and Enhancing Recall</a> was originally published in <a href="https://medium.com/dynamicyield-techblog">DynamicYield Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>