<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Cox on Medium]]></title>
        <description><![CDATA[Stories by Samuel Cox on Medium]]></description>
        <link>https://medium.com/@samuelcox_77452?source=rss-c217ad87d341------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Samuel Cox on Medium</title>
            <link>https://medium.com/@samuelcox_77452?source=rss-c217ad87d341------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:02:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samuelcox_77452/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What I learned from 5 years working with OpenSearch]]></title>
            <link>https://medium.com/@samuelcox_77452/what-i-learned-from-5-years-working-with-opensearch-d326a0029800?source=rss-c217ad87d341------2</link>
            <guid isPermaLink="false">https://medium.com/p/d326a0029800</guid>
            <dc:creator><![CDATA[Samuel Cox]]></dc:creator>
            <pubDate>Mon, 02 Mar 2026 20:45:15 GMT</pubDate>
            <atom:updated>2026-03-03T00:01:41.255Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>I recently left a job as a Senior Software Engineer for a large e-commerce company with 12 million active customers. I worked in the “Search” squad there, essentially, a team that works on surfacing products via searches, how those products are ranked, and lots of stuff in between. We used Opensearch very heavily for this.</p><p>I had never worked with Opensearch before, and learned lots of lessons over the years about running Opensearch at scale that I wanted to distill down and release to the public.</p><p>You may at times see me linking to ElasticSearch documentation rather than Opensearch. That’s typically just because the ElasticSearch documentation is better, but generally applies to both ElasticSearch and Opensearch.</p><h3>Avoid Painless like the devil</h3><p>If I had a pound for every time the team made the joke “painless, more like painful” I would be a very rich man. For the un-initiated, painless is a scripting language that allows you to do some very powerful and custom things, like using something other than BM25 for ranking. <a href="https://www.elastic.co/docs/reference/scripting-languages/painless/brief-painless-walkthrough">https://www.elastic.co/docs/reference/scripting-languages/painless/brief-painless-walkthrough</a></p><pre>GET products/_search <br>{<br>  &quot;query&quot;: {<br>    &quot;script_score&quot;: {<br>      &quot;query&quot;: {<br>        &quot;match&quot;: {<br>          &quot;name&quot;: &quot;laptop&quot;<br>        }<br>      },<br>      &quot;script&quot;: {<br>        &quot;lang&quot;: &quot;painless&quot;,<br>        &quot;source&quot;: &quot;_score + doc[&#39;popularity&#39;].value * 0.1&quot;<br>      }<br>    }<br>  }<br>}</pre><p>In the example above, we are using painless to have a weighted sort between popularity and the original document score provided by Opensearch. Initially, this seems very flexible and powerful, and indeed it’s something we would initially reach for quite often. But, I came to think of painless as a “footgun”. While opensearch/elastic may bill painless as performant, operating at scale and trying to get as low latency as possible, you are better off not using it. Because the script executes on every single document, it can be really harmful to latency.</p><p>While painless was often the “easiest” way to implement a feature, it was rarely if ever the best. In most cases there was always a way to do it without painless, such as a rank_features query or an external reranker, which some of the other sections of this blog will go over.</p><p>While I’ve only gone over examples of scoring here, we also saw painless hurt performance in other areas such as aggregations and etc. If I were to implement an opensearch service from the ground up tomorrow, one of my guiding philosophies would be “Can we implement this without using painless, and if so we should.”</p><h3>Revisit your query every so often</h3><p>Code doesn’t stand still, and at a high velocity organisation the query is going to evolve over time. In particular, we invested very heavily in A/B testing. So we would add things on to the query very often to support these A/B tests, and over time the query grew large and very complicated, and hard to reason about/understand, especially for new engineers or engineers that weren’t experienced in Opensearch.</p><p>We used the high-level .net client to build up our queries <a href="https://docs.opensearch.org/latest/clients/dot-net/">https://docs.opensearch.org/latest/clients/dot-net/</a>, which meant that our code was not exactly 1–1 with the query structure sent over the wire to Opensearch. I’ll dive deeper into this in the next section as I have some thoughts on this, but it meant the query ended up with lots of unnecessary nesting, nested should clauses, etc.</p><p>Unnecessary nesting below:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;bool&quot;: {<br>            &quot;must&quot;: [<br>              {<br>                &quot;term&quot;: {<br>                  &quot;status&quot;: &quot;active&quot;<br>                }<br>              }<br>            ]<br>          }<br>        },<br>        {<br>          &quot;bool&quot;: {<br>            &quot;must&quot;: [<br>              {<br>                &quot;term&quot;: {<br>                  &quot;category&quot;: &quot;electronics&quot;<br>                }<br>              }<br>            ]<br>          }<br>        },<br>        {<br>          &quot;bool&quot;: {<br>            &quot;must&quot;: [<br>              {<br>                &quot;range&quot;: {<br>                  &quot;price&quot;: {<br>                    &quot;gte&quot;: 100<br>                  }<br>                }<br>              }<br>            ]<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>Every so often, you should at least attempt to rewrite the query from the ground up, with all the same features but in the simplest way available to you in Opensearch.Client. Often this will result in a performance win, and will enable you to understand and debug the query much easier.</p><p>Fixed version below:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;term&quot;: {<br>            &quot;status&quot;: &quot;active&quot;<br>          }<br>        },<br>        {<br>          &quot;term&quot;: {<br>            &quot;category&quot;: &quot;electronics&quot;<br>          }<br>        },<br>        {<br>          &quot;range&quot;: {<br>            &quot;price&quot;: {<br>              &quot;gte&quot;: 100<br>            }<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><h3>Is the high-level client worth it?</h3><p>So, this particular section will be language specific, as we used .NET. Opensearch provided 2 clients, a high-level client and a low-level client. The high level client is quite nice as it provides some fairly idiomatic fluent C# methods to build up the search request. However, over the years we experienced a fair few problems on it.</p><ol><li>Missing client functionality: The high level .net client appears to have a fairly small team working on it, and they do a great job! But, a small team for the amount of functionality Opensearch provides and adds in new releases isn’t enough. I often found there were small bits of functionality missing, such as the min_score property <a href="https://github.com/opensearch-project/opensearch-net/issues/622">https://github.com/opensearch-project/opensearch-net/issues/622</a> or the sltr query: <a href="https://github.com/opensearch-project/opensearch-net/issues/890">https://github.com/opensearch-project/opensearch-net/issues/890</a> These are just the issues I personally logged, there are a lot more.</li><li>Performance: Opensearch.Client uses a no longer maintained Json library, called Utf8Json <a href="https://github.com/opensearch-project/opensearch-net/issues/388">https://github.com/opensearch-project/opensearch-net/issues/388</a> There was a period of time when Utf8Json was the fastest json library by a significant margin, perhaps that’s even still true now, but the project has not been actively maintained for years. One of the big problems with this is it prevents AOT compilation for .NET, which could result in a big performance increase.</li></ol><p>For these reasons, I often wondered if we would be better off moving off of the high level client, or indeed even directly constructing REST requests and sending them to Opensearch ourselves (or perhaps even rewriting in a different language, since .NET seems like the least favourite child of the family from Amazon’s investment in Opensearch SDKs.) All of these are fraught with problems of course, you lose a fair bit of productivity moving to the low level client, directly constructing REST requests would leave you open to injection attacks if you aren’t very careful about how you validate data, and more. But if you’re trying to get all the performance possible, these seem like options that should be considered.</p><p>It also seems like there are questions around if Opensearch.Client for .NET is actively maintained at all anymore, I know there have been some efforts in Java to move to a single client:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*75_-D6_44_RghYbEWCNBLw.png" /></figure><p>But, I’d be pretty surprised if the .NET client was no longer maintained at all, or if so I’d imagine they are working on a successor. This is just a rumour for now: <a href="https://github.com/opensearch-project/opensearch-net/issues/957">https://github.com/opensearch-project/opensearch-net/issues/957</a></p><h3>Do as much at index time as possible</h3><p>Most people are probably reading this and thinking “well, duh”. Of course even outside of Opensearch, in for example event driven architecture or just any asynchronous writes, it’s best practice to do as much of the hard work when you write to the database, so that reading from the database is as simple and low latency as possible.</p><p>Indeed, the same applies to Opensearch, but I personally found it wasn’t always as obvious that things could be done at index time compared to SQL or a key-value/document database.</p><p>Let’s go back to our painless example:</p><pre>GET products/_search <br>{<br>  &quot;query&quot;: {<br>    &quot;script_score&quot;: {<br>      &quot;query&quot;: {<br>        &quot;match&quot;: {<br>          &quot;name&quot;: &quot;laptop&quot;<br>        }<br>      },<br>      &quot;script&quot;: {<br>        &quot;lang&quot;: &quot;painless&quot;,<br>        &quot;source&quot;: &quot;_score + doc[&#39;popularity&#39;].value * 0.1&quot;<br>      }<br>    }<br>  }<br>}</pre><p>If instead you index a document with the different scores pre-computed you want to use in different scenarios, you can make the read side much simpler and often more performant.</p><pre>POST products/_doc/1<br>{<br>  &quot;name&quot;: &quot;Gaming laptop&quot;,<br>  &quot;popularity&quot;: 50,<br>  &quot;precomputed_score&quot;: 5.0<br>}<br><br>GET products/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;function_score&quot;: {<br>      &quot;query&quot;: {<br>        &quot;match&quot;: {<br>          &quot;name&quot;: &quot;laptop&quot;<br>        }<br>      },<br>      &quot;field_value_factor&quot;: {<br>        &quot;field&quot;: &quot;precomputed_score&quot;,<br>        &quot;factor&quot;: 1.0,<br>        &quot;missing&quot;: 0<br>      },<br>      &quot;boost_mode&quot;: &quot;sum&quot;<br>    }<br>  }<br>}</pre><p>As alluded to in the painless section, we just got rid of all of the painless by doing this. Opensearch provides lots of powerful ways to do scoring, such as function-score or rank_features , use them where you can and where it is sensible to!</p><h3>Don’t assume Opensearch doesn’t have bugs</h3><p>In general, as software engineers, we tend to assume the infrastructure we’ve built functionality on top of can’t be buggy, it must be our code! 99 times out of 100, this is correct. But by the law of big numbers, you are going to run into this eventually.</p><p>This bug <a href="https://github.com/opensearch-project/OpenSearch/issues/19813">https://github.com/opensearch-project/OpenSearch/issues/19813</a> was a real head-scratcher until we found the corresponding issue in the Elasticsearch repo (and because opensearch was originally forked from ES, it has the same issue).</p><p>This breaking change, which was not announced as a breaking change! <a href="https://github.com/opensearch-project/k-NN/issues/2561">https://github.com/opensearch-project/k-NN/issues/2561</a></p><h3>Sharding and Replicas</h3><p>This is almost certainly the biggest impact section of all of the above. Getting the size of your shards right has a huge impact on Opensearch performance, and this is already well documented. In addition, choosing the right amount of replicas based on how read-heavy your workload vs write-heavy your workload has a huge effect. As there are many good resources online about this, I’m not going to relitigate this. But I’ll point to some great resources I used over the years:<br><a href="https://opensearch.org/blog/optimize-opensearch-index-shard-size/">https://opensearch.org/blog/optimize-opensearch-index-shard-size/</a><br><a href="https://opster.com/blogs/opensearch-shards-and-replicas/">https://opster.com/blogs/opensearch-shards-and-replicas/</a><br><a href="https://opensearch.org/blog/improve-opensearch-cluster-performance-by-separating-search-and-indexing-workloads/">https://opensearch.org/blog/improve-opensearch-cluster-performance-by-separating-search-and-indexing-workloads/</a></p><h3>Don’t overuse Opensearch</h3><p>Opensearch has had a ton of investment over the years, and supports some great advanced features like kNN and learning to rank. These features are great, but sometimes you are going to hit limits in Opensearch where you’d be better suited using something else.</p><p>I eluded to the use of external re-rankers earlier, and this is where it comes to fruition. Offloading some features from Opensearch allows you to use less CPU on Opensearch, and external rerankers typically support significantly more features (features as in a feature set in machine learning, not features in the traditional sense) than Opensearch. Opensearch supports dozens to hundreds, an external reranker can support essentially unbounded feature sets.</p><h3><strong>Upgrade Opensearch as often as possible</strong></h3><p>As I mentioned before, Opensearch has had a ton of investment over the years, and continues to. With that, every single version upgrade of the database comes with a great performance win, essentially for free.</p><p>Opensearch will typically go over these performance improvements in their release posts, for example: <a href="https://opensearch.org/blog/unveiling-opensearch-3-0/">https://opensearch.org/blog/unveiling-opensearch-3-0/</a></p><p>But, even minor version upgrades can have significant wins, so I’d recommend upgrading as often as possible. Obviously, keep your eye out for breaking changes.</p><h3>Set refresh interval appropriately</h3><p>Opensearch exposes an index setting called refresh_interval. This controls how frequently data is refreshed. If you don’t need truly real-time data, you should consider raising it from the default 1s to something higher, like 60s. Here’s a really good resource on this that explains it better than I can: <a href="https://opensearch.org/blog/optimize-refresh-interval/">https://opensearch.org/blog/optimize-refresh-interval/</a></p><h3>Use advancements in OpenSearch</h3><p>If I were to implement a search service from the ground up tomorrow, there’s really very little I’d do exactly the same. I’ve mentioned opensearch has received a lot of investment recently, and hopefully mentioning it again brings the point home. (Disclaimer, some of these concepts exist in elasticsearch too, so I’m not saying Opensearch solely did it)</p><p>Search Pipelines and Search Templates are some really interesting additions that make me think things could be done differently.</p><p>Search Pipelines, <a href="https://docs.opensearch.org/latest/search-plugins/search-pipelines/index/">https://docs.opensearch.org/latest/search-plugins/search-pipelines/index/</a> are basically lego blocks that you stick together to add on functionality in search. Over time, using something like Opensearch.client to build up a request becomes complex as previously mentioned. Pipelines allow you to do that more declaratively, like having a pipeline that filters out sold out products. For example:</p><pre>PUT /_search/pipeline/my_pipeline <br>{<br>  &quot;request_processors&quot;: [<br>    {<br>      &quot;filter_query&quot; : {<br>        &quot;tag&quot; : &quot;tag1&quot;,<br>        &quot;description&quot; : &quot;This processor is going to restrict to sold out documents&quot;,<br>        &quot;query&quot; : {<br>          &quot;term&quot;: {<br>            &quot;soldOut&quot;: &quot;false&quot;<br>          }<br>        }<br>      }<br>    }<br>  ],<br>  &quot;response_processors&quot;: [<br>    {<br>      &quot;rename_field&quot;: {<br>        &quot;field&quot;: &quot;message&quot;,<br>        &quot;target_field&quot;: &quot;notification&quot;<br>      }<br>    }<br>  ]<br>}<br><br>GET /my_index/_search?search_pipeline=my_pipeline</pre><p>This allows you to define the soldOut behaviour outside of your core query (and you can compose multiple pipelines together to build up your search behaviour). This does always make me think of the old adage of “don’t put business logic in your stored procedures”. It feels like you might end up in that world again, but in opensearch rather than SQL. But, I think it’s an interesting idea and I can see the benefits it may have.</p><p>Search Templates, I think are a more clear and obvious improvement. <a href="https://docs.opensearch.org/latest/api-reference/search-apis/search-template/index/">https://docs.opensearch.org/latest/api-reference/search-apis/search-template/index/</a><br>Essentially, instead of building up your entire search request from scratch and sending it over the wire, you just define a templated search query on the index. Then you send a request to Opensearch saying “use this template, with these parameters”. And job done. It allows you to avoid most of the pain of creating opensearch queries with libraries, and it means you use less bandwidth over the wire by only sending the parameters. In theory, it could improve performance. In practice, I never got to try it, but if I were building search tomorrow I’d use it.</p><pre>POST _scripts/product_search_template<br>{<br>  &quot;script&quot;: {<br>    &quot;lang&quot;: &quot;mustache&quot;,<br>    &quot;source&quot;: {<br>      &quot;query&quot;: {<br>        &quot;bool&quot;: {<br>          &quot;must&quot;: [<br>            {<br>              &quot;match&quot;: {<br>                &quot;name&quot;: &quot;{{search_term}}&quot;<br>              }<br>            }<br>          ],<br>          &quot;filter&quot;: [<br>            {<br>              &quot;term&quot;: {<br>                &quot;category&quot;: &quot;{{category}}&quot;<br>              }<br>            },<br>            {<br>              &quot;range&quot;: {<br>                &quot;price&quot;: {<br>                  &quot;gte&quot;: &quot;{{min_price}}&quot;,<br>                  &quot;lte&quot;: &quot;{{max_price}}&quot;<br>                }<br>              }<br>            }<br>          ]<br>        }<br>      }<br>    }<br>  }<br>}<br><br>POST products/_search/template<br>{<br>  &quot;id&quot;: &quot;product_search_template&quot;,<br>  &quot;params&quot;: {<br>    &quot;search_term&quot;: &quot;Laptop&quot;,<br>    &quot;category&quot;: &quot;electronics&quot;,<br>    &quot;min_price&quot;: 500,<br>    &quot;max_price&quot;: 2000<br>  }<br>}</pre><p>Well, that’s it for now. I’m out of the search game as I’ve gone into a non search role, but I’ll be attending search meetups and talks to keep my mind sharp on this domain, and maybe one day I’ll make a part two!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d326a0029800" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>