<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gopalgiridhar on Medium]]></title>
        <description><![CDATA[Stories by Gopalgiridhar on Medium]]></description>
        <link>https://medium.com/@gopalgiridhar123?source=rss-a95b5944a5af------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*avAkok_z8EdL_-IM</url>
            <title>Stories by Gopalgiridhar on Medium</title>
            <link>https://medium.com/@gopalgiridhar123?source=rss-a95b5944a5af------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 00:08:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gopalgiridhar123/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Migrating from Elasticsearch to OpenSearch: A Practical Journey]]></title>
            <link>https://medium.com/@gopalgiridhar123/migrating-from-elasticsearch-to-opensearch-a-practical-journey-62c9ff266c4a?source=rss-a95b5944a5af------2</link>
            <guid isPermaLink="false">https://medium.com/p/62c9ff266c4a</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[opensearch]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Gopalgiridhar]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 08:46:01 GMT</pubDate>
            <atom:updated>2026-04-06T08:46:01.339Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*dgT0SC2ZymJftmSIQh8x7Q.png" /></figure><p>Why Migrate?</p><p>In 2021, Elastic changed the licensing of Elasticsearch from the permissive Apache 2.0 license to the Server Side Public License (SSPL) and the Elastic License. For many organizations, this shift introduced legal ambiguity — especially for teams running Elasticsearch as part of a managed service or embedding it within a product offering.</p><p>AWS responded by forking Elasticsearch 7.10.2 into <strong>OpenSearch</strong>, an open-source, Apache 2.0–licensed project. For teams that valued licensing clarity, community-driven governance, and freedom from vendor lock-in, the choice was straightforward: migrate to OpenSearch.</p><p>Our motivations boiled down to three things:</p><ol><li><strong>Licensing certainty</strong> — Apache 2.0 is well-understood, with no restrictions on how the software is used or hosted.</li><li><strong>Long-term viability</strong> — OpenSearch has strong community backing and an active release cadence.</li><li><strong>Cost</strong> — Removing dependency on Elastic’s commercial offerings reduced our operational spend.</li></ol><h3>What Changed?</h3><p>At its core, OpenSearch 1.x is a fork of Elasticsearch 7.10.2. The APIs, query DSL, and data structures are nearly identical. But “nearly” does a lot of heavy lifting. Here’s what we encountered:</p><h3>Namespace and Package Changes</h3><p>Every reference to org.elasticsearch in your Java codebase needs to become org.opensearch. The same applies to REST endpoint prefixes, configuration keys, and client libraries.</p><ul><li><strong>Elasticsearch High-Level REST Client (RHLC)</strong> → replaced by the <strong>OpenSearch Java Client</strong> (though OpenSearch 1.x still ships a compatibility layer based on the RHLC).</li><li>Internal plugin and module names changed.</li><li>Kibana became <strong>OpenSearch Dashboards</strong>.</li></ul><h3>Feature Gaps in 1.x</h3><p>OpenSearch 1.x inherited the Elasticsearch 7.10 feature set. Any feature Elastic introduced <em>after</em> 7.10 — runtime fields, frozen-tier searchable snapshots, the new Kibana alerting framework — simply doesn’t exist in OpenSearch 1.x. We audited our usage and confirmed none of our workloads depended on post-7.10 features.</p><h3>Migration Steps (The “How”)</h3><p>Our migration followed this path:</p><p><strong>1. Snapshot &amp; Restore</strong> We took a snapshot of our Elasticsearch 7.10 cluster and restored it directly into OpenSearch 1.x. Index compatibility was seamless since OpenSearch 1.x reads Elasticsearch 7.x indices natively.</p><p><strong>2. Client Library Swap</strong> We replaced the Elasticsearch RHLC dependency with the OpenSearch-compatible RHLC (opensearch-rest-high-level-client). At this stage, the API surface is almost identical — most changes were mechanical find-and-replace operations on imports.</p><p><strong>3. Configuration &amp; Tooling Updates</strong> Security plugin configuration moved from Open Distro conventions to the OpenSearch Security plugin. Dashboards configuration files, index patterns, and saved objects required re-export and re-import.</p><p><strong>4. Validation</strong> We ran our full integration test suite against the new cluster, paying close attention to aggregation results, scoring behavior, and ingest pipeline outputs. Everything matched.</p><h3>The Pagination Problem</h3><p>Once we were stable on OpenSearch 1.x, a new requirement surfaced: <strong>deep pagination</strong>.</p><p>Our application needed to page through large result sets — tens of thousands of documents — reliably. The classic from + size approach is well-known to degrade badly beyond 10,000 hits (and is hard-capped at index.max_result_window). We needed a cursor-based approach.</p><p>OpenSearch 1.x supports search_after, which solves the forward-pagination problem by using the sort values of the last document as a cursor for the next page. However, it has a critical limitation: <strong>the index can change between requests</strong>. If documents are indexed, updated, or deleted mid-pagination, results can shift — you might see duplicates or miss records entirely.</p><p>The robust solution is <strong>search_after + Point in Time (PIT)</strong>. A PIT creates a lightweight, frozen view of the index at a specific moment. Combined with search_after, it guarantees consistent pagination even as the underlying data changes.</p><p><strong>The catch:</strong> PIT was not available in OpenSearch 1.x. It was introduced in <strong>OpenSearch 2.x</strong>.</p><h3>Upgrading to OpenSearch 2.x</h3><p>This gave us a strong technical reason to upgrade. OpenSearch 2.x is not a drop-in upgrade from 1.x — it involves breaking changes:</p><ul><li>Minimum Java version moved to Java 11 (and later releases target Java 17).</li><li>Several deprecated APIs were removed.</li><li>Security plugin defaults changed.</li><li>Some query behaviors were tightened.</li></ul><p>We performed a rolling upgrade in our staging environment, resolved a handful of deprecation warnings, and validated again. The process was smoother than the original Elasticsearch-to-OpenSearch migration.</p><h3>Building the Paginated API with search_after</h3><p>With OpenSearch 2.x in place, we implemented cursor-based pagination. Here’s the approach:</p><h3>Basic search_after (Without PIT)</h3><p>This is the simpler variant and works well when your index is relatively stable during the pagination window.</p><pre>SearchRequest request = new SearchRequest(&quot;my-index&quot;);<br>SearchSourceBuilder source = new SearchSourceBuilder();<br>source.query(QueryBuilders.matchAllQuery());<br>source.size(20);<br>source.sort(&quot;timestamp&quot;, SortOrder.DESC);<br>source.sort(&quot;_id&quot;, SortOrder.ASC); // tiebreaker<br><br>// For subsequent pages, set the search_after values<br>if (lastSortValues != null) {<br>    source.searchAfter(lastSortValues);<br>}<br><br>request.source(source);<br>SearchResponse response = client.search(request, RequestOptions.DEFAULT);<br><br>// Extract sort values from the last hit for the next page<br>SearchHit[] hits = response.getHits().getHits();<br>if (hits.length &gt; 0) {<br>    Object[] nextCursor = hits[hits.length - 1].getSortValues();<br>    // Encode and return nextCursor to the API caller<br>}</pre><p>The API contract looks like:</p><pre>GET /api/v1/records?pageSize=20<br>GET /api/v1/records?pageSize=20&amp;searchAfter=&lt;sort_value_string&gt;</pre><p>The searchAfter parameter is a string representing the sort values of the last document in the previous page. The client receives it in the response and passes it back on the next request to fetch the following page.</p><h3>Tiebreaker Sort</h3><p>Always include a unique tiebreaker field (typically _id or a unique timestamp) as the final sort criterion. Without it, documents with identical sort values may be returned in unpredictable order, causing missed or duplicated results.</p><h3>Future: search_after + PIT and RHLC Migration</h3><h3>Point in Time (PIT) for Consistent Pagination</h3><p>For use cases where data mutates frequently during a user’s pagination session, the next step is adding PIT:</p><pre>// 1. Create a PIT<br>CreatePitRequest pitRequest = new CreatePitRequest(&quot;my-index&quot;);<br>pitRequest.setKeepAlive(TimeValue.timeValueMinutes(5));<br>CreatePitResponse pitResponse = client.createPit(pitRequest, RequestOptions.DEFAULT);<br>String pitId = pitResponse.getId();<br><br>// 2. Search using PIT + search_after<br>SearchRequest request = new SearchRequest(); // no index needed — PIT encodes it<br>SearchSourceBuilder source = new SearchSourceBuilder();<br>source.query(QueryBuilders.matchAllQuery());<br>source.size(20);<br>source.sort(&quot;timestamp&quot;, SortOrder.DESC);<br>source.sort(&quot;_id&quot;, SortOrder.ASC);<br><br>// Attach PIT<br>source.pointInTimeBuilder(new PointInTimeBuilder(pitId).setKeepAlive(TimeValue.timeValueMinutes(5)));<br><br>if (lastSortValues != null) {<br>    source.searchAfter(lastSortValues);<br>}<br><br>request.source(source);<br>SearchResponse response = client.search(request, RequestOptions.DEFAULT);<br><br>// 3. When pagination is complete, delete the PIT<br>DeletePitRequest deleteRequest = new DeletePitRequest(pitId);<br>client.deletePit(deleteRequest, RequestOptions.DEFAULT);</pre><p><strong>Key behaviors:</strong></p><ul><li>The PIT freezes the searcher’s view of the index segments. New writes won’t appear; deletes won’t cause records to vanish mid-pagination.</li><li>The PIT has a TTL (keep_alive). Refresh it on each request by passing keep_alive again, or it expires and subsequent requests fail.</li><li>PIT IDs can change between requests — always use the PIT ID returned in the <em>latest</em> response.</li></ul><h3>Migrating from RHLC to the OpenSearch Java Client</h3><p>The OpenSearch-compatible RHLC is a bridge, not a destination. It carries the weight of the old Elasticsearch client design and will eventually be deprecated. The <strong>OpenSearch Java Client</strong> (built on the opensearch-java artifact) is the recommended long-term client.</p><p>The migration involves:</p><ol><li><strong>Dependency swap</strong>: Replace opensearch-rest-high-level-client with org.opensearch.client:opensearch-java.</li><li><strong>Client construction</strong>: Move from RestHighLevelClient to OpenSearchClient, backed by an ApacheHttpClient5Transport.</li><li><strong>Query building</strong>: The new client uses a functional builder pattern instead of the old SearchSourceBuilder / QueryBuilders static methods.</li></ol><pre>// New client setup<br>OpenSearchClient client = new OpenSearchClient(<br>    ApacheHttpClient5TransportBuilder<br>        .builder(HttpHost.create(&quot;https://opensearch-node:9200&quot;))<br>        .build()<br>);<br><br>// search_after with PIT using the new client<br>SearchResponse&lt;MyDocument&gt; response = client.search(s -&gt; s<br>    .pit(p -&gt; p.id(pitId).keepAlive(t -&gt; t.time(&quot;5m&quot;)))<br>    .size(20)<br>    .sort(so -&gt; so.field(f -&gt; f.field(&quot;timestamp&quot;).order(SortOrder.Desc)))<br>    .sort(so -&gt; so.field(f -&gt; f.field(&quot;_id&quot;).order(SortOrder.Asc)))<br>    .searchAfter(lastSortValues)<br>    .query(q -&gt; q.matchAll(m -&gt; m)),<br>    MyDocument.class<br>);</pre><p>This migration can be done incrementally — the old and new clients can coexist during a transition period since they use separate HTTP transports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1uXGDjeunWNXGQLVCz80HQ.png" /></figure><p>Each phase was a self-contained, deployable milestone. We didn’t try to do everything at once — and that’s what made the migration manageable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62c9ff266c4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>