<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeevanandham Selvaraj on Medium]]></title>
        <description><![CDATA[Stories by Jeevanandham Selvaraj on Medium]]></description>
        <link>https://medium.com/@jeevanandham.selvaraj42?source=rss-2dc78bec69de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*PHSZrvbZ8z_bJpch</url>
            <title>Stories by Jeevanandham Selvaraj on Medium</title>
            <link>https://medium.com/@jeevanandham.selvaraj42?source=rss-2dc78bec69de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:50:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeevanandham.selvaraj42/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: significant_text Aggregation for Keyword Extraction]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-significant-text-aggregation-for-keyword-extraction-bdbf16ea3d14?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdbf16ea3d14</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 24 Jan 2024 16:49:21 GMT</pubDate>
            <atom:updated>2024-01-24T17:03:53.726Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the twenty eighth episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today let us try to learn about what is significant_text and how it can be used to create details keyword extraction techniques in Elasticsearch. Since this is a progressive series and if you are new to Elasticsearch DSL or to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>Syntax</h3><p>The significant_text aggregation is very useful when we want to extract meaningful keywords from a large chunk of texts. Usually the traditional aggregations models that work mostly based on the term frequency but the significatn_text aggregation uses statistical significance to actually find out the terms that stands out than the rest.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;significant_keywords&quot;: {<br>      &quot;significant_text&quot;: {<br>        &quot;field&quot;: &quot;content&quot;,<br>        &quot;size&quot;: 10<br>      }<br>    }<br>  }<br>}</pre><p>In this query snippet, we can extract the power of significant_text by specifying the field (content) from which we want to extract significant keywords and setting the size of the result set to 10.</p><h3>How is it different from Term Frequency</h3><p>Usually keyword extraction is heavily based upon term frequency to identify important words within a document. But the problemis this approach may overlook some of the terms that are significant even when they are not that much frequent in an index. By using its algorithmic powers the significant_text aggregation will be able to identify the terms that occur more often together than random chance.</p><h3>Real time example:</h3><p>Let us take a look into a real-world scenario where significant_text can help us. The requirement is to extract key concepts from a collection of fruit descriptions. Let us consider that we have an index that contains the details of fruits. Our requirement is to identify terms that are not only frequent but are also statistically significant that shows their importance.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;fruit_description_keywords&quot;: {<br>      &quot;significant_text&quot;: {<br>        &quot;field&quot;: &quot;description&quot;,<br>        &quot;size&quot;: 5<br>      }<br>    }<br>  }<br>}</pre><p>In this example, we apply the significant_text aggregation to the description field and then we are aiming to extract five statistically significant fruit keywords whcih will contain the essence of the fruit descriptions.</p><h3>Combining with Other Aggregations</h3><p>As I have demonstrated in previous episodes, the true power of significant_text(any queries or aggregations) emerges when we combine it with other aggregations. We can use it here to gain a detailed understanding of fruit-related data. here integrating significant_text with the terms aggregation to not only extract significant fruit keywords but also categorize them based on their occurrence in different fruit categories.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;category_keywords&quot;: {<br>      &quot;terms&quot;: {<br>        &quot;field&quot;: &quot;category.keyword&quot;<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;significant_keywords&quot;: {<br>          &quot;significant_text&quot;: {<br>            &quot;field&quot;: &quot;content&quot;,<br>            &quot;size&quot;: 5<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In this query, we are using conventional pattern by combining significant_text with the terms aggregation. Now, we will be able to get both significant fruit keywords(terms query) annd also categorize them based on their prevalence in different fruit categories(significant_text).</p><p>The variedness of significant_text makes it very useful across various types of usecases or industries. In legal documents, we can use it to identify legally significant terms. In customer review use cases, we can use it to pinpoint crucial sentiments and so on.</p><p>For more details on significant_text aggregation, the official Elastic documentation can help you a lot: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significanttext-aggregation.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significanttext-aggregation.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdbf16ea3d14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Custom Aggregations using bucket_script and bucket_selector…]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-custom-aggregations-using-bucket-script-and-bucket-selector-342bb9bd5050?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/342bb9bd5050</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 24 Jan 2024 13:20:37 GMT</pubDate>
            <atom:updated>2024-01-24T13:22:22.673Z</atom:updated>
            <content:encoded><![CDATA[<h3>Decoding Elasticsearch Query DSL: Custom Aggregations using bucket_script and bucket_selector Pipelines</h3><p>Welcome to the twenty seventh episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Let us take a look into two important custom aggregration tools called bucket_script and bucket_selector today. I am calling them as custom aggregation since they can be customized a lot unlike the traditional aggregation models. The can help us to define custom logic for aggregating, filtering, and transforming our data. Since this is a progressive series and if you are new to Elasticsearch DSL or to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>What is bucket_script:</h3><p>The bucket_script pipeline is a important tool for creating custom aggregations based on maths and calculations. It enables us to define complex logic that combines and then also transforms the metrics as per our logic and will help to create a new level of flexibility.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;custom_aggregation&quot;: {<br>      &quot;bucket_script&quot;: {<br>        &quot;buckets_path&quot;: {<br>          &quot;totalApples&quot;: &quot;sum_apples&quot;,<br>          &quot;averageBananas&quot;: &quot;avg_bananas&quot;<br>        },<br>        &quot;script&quot;: &quot;params.totalApples / params.averageBananas&quot;,<br>        &quot;gap_policy&quot;: &quot;insert_zeros&quot;<br>      }<br>    }<br>  }<br>}</pre><p>If you are thinking, why cant we do these sort of calculations with traditional metrics aggregations itself, the answer is here. The traditional metrics aggregations can only provide a already defined set of calculations such as sum, average, min, max, and more. But we can use the bucket_script and bucket_selector aggregations to define and execute custom calculations and filters.</p><p>In the above example, we are creating a custom aggregation by dividing the total apples (sum_apples) by the average bananas (avg_bananas) which is not possible in the traditional aggregation models. They would have only let us to either do the sum or the average and not both.</p><h3>What is bucket_selector:</h3><p>The bucket_selector aggregation on the other hand allows users to filter data based on specific conditions. Like we can use it to narrow down the dataset for more focused analysis. This is very useful when we want to include or exclude buckets based on our defined criteria or as per customer requirements.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;filtered_aggregation&quot;: {<br>      &quot;terms&quot;: {<br>        &quot;field&quot;: &quot;fruit_type.keyword&quot;<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;total_apples&quot;: {<br>          &quot;sum&quot;: {<br>            &quot;field&quot;: &quot;apples_count&quot;<br>          }<br>        },<br>        &quot;filter_high_apples&quot;: {<br>          &quot;bucket_selector&quot;: {<br>            &quot;buckets_path&quot;: {<br>              &quot;totalApples&quot;: &quot;total_apples&quot;<br>            },<br>            &quot;script&quot;: &quot;params.totalApples &gt; 50&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>From the above query snippet, we are first trying to aggregate the apple data by fruit type and then we are trying to use the bucket_selector to filter out fruit types with total apple counts less than 50. This will allow us to do more refined analysis of higher-performing apple types.</p><h3>Combining bucket_script and bucket_selector:</h3><p>As we have seen throughout this series, any query or aggregation turns very powerful when we combine them based on our usecase. Let’s explore a scenario where we first try to calculate a custom metric using bucket_script and then filter the results based on metrics using bucket_selector.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;custom_workflow&quot;: {<br>      &quot;bucket_script&quot;: {<br>        &quot;buckets_path&quot;: {<br>          &quot;totalApples&quot;: &quot;sum_apples&quot;,<br>          &quot;averageBananas&quot;: &quot;avg_bananas&quot;<br>        },<br>        &quot;script&quot;: &quot;params.totalApples / params.averageBananas&quot;,<br>        &quot;gap_policy&quot;: &quot;insert_zeros&quot;,<br>        &quot;scripted_metric&quot;: {<br>          &quot;init_script&quot;: &quot;...&quot;,<br>          &quot;map_script&quot;: &quot;...&quot;,<br>          &quot;combine_script&quot;: &quot;...&quot;,<br>          &quot;reduce_script&quot;: &quot;...&quot;<br>        }<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;filtered_workflow&quot;: {<br>          &quot;bucket_selector&quot;: {<br>            &quot;buckets_path&quot;: {<br>              &quot;customMetric&quot;: &quot;custom_workflow&quot;<br>            },<br>            &quot;script&quot;: &quot;params.customMetric &gt; 0.5&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In this little complex example, we are first creating a custom metric using bucket_script and then we are filtering the results based on that metric using bucket_selector.</p><h3>Time series analysis:</h3><p>Alright let us try to make use of a practical example where these custom aggregations are demonstrated better: time series analysis. Consider a scenario where we want to identify months with a significant increase in sales compared to the previous month.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;monthly_apple_sales&quot;: {<br>      &quot;date_histogram&quot;: {<br>        &quot;field&quot;: &quot;timestamp&quot;,<br>        &quot;interval&quot;: &quot;month&quot;<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;monthly_total_apple_sales&quot;: {<br>          &quot;sum&quot;: {<br>            &quot;field&quot;: &quot;apples_count&quot;<br>          }<br>        },<br>        &quot;custom_increase&quot;: {<br>          &quot;bucket_script&quot;: {<br>            &quot;buckets_path&quot;: {<br>              &quot;prevAppleSales&quot;: &quot;monthly_total_apple_sales&gt;prev&quot;<br>            },<br>            &quot;script&quot;: &quot;params.prevAppleSales / params.prevAppleSales &gt; 1.5&quot;,<br>            &quot;gap_policy&quot;: &quot;insert_zeros&quot;<br>          }<br>        },<br>        &quot;filtered_months&quot;: {<br>          &quot;bucket_selector&quot;: {<br>            &quot;buckets_path&quot;: {<br>              &quot;increaseFlag&quot;: &quot;custom_increase&quot;<br>            },<br>            &quot;script&quot;: &quot;params.increaseFlag == true&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>So what are we trying to do in this example is,</p><ul><li>we are aggregating the apple sales data on a monthly basis,</li><li>then calculate a custom metric using bucket_script to identify months with an apple sales increase of more than 50%</li><li>then filter the results using bucket_selector to focus on significant increases.</li></ul><p>I understand it may sound confusing at the first glance, but it will get better and better with each iterations or tries. For more details on these two custom aggregation models, the official Elastic documentation can help you a lot:</p><p><strong>Bucket_script:</strong> <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html</a></p><p><strong>Bucket_selector:</strong> <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-selector-aggregation.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-selector-aggregation.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=342bb9bd5050" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Navigating Statistical Insights with extended_stats Aggregation]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-navigating-statistical-insights-with-extended-stats-aggregation-ed63b00ecd53?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed63b00ecd53</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Tue, 23 Jan 2024 17:28:57 GMT</pubDate>
            <atom:updated>2024-01-23T17:39:55.185Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the twenty sixth episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, let us try to learn what is extended_stats aggregation and how can leverage it for our statistical analytics. Since this is a progressive series and if you are new to Elasticsearch DSL or to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>The Robust extended_stats Aggregation</h3><p>The extended_stats aggregation plays a very important role in sophisticated statistical analytics area or usecases. This aggregation jumps up from the limitations of conventional/usual metrics and it provides a wide variety of statistical measures. Let us first take a look at the syntact of extended_stats:</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;advanced_stats&quot;: {<br>      &quot;extended_stats&quot;: {<br>        &quot;field&quot;: &quot;numeric_field&quot;<br>      }<br>    }<br>  }<br>}</pre><p>In this code snippet, the extended_stats aggregation is applied to a numeric field which will help us to generate a detailed set of metrics such as count, mean, min, max, sum, variance, standard deviation, and more.</p><p>But before moving further, lets take a quick look at what are these metrices for the betterment of wider audience.</p><h4>Count</h4><p>The total count of values within the index or the dataset.</p><h4>Mean</h4><p>The arithmetic average which is calculated by summing all values and the dividing by the count(or number of values).</p><h4>Minimum and Maximum</h4><p>The smallest and largest values in the index whcih can provide us the range of the values.</p><h4>Sum</h4><p>It is the the summation or total of all values within the index.</p><h3>Combining Aggregations with extended_stats</h3><p>In the below example, we are using a terms aggregation to first categorize the data, and then the nested extended_stats aggregation to analyze statistical measures in each of those categories.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;category_stats&quot;: {<br>      &quot;terms&quot;: {<br>        &quot;field&quot;: &quot;category.keyword&quot;<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;numeric_stats&quot;: {<br>          &quot;extended_stats&quot;: {<br>            &quot;field&quot;: &quot;numeric_field&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><h3>Time-Aware Statistical Analysis:</h3><p>In the last episode of our series, we saw in detail about the date_histogram aggregation. Combining the data histogram with the extended stats aggregation will help to produce awesome insights that can be heavily leveraged in the many use cases.</p><p>For example, these two aggregations in combo can help a lot in understanding what is the resources of a particular instance, over what period of time and what is the range of them between different instances and what particular instance’s usage deviates from the normal mean or average usage.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;monthly_stats&quot;: {<br>      &quot;date_histogram&quot;: {<br>        &quot;field&quot;: &quot;timestamp&quot;,<br>        &quot;interval&quot;: &quot;month&quot;<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;numeric_stats&quot;: {<br>          &quot;extended_stats&quot;: {<br>            &quot;field&quot;: &quot;numeric_field&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>This query uses a date_histogram aggregation to segment data into monthly intervals, while the nested extended_stats aggregation computes statistical measures for the numeric field within each month.</p><p>The extended_stats aggregation in Elasticsearch not just provide metrics but it can also open up heavily detailed insights just like the example we saw above with the server instances and their resource usages.</p><p>For more detailed documentation on the extended_stats aggregation, the official Elastic documentation can help you a lot: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-extendedstats-aggregation.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-extendedstats-aggregation.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed63b00ecd53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Exploring the top_hits aggregation for retrieving top matching…]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-exploring-the-top-hits-aggregation-for-retrieving-top-matching-09f23723a6a3?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/09f23723a6a3</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Tue, 23 Jan 2024 15:38:12 GMT</pubDate>
            <atom:updated>2024-01-23T15:38:12.117Z</atom:updated>
            <content:encoded><![CDATA[<h3>Decoding Elasticsearch Query DSL: Exploring the top_hits aggregation for retrieving top matching documents</h3><p>Welcome to the twenty fifth episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, lets us see the capabilities of the top_hits aggregation which is a powerful tool for retrieving the top matching documents in any index. Since this is a progressive series and if you are new to Elasticsearch DSL or to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>The top_hits Aggregation</h3><p>The top_hits aggregation is a very useful tool in Elasticsearch, which allows us to retrieve the most relevant documents based on certain search criteria.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;top_documents&quot;: {<br>      &quot;top_hits&quot;: {<br>        &quot;size&quot;: 5,<br>        &quot;sort&quot;: [<br>          {<br>            &quot;timestamp&quot;: {<br>              &quot;order&quot;: &quot;desc&quot;<br>            }<br>          }<br>        ]<br>      }<br>    }<br>  }<br>}</pre><p>In this query, we can see that the top_hits aggregation is used, which is trying to fetch the top 5 documents. The &quot;sort&quot; parameter further better look by ordering the results based on the &quot;timestamp&quot; field in descending order.</p><h3>Combining top_hits with other Aggregations</h3><p>One of the strengths of Elasticsearch is its ability to easily integrate with other types of aggregations or term queries. Let’s consider a scenario where we want to find the top documents within each category. Nesting the top_hits aggregation within a terms aggregation can help us to accomplish this task.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;category_hits&quot;: {<br>      &quot;terms&quot;: {<br>        &quot;field&quot;: &quot;category.keyword&quot;,<br>        &quot;size&quot;: 10<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;top_category_documents&quot;: {<br>          &quot;top_hits&quot;: {<br>            &quot;size&quot;: 1,<br>            &quot;sort&quot;: [<br>              {<br>                &quot;score&quot;: {<br>                  &quot;order&quot;: &quot;desc&quot;<br>                }<br>              }<br>            ]<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>This query helps us in not only breaking down the data by category but also in fetching the top document within each category using the nested top_hits aggregation.</p><h3>Using Filter Criteria</h3><p>The true power of top_hits comes out when we introduce the filter criteria. Lets say we want to retrieve the top documents only from a specific timeframe. Here&#39;s how we can achieve that:</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;filtered_top_documents&quot;: {<br>      &quot;top_hits&quot;: {<br>        &quot;size&quot;: 3,<br>        &quot;sort&quot;: [<br>          {<br>            &quot;timestamp&quot;: {<br>              &quot;order&quot;: &quot;desc&quot;<br>            }<br>          }<br>        ],<br>        &quot;query&quot;: {<br>          &quot;range&quot;: {<br>            &quot;timestamp&quot;: {<br>              &quot;gte&quot;: &quot;2024-01-01&quot;,<br>              &quot;lte&quot;: &quot;2024-01-23&quot;<br>            }<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In this example, we are using a range query within the top_hits aggregation to fetch the top 3 documents within the specified date range.</p><h3>Navigating Nested Fields</h3><p>Elasticsearch supports nested fields, and the top_hits aggregation can be used to retrieve top documents within these nested structures. Consider a scenario where documents have an array of comments, and we want the top comment within each document:</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;top_comments&quot;: {<br>      &quot;top_hits&quot;: {<br>        &quot;size&quot;: 1,<br>        &quot;sort&quot;: [<br>          {<br>            &quot;comments.timestamp&quot;: {<br>              &quot;order&quot;: &quot;desc&quot;<br>            }<br>          }<br>        ]<br>      }<br>    }<br>  }<br>}</pre><p>Here, the top_hits aggregation is applied to the nested field &quot;comments,&quot; which will help us to fetch the top comment within each document.</p><p>The top_hits aggregation in Elasticsearch is a useful tool for retrieving the most relevant documents based on specific criteria. Whether we have to fetch the top documents globally, within categories orwe need to navigate the nested fields, the top_hits offers flexibility and precision.</p><p>Official documentation from Elastic can help us more to understand further details about the top_hits aggregation in detail: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09f23723a6a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Leveraging the date_histogram Aggregation for Date-Based Analysis]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-leveraging-the-date-histogram-aggregation-for-date-based-analysis-0368f30153f1?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/0368f30153f1</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Tue, 23 Jan 2024 14:56:03 GMT</pubDate>
            <atom:updated>2024-01-23T14:56:47.328Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the twenty fourth episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, let us try to learn what is date_histogram and how it helps us to transform the raw time data into useful insights. Since this is a progressive series and if you are new to Elasticsearch DSL and to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>Syntax of date_histogram</h3><p>Let us imagine that we have an index that contains the collection of reccords/events and each of those records tagged with a timestamp(field), and our requirement is to find out and use the pattern that is present within individual timeframs within this index. This is exactly where date_histogram aggregation can help us a lot in dissecting time-based information.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;time_distribution&quot;: {<br>      &quot;date_histogram&quot;: {<br>        &quot;field&quot;: &quot;timestamp&quot;,<br>        &quot;interval&quot;: &quot;day&quot;<br>      }<br>    }<br>  }<br>}</pre><p>In this query snip, we can see that it tells Elasticsearch to group the data based on the “timestamp” field and then break it down into daily intervals. This will help us in creating a detailed account of how events got created for each day.</p><h3>Using with aggregations for Deeper Insights</h3><p>To go deeper in our analysis, we can combine date_histogram with other aggregations. Lets say we want not only a monthly breakdown but also a look into which categories actually dominate each month.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;monthly_distribution&quot;: {<br>      &quot;date_histogram&quot;: {<br>        &quot;field&quot;: &quot;timestamp&quot;,<br>        &quot;interval&quot;: &quot;month&quot;<br>      },<br>      &quot;aggs&quot;: {<br>        &quot;category_distribution&quot;: {<br>          &quot;terms&quot;: {<br>            &quot;field&quot;: &quot;category.keyword&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>This query easily integrates the date_histogram aggregation with the terms aggregation, providing a very different perspective on how events unfold monthly and at the same time, it also provides another insight into the dominant categories during each period.</p><h3>Using different Country timezones</h3><p>One common issue in creating date-based analysis is neglecting the impact or importance of time zones. When dealing with data originating from different geographical regions or countries, specifying the time zone becomes very very important. By using the time_zone parameter, we can easliy achieve the same.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;daily_distribution&quot;: {<br>      &quot;date_histogram&quot;: {<br>        &quot;field&quot;: &quot;timestamp&quot;,<br>        &quot;interval&quot;: &quot;day&quot;,<br>        &quot;time_zone&quot;: &quot;America/New_York&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Incorporating the “time_zone” parameter ensures a usage of standardized time while doing the calculations in the backend by taking into account about the data from various time zones.</p><h3>Time Formats</h3><p>Elasticsearch actually supports various time units for the date_histogram aggregation. Whether we want to use timeframes as years, months, weeks, days, hours, minutes, or seconds, there is a time format to suit analytical needs.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;hourly_distribution&quot;: {<br>      &quot;date_histogram&quot;: {<br>        &quot;field&quot;: &quot;timestamp&quot;,<br>        &quot;interval&quot;: &quot;hour&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Adjusting the “interval” parameter to “hour” allows us to take a look into events on a small, hourly basis, showing us the imporant minute details.</p><p>Overall, the date_histogram aggregation becomes an important asset in Elasticsearch by helping users to extract detailed insights from timebased datasets. Official documentation from Elastic can help us more to understand further details about the date_histogram aggregation in detail: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0368f30153f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Unleashing the Power of significant_terms Aggregation in…]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-unleashing-the-power-of-significant-terms-aggregation-in-e54fdb354eaf?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/e54fdb354eaf</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 17:34:23 GMT</pubDate>
            <atom:updated>2024-01-03T17:34:23.929Z</atom:updated>
            <content:encoded><![CDATA[<h3>Decoding Elasticsearch Query DSL: Unleashing the Power of significant_terms Aggregation in Elasticsearch</h3><p>Welcome to the twenty third episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, we will look into what is significant_terms and how we can use it to extract valuable insights from our Elasticsearch. Since this is a progressive series and if you are new to Elasticsearch DSL and to this series, please feel free to go back to my older episodes to get the full understanding.</p><p><strong>Decoding the significant_terms Aggregation</strong></p><p>The significant_terms aggregation in Elasticsearch is a very important component which is offering us with a detailed mechanism for finding terms with accurate significance in a context. Unlike the standard aggregation types, the significant_terms aggregation runs through the noise and finds the terms that truly matters to the users.</p><p><strong>Essential Syntax and Parameters</strong></p><p>Let us take a look the basic syntax of the significant_terms aggregation:</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;interesting_terms&quot;: {<br>      &quot;significant_terms&quot;: {<br>        &quot;field&quot;: &quot;fieldA&quot;,<br>        &quot;size&quot;: 10<br>      }<br>    }<br>  }<br>}</pre><ul><li>field: This should be the field from where the significant terms will be extracted from.</li><li>size: This size provides us the number of significant terms that needs to be retrieved in the end result.</li></ul><p><strong>Real-World Application</strong></p><p>Imagine a scenario where a social media platform like Facebook or X.com, tries to find the current trending topics by going through all its user’s posts. The significant_terms aggregation can play a major part in such use cases.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;trending_terms&quot;: {<br>      &quot;significant_terms&quot;: {<br>        &quot;field&quot;: &quot;post_content&quot;,<br>        &quot;size&quot;: 7<br>      }<br>    }<br>  }<br>}</pre><p>In the above example, the aggregation analyses the post content, and the finds and comes up with the top 7 significant terms. This information becomes very useful in findinng out the emerging trends in the platform by using the user’s posts.</p><p><strong>Managing Diverse Fields</strong></p><p>The flexibility of the significant_terms aggregation really comes out when we are applying it for multiple fields. Let us assume that we plan to identify terms of significance across both user profiles and their posted content:</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;cross_field_terms&quot;: {<br>      &quot;significant_terms&quot;: {<br>        &quot;field&quot;: [&quot;user_profile&quot;, &quot;post_content&quot;],<br>        &quot;size&quot;: 8<br>      }<br>    }<br>  }<br>}</pre><p>Here, the aggregation easily/smoothly goes through both user profiles and post content fields, and then presents a consolidated set of significant terms.</p><p><strong>Statistical Significance for Precision</strong></p><p>To keep up precision in the end result, the significant_terms aggregation actually uses statistical significance tests. Parameters like chi_square, gnd, and mutual_information helps the users to fine-tune the analysis based on distinct methodologies.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;statistical_terms&quot;: {<br>      &quot;significant_terms&quot;: {<br>        &quot;field&quot;: &quot;fieldA&quot;,<br>        &quot;size&quot;: 7,<br>        &quot;chi_square&quot;: {},<br>        &quot;gnd&quot;: {},<br>        &quot;mutual_information&quot;: {}<br>      }<br>    }<br>  }<br>}</pre><p>In this instance, we have written the significant terms aggregation by using three different statistical methods, to provide a detailed perspective on term significance.</p><p><strong>Adapting to Real-World Scenarios</strong></p><p>In practical scenarios, we may often have requirements to filter out specific terms or customize the analysis based on unique requirements. The filter_duplicate_text and filter parameters inside the significant_terms aggregation help the users to customise the analysis to their datasets.</p><pre>{<br>  &quot;aggs&quot;: {<br>    &quot;custom_terms&quot;: {<br>      &quot;significant_terms&quot;: {<br>        &quot;field&quot;: &quot;fieldA&quot;,<br>        &quot;size&quot;: 6,<br>        &quot;filter_duplicate_text&quot;: true,<br>        &quot;filter&quot;: {<br>          &quot;range&quot;: {<br>            &quot;date&quot;: {<br>              &quot;gte&quot;: &quot;2024-01-03&quot;<br>            }<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In this configuration, the aggregation filters duplicate text and concentrates on terms emerging after a specific date(greater than Jan 03, 2024) in this example.</p><p>While we have seen the overview about significant_terms aggregation, Official documentation from Elastic can help us more to understand further details about the significant_terms query: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html">Significant terms aggregation | Elasticsearch Guide [8.11] | Elastic</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e54fdb354eaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Exploring the more_like_this Query in Elasticsearch]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-exploring-the-more-like-this-query-in-elasticsearch-04f9e6a9184f?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/04f9e6a9184f</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 15:47:01 GMT</pubDate>
            <atom:updated>2024-01-03T15:48:24.631Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the twenty second episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, we will try to learn about more_like_this query which can help us to create similarity-based search systems. Since this is a progressive series and if you are new to Elasticsearch DSL and to this series, please feel free to go back to my older episodes to get the full understanding.</p><p><strong>Understanding the more_like_this Query</strong></p><p>The more_like_this query is an important feature for content recommendation systems(Example: Youtube, Amazon suggestion mechanisms) and similarity-based searches. It operates by finding out the relevant documents based on the content of a given document. Instead of relying completelyy on the keywords, this query tries and goes through the document&#39;s actual content, creating a specific approach to similarity.</p><p><strong>Basic Syntax and Parameters</strong></p><p>Let’s take a look at the into the basic syntax of more_like_this query:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;more_like_this&quot;: {<br>      &quot;fields&quot;: [&quot;field1&quot;, &quot;field2&quot;],<br>      &quot;like&quot;: {<br>        &quot;_id&quot;: &quot;document_id&quot;,<br>        &quot;index&quot;: &quot;index&quot;,<br>        &quot;routing&quot;: &quot;routing&quot;<br>      },<br>      &quot;min_term_freq&quot;: 1,<br>      &quot;max_query_terms&quot;: 25<br>    }<br>  }<br>}</pre><ul><li>fields: These are the two fields for which we need to check thee similarity comparison.</li><li>like: Define the document to compare with, either by ID or index.</li><li>min_term_freq: This is the minimum term frequency for a term to be considered.</li><li>max_query_terms: This parameter will control the number of query terms extracted from the source document.</li></ul><p><strong>Practical Application</strong></p><p>Consider an e-commerce platform like Amazon or Flipkart searching for products similar to a given one. The more_like_this query becomes very much useful in this scenario like mentioned above. Let&#39;s look into a practical example:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;more_like_this&quot;: {<br>      &quot;fields&quot;: [&quot;product_name&quot;, &quot;description&quot;],<br>      &quot;like&quot;: {<br>        &quot;_id&quot;: &quot;product_id&quot;,<br>        &quot;index&quot;: &quot;products&quot;,<br>        &quot;routing&quot;: &quot;user_id&quot;<br>      },<br>      &quot;min_term_freq&quot;: 2,<br>      &quot;max_query_terms&quot;: 20<br>    }<br>  }<br>}</pre><p>Here, we search for products that are similar to the one with the specified product_id. The fields that aree considered for similarity here are the product name and description fields. Adjusting parameters like min_term_freq and max_query_termsiwill help us to fine-tune the query based on specific requirements.</p><p><strong>Handling Multiple Documents</strong></p><p>The more_like_this query can even help us in creating suggestion systems while dealing with multiple documents. Consider the following scenario:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;more_like_this&quot;: {<br>      &quot;fields&quot;: [&quot;content&quot;],<br>      &quot;like&quot;: [<br>        { &quot;_index&quot;: &quot;articles&quot;, &quot;_id&quot;: &quot;1&quot; },<br>        { &quot;_index&quot;: &quot;articles&quot;, &quot;_id&quot;: &quot;2&quot; },<br>        { &quot;_index&quot;: &quot;articles&quot;, &quot;_id&quot;: &quot;3&quot; }<br>      ],<br>      &quot;min_term_freq&quot;: 1,<br>      &quot;max_query_terms&quot;: 15<br>    }<br>  }<br>}</pre><p>This query retrieves documents similar to the specified set, allowing for broader exploration of similarity within a corpus.</p><p><strong>Fine-Tuning for Precision</strong></p><p>To enhance precision, the more_like_this query offers parameters like stop_words and analyzer. Utilizing these parameters refines the comparison process, ensuring more accurate and relevant results.</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;more_like_this&quot;: {<br>      &quot;fields&quot;: [&quot;content&quot;],<br>      &quot;like&quot;: {<br>        &quot;_id&quot;: &quot;document_id&quot;,<br>        &quot;index&quot;: &quot;index&quot;,<br>        &quot;routing&quot;: &quot;routing&quot;<br>      },<br>      &quot;min_term_freq&quot;: 1,<br>      &quot;max_query_terms&quot;: 15,<br>      &quot;stop_words&quot;: [&quot;common_word1&quot;, &quot;common_word2&quot;],<br>      &quot;analyzer&quot;: &quot;custom_analyzer&quot;<br>    }<br>  }<br>}</pre><p>Here, we exclude common words using stop_words and specify a custom analyzer to control how the text is processed. The values that we have provided inside the stop_words field will not be processed for the similarity. This is help a lot in creating much more precise suggestion systems.</p><p>Official documentation from Elastic can help us more to understand further details about the more_lik_this query: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html">More like this query | Elasticsearch Guide [8.11] | Elastic</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04f9e6a9184f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Leveraging the geo_bounding_box query for bounding box filtering]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-leveraging-the-geo-bounding-box-query-for-bounding-box-filtering-2ed974dc0ee3?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ed974dc0ee3</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 13:59:51 GMT</pubDate>
            <atom:updated>2024-01-03T13:59:51.536Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the twenty first episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, let us try to learn what is geo_bounding_box query and how we can add other queries over it to achieve better spatial bounding box filtering. Since this is a progressive series and if you are new to Elasticsearch DSL and to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>Leveraging the geo_bounding_box Query for Bounding Box Filtering</h3><p>Alright lets jump straight into the topic. The geo_bounding_box query is an important tool for bounding box filtering in location-based searches. In this article, let us try to understand what is geo_bounding_box query and let us also see some practical examples to understand it better to empower us in spatial filtering within Elasticsearch.</p><h3>The Significance of geo_bounding_box Query</h3><p>When it comes to spatial queries and filtering, the geo_bounding_box query takes center stage. It allows us to filter documents based on their presence within a specified area. For example, it can be a rectangular area defined by two sets of latitude and longitude coordinates.</p><h3>Constructing a Basic geo_bounding_box Query</h3><p>Let’s start with a very basic example. Assume that we have an index called “Properties” index which is containing documents with location information. To find properties within a bounding box defined by two points (bottom-left: xx.xxxx, -yyy.yyyy, top-right: aa.aaaa, -bbb.bbbb), we can make use of the geo_bounding_box query like the below:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;geo_bounding_box&quot;: {<br>      &quot;location&quot;: {<br>        &quot;bottom_left&quot;: {<br>          &quot;lat&quot;: xx.xxxx8,<br>          &quot;lon&quot;: -yyy.yyyy<br>        },<br>        &quot;top_right&quot;: {<br>          &quot;lat&quot;: aa.aaaa,<br>          &quot;lon&quot;: -bbb.bbbb<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>This query will help us in retrieving the documents from the “Properties” index that fall within the bounding box.</p><h3>Enhancing Filtering with a Term Query</h3><p>We can further refine our spatial queries by combining the geo_bounding_box query with different queries. In this example, we are trying to integrate a term query for filtering properties of a specific type within the bounding box:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: [<br>        {<br>          &quot;geo_bounding_box&quot;: {<br>            &quot;location&quot;: {<br>              &quot;bottom_left&quot;: {<br>                &quot;lat&quot;: xx.xxxx,<br>                &quot;lon&quot;: -yyy.yyyy<br>              },<br>              &quot;top_right&quot;: {<br>                &quot;lat&quot;: aa.aaaa,<br>                &quot;lon&quot;: -bbb.bbbb<br>              }<br>            }<br>          }<br>        },<br>        {<br>          &quot;term&quot;: {<br>            &quot;property_type&quot;: &quot;apartment&quot;<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>In this query, we are trying to search for apartments within a bounding box defined by specified coordinates.</p><h3>Combining with Range Query</h3><p>lets try to understand it more by using a different example. If we need to find properties within a certain price range within the same bounding box, we can combine it with a range query. Below is an example of how we ccan combine the geo_bounding_box query with a range query:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: [<br>        {<br>          &quot;geo_bounding_box&quot;: {<br>            &quot;location&quot;: {<br>              &quot;bottom_left&quot;: {<br>                &quot;lat&quot;: xx.xxxx,<br>                &quot;lon&quot;: -yyy.yyyy<br>              },<br>              &quot;top_right&quot;: {<br>                &quot;lat&quot;: aa.aaaa,<br>                &quot;lon&quot;: -bbb.bbbb<br>              }<br>            }<br>          }<br>        },<br>        {<br>          &quot;range&quot;: {<br>            &quot;price&quot;: {<br>              &quot;gte&quot;: 1000,<br>              &quot;lte&quot;: 3000<br>            }<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>This query retrieves properties within the specified bounding box with prices between 1000 US Dollars and 3000 US Dollars.</p><h3>Practical Use Cases</h3><h3>Real Estate Search</h3><p>In real estate applications, where location plays a crucial role, the geo_bounding_box query proves very useful and important. Users can easily filter properties based on a defined geographical area, facilitating a seamless property search experience.</p><h3>Geographic Data Analysis</h3><p>For applications involving geographic data analysis, bounding box filtering is a very good option. Researchers and analysts can focus their studies on specific regions defined by coordinates, ensuring targeted and relevant results.</p><p>In conclusion, the geo_bounding_box query in Elasticsearch Query DSL empowers us to perform efficient bounding box filtering in spatial searches. Official documentation from Elastic can help us more to understand further details about this query: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-bounding-box-query.html">Geo-bounding box query | Elasticsearch Guide [8.11] | Elastic</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ed974dc0ee3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Understanding the geo_distance Query for Location-Based Searching]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-understanding-the-geo-distance-query-for-location-based-searching-04e109daba9c?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/04e109daba9c</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 13:08:45 GMT</pubDate>
            <atom:updated>2024-01-03T13:08:45.923Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the nineteenth episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, we are going to talk about what is geo_distance query and how it can help us in performing llocation based searches and how we can use it as a combo along with other types of queries for better user experience. Since this is a progressive series and if you are new to Elasticsearch DSL and to this series, please feel free to go back to my older episodes to get the full understanding.</p><p>The geo_distance query is a very important tool for conducting any location-based searches within Elasticsearch. I am going to try and provide a comprehensive understanding of the geo_distance query in this article, by using practical examples and insights to enhance your understanding in location-based querying.</p><h3>The Essence of Geo_distance Query</h3><p>Geo-location data is a very very important thing in modern applications, playing tons of different roles based on what is the field it is used in. Elasticsearch provides powerful features to use its power to the full by using this query. The geo_distance query helps in searching for documents based on their geo proximity to a specified geographical point and so making it an important component in location-based applications.</p><h3>Crafting a Basic geo_distance Query</h3><p>Let us look into a basicc example to grasp the concept of the geo_distance query. Let us assume that we have a “Locations” index which is containing documents with latitude and longitude information. To find locations within a 10-kilometer radius of a given point (e.g., latitude xx.xxxx, longitude -xx.xxxx), we can create a simple geo_distance query as follows:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;geo_distance&quot;: {<br>      &quot;distance&quot;: &quot;10km&quot;,<br>      &quot;location&quot;: {<br>        &quot;lat&quot;: xx.xxxx,<br>        &quot;lon&quot;: -xx.xxxx<br>      }<br>    }<br>  }<br>}</pre><p>This query will help us in retrievinng documents from the “Locations” index that actually falls within the specified distance of the given coordinates.</p><h3>Using Filter Context for Geo-distance Queries</h3><p>In scenarios where we need to focus on filtering rather than scoring, employing the geo_distance query within a filter context can help to optimize the query performance.</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: {<br>        &quot;geo_distance&quot;: {<br>          &quot;distance&quot;: &quot;5km&quot;,<br>          &quot;location&quot;: {<br>            &quot;lat&quot;: xx.xxxx,<br>            &quot;lon&quot;: -xx.xxxx<br>          }<br>        }<br>      },<br>      &quot;must&quot;: {<br>        &quot;match&quot;: {<br>          &quot;category&quot;: &quot;restaurant&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>This query retrieves documents that are both within a 5-kilometer radius of the specified coordinates and belong to the “restaurant” category. The record will basically have to follow both the conditions of 5 km radius and category as Restaurant for it to be picked by the query</p><h3>Combining geo_distance with Additional Queries</h3><p>To enhance the precision of our location-based searches, we can actually combine the geo_distance query with other queries. For example, combining a term query for specific attributes can further refine our results:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;filter&quot;: [<br>        {<br>          &quot;geo_distance&quot;: {<br>            &quot;distance&quot;: &quot;10km&quot;,<br>            &quot;location&quot;: {<br>              &quot;lat&quot;: xx.xxxx,<br>              &quot;lon&quot;: -xx.xxxx<br>            }<br>          }<br>        },<br>        {<br>          &quot;term&quot;: {<br>            &quot;property_type&quot;: &quot;apartment&quot;<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>In this query, we’re searching for apartments within a 10-kilometer radius of a specified location. Again, same like the above example, the record will basically have to follow both the conditions of 10 km radius and property_type as apartment for it to be picked by the query.</p><h3>Practical Use Cases</h3><h3>Location-Based Recommendations</h3><p>In applications where user recommendations are based on geographical proximity/distance, the geo_distance query proves very useful and effective as well. For example, finding nearby restaurants or services becomes very easy by specifying a user’s location.</p><h3>Event Location Filtering</h3><p>Consider a scenario where events are stored in an index with associated locations. Using geo_distance queries, filtering events within a certain radius of a user’s location enhances the event discovery experience. For example, what are the loads imparted on a specific data center, what are the restaurants that pulls what type of customers at what locations etc.</p><p>So basically, the geo_distance query in Elasticsearch Query DSL helps us to perform accurate and efficient location-based searches. Official documentation from Elastic can help us more to understand further details about this query: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html">Geo-distance query | Elasticsearch Guide [8.11] | Elastic</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04e109daba9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Elasticsearch Query DSL: Leveraging has_child and has_parent Queries in Elasticsearch for…]]></title>
            <link>https://medium.com/@jeevanandham.selvaraj42/decoding-elasticsearch-query-dsl-leveraging-has-child-and-has-parent-queries-in-elasticsearch-for-18f2b930e4c8?source=rss-2dc78bec69de------2</link>
            <guid isPermaLink="false">https://medium.com/p/18f2b930e4c8</guid>
            <dc:creator><![CDATA[Jeevanandham Selvaraj]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 09:30:58 GMT</pubDate>
            <atom:updated>2024-01-03T09:30:58.178Z</atom:updated>
            <content:encoded><![CDATA[<h3>Decoding Elasticsearch Query DSL: Leveraging has_child and has_parent Queries in Elasticsearch for Parent-Child Document Searching</h3><p>Welcome to the eighteenth episode of our Decoding Elasticsearch query DSL series. I am going to try and explain the DSL with progressively increasing levels of complex concepts to try and provide you a deeper understanding by the end of this series. Today, lets dive into has_child query and has_parent query which will help us to combine 2 different documents in 2 different indices thus enabling us to associate them with a relationship. Doing so will help us a lot while we are searching for documents that are related. Since this is a progressive series and if you are new to Elasticsearch DSL and to this series, please feel free to go back to my older episodes to get the full understanding.</p><h3>Understanding Parent-Child Relationships</h3><p>Parent-child relationships in Elasticsearch involve associating documents in one index as parents to documents in another index. This relationship is very important for scenarios where data organization requires a hierarchical structure and needed to be searched in a hierarchical way. The has_child and has_parent queries provide us with an option to navigate and search within these relationships easily and accurately.</p><h3>The has_child Query</h3><p>The has_child query allows us to search for parent documents based on the content of their associated child documents. This query is mainly useful when we would like to filter or score parents based on the properties of their children.</p><p>Syntax:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;has_child&quot;: {<br>      &quot;type&quot;: &quot;child_type&quot;,<br>      &quot;query&quot;: {<br>        &quot;match&quot;: {<br>          &quot;field&quot;: &quot;value&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>Let us consider, we have a parent-child relationship in our “Fruits” index, where each fruit document is a parent, and nutritional information of those fruits is stored as child documents. The has_child query can be used to find fruits with a specific nutritional value.</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;has_child&quot;: {<br>      &quot;type&quot;: &quot;nutritional_info&quot;,<br>      &quot;query&quot;: {<br>        &quot;range&quot;: {<br>          &quot;vitamin_c&quot;: {<br>            &quot;gte&quot;: 20<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In this example, we’re searching for fruits with child documents (nutritional information) where the vitamin C content is greater than or equal to 20.</p><h3>The has_parent Query</h3><p>Just like has_child query, but in an opposite way, the has_parent query enables us to search for child documents based on the characteristics of their parent documents. This is very useful when we are tying to filter or score child documents according to properties of their parent documents.</p><p>Syntax:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;has_parent&quot;: {<br>      &quot;parent_type&quot;: &quot;parent_type&quot;,<br>      &quot;query&quot;: {<br>        &quot;term&quot;: {<br>          &quot;field&quot;: &quot;value&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In the same example with “Fruits” index that has parent-child relationship like the one we used above, let&#39;s assume that each fruit document is a parent, and nutritional information is stored as child documents. The has_child query can be employed to find fruits with a specific nutritional value. Here’s an example:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;has_child&quot;: {<br>      &quot;type&quot;: &quot;nutritional_info&quot;,<br>      &quot;query&quot;: {<br>        &quot;range&quot;: {<br>          &quot;vitamin_c&quot;: {<br>            &quot;gte&quot;: 20<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In this example, we’re searching for fruits with child documents (nutritional information) where the vitamin C content is greater than or equal to 20.</p><p>So basically has_child and has_parent queries are just the reciprocal of each other.</p><h3>Realtime Use cases:</h3><h3>Hierarchical Data Modeling</h3><p>Let us consider a scenario where we are modeling hierarchical data such as organizational structures or product categories. The has_child and has_parent queries enable us to retrieve relevant information at various levels of the hierarchy, facilitating efficient data navigation.</p><h3>Content Filtering</h3><p>In a content management system, we might have parent documents representing articles and child documents representing comments. Utilizing has_child queries, we can be able to filter articles based on specific criteria within their comments and will be able to provide a refined search experience for the customers.</p><p>While has_child and has_parent queries offer powerful options for document searching that are related, we should always be careful about usual issues like mapping inconsistency between the indices, query performance issues for larger hierarchies etc..</p><p>To maximize the effectiveness of has_child and has_parent queries, we should make sure that we are always following below best practices.</p><ol><li>Index Settings: index settings should be configured to support efficient parent-child relationships. Tune settings like ‘index.mapping.single_type’ properly as per the use case.</li><li>Query Optimization: Query optimization techniques such as caching and filtering should be used to enhance search performance.</li></ol><p>So if this was a starter pack for understanding the has_child and has_parent queries, the official documentation from Elastic can help us more to understand further details about the same.</p><p>has_child query: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-has-child-query.html">Has child query | Elasticsearch Guide [8.11] | Elastic</a></p><p>has_parent query: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-has-parent-query.html">Has parent query | Elasticsearch Guide [8.11] | Elastic</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18f2b930e4c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>