<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[elasticsearch - Medium]]></title>
        <description><![CDATA[Your search, starts here - Medium]]></description>
        <link>https://medium.com/elasticsearch?source=rss----7d515a650486---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>elasticsearch - Medium</title>
            <link>https://medium.com/elasticsearch?source=rss----7d515a650486---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:22:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/elasticsearch" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Elasticsearch queries — part 03 — term level queries]]></title>
            <link>https://medium.com/elasticsearch/elasticsearch-queries-part-03-term-level-queries-2df2dbbe7477?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/2df2dbbe7477</guid>
            <category><![CDATA[elasticsearch-queries]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 26 Oct 2019 02:01:01 GMT</pubDate>
            <atom:updated>2019-10-26T02:01:01.630Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/e5ffe8fb3110/edit">Elasticsearch queries — part 03 — term level queries</a></h3><h4>Phase 03 — Elasticsearch Term queries — Blog 13</h4><p>Last blog we have seen a few important and most used full-text queries in the Elasticsearch world. We will have a look at some of the most useful and relevant <strong>term level</strong> queries in this blog.</p><h4>What are term level queries?</h4><p>Term level queries are those queries which are used to find the records/documents based on exact values in organised/structured data. <br>The examples of structured data are product codes, date ranges, pin codes, IPs etc.</p><p>What makes term query different from that of full-text queries is the fact that the search keywords (the text to be searched) will undergo the analyzing process and then gets queried. Where as in the term query execution, the search query keyword will go as it is. (NB: normalisation of the keywords can be done by the normalizer settings, but by default, the search query keyword goes as it is)<br>For example<br>If I am searching for “Arun Mohan” using a full text query, the search will be going as “Arun” and “Mohan”. <br>But when I am using term query for the same keyword as above, the search will be done as a single keyword without splitting as “Arun Mohan” itself.</p><h4>Sample data set</h4><p>The data set performed for this operation can be found <a href="https://github.com/2arunpmohan/elasticsearch_blogs/blob/master/blog-13/blog-13_sampleData.csv">here</a> . The steps to load this CSV can be found in my previous <a href="https://medium.com/elasticsearch/introduction-to-kibana-as-a-dev-tool-2937bcb35aee">blog on Kibana</a>. I have indexed the data to an index named “testindexterms”. A sample document in this index is as shown below:</p><pre>{<br>        &quot;_index&quot; : &quot;testindexterms&quot;,<br>        &quot;_type&quot; : &quot;_doc&quot;,<br>        &quot;_id&quot; : &quot;-6Ry3m0B5ZMO9aBbs5Sg&quot;,<br>        &quot;_score&quot; : 1.0,<br>        &quot;_source&quot; : {<br>          &quot;id&quot; : &quot;1&quot;,<br>          &quot;first_name&quot; : &quot;Jasmina&quot;,<br>          &quot;last_name&quot; : &quot;Crocetto&quot;,<br>          &quot;email&quot; : &quot;<a href="/service/mailto:jcrocetto0@odnoklassniki.ru">jcrocetto0@odnoklassniki.ru</a>&quot;,<br>          &quot;gender&quot; : &quot;Female&quot;,<br>          &quot;ip_address&quot; : &quot;90.139.240.83&quot;,<br>          &quot;socialSecurity&quot; : &quot;896-40-5515&quot;,<br>          &quot;age&quot; : &quot;33&quot;<br>        }<br>  }</pre><h4>Basic term query</h4><p>The basic term query will not analyze the search keyword given to it while searching. Considering an example, let us search for the keyword “Male” on the field “gender” as shown below:</p><pre>#term query<br>POST testindexterms/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;term&quot;: {<br>      &quot;gender&quot;: {<br>        &quot;value&quot;: &quot;Male&quot;<br>      }<br>    }<br>  }<br>}</pre><p>The above query would result in returning all the documents with “gender” field having the value “Male”. <br>Now if we give the same query by changing the case as “male”, the same query will yield no results.</p><pre>POST testindexterms/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;term&quot;: {<br>      &quot;first_name&quot;: {<br>        &quot;value&quot;: &quot;jasmina&quot;<br>      }<br>    }<br>  }<br>}</pre><p>This shows that no where in the <a href="https://medium.com/elasticsearch/what-happens-when-a-document-is-indexed-in-elasticsearch-16b7ae3415bc">inverted index</a> for the field “gender”, there is a value “male”. There are only two values “Male”and “Female” (note the difference in casing).</p><h4>terms query</h4><p>In some cases, we need to fetch multiple search keys from the same field. We can use the terms query for such cases as shown in the below example:</p><pre>POST testindexterms/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;terms&quot;: {<br>      &quot;first_name&quot;: [<br>        &quot;Jasmina&quot;,<br>        &quot;Claresta&quot;<br>      ]<br>    }<br>  }<br>}</pre><p>The above query will return the documents matching “Jasmina” and “Claresta” on the field “first_name”</p><h4>prefix query</h4><p>The prefix query would return the documents starting with a pattern of words. Suppose we need to get the documents where the “first_name” starts with “Ja”, we can use the prefix query like below:</p><pre>#prefix query<br>POST testindexterms/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;prefix&quot;: {<br>            &quot;first_name&quot;: {<br>                &quot;value&quot;: &quot;Ja&quot;<br>            }<br>        }<br>    }<br>}</pre><h4>range query</h4><p>This is one of the most useful and frequent queries which are used in Elasticseaerch. This query would return us the documents which contain terms within a provided range.<br>Example, to find all the employees falling between the age 20 and 40. <br>Or to find all the employees, whose salary is above 100,000 etc.</p><p>The below query can be used to fetch the documents where the age is falling between 20 and 40 (including 20 and 40)</p><pre>POST testindexterms/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;range&quot; : {<br>            &quot;age&quot; : {<br>                &quot;gte&quot; : 20,<br>                &quot;lte&quot; : 30<br>            }<br>        }<br>    }<br>}</pre><p>gte : greater than or equal to<br>lte: less than or equal to</p><p>the options, gt (greater than) and lt (less than) is also available to determine the range.</p><h4>fuzzy query</h4><p>Now another common use case is to search for a word and find the results irrespective of minor spell issues. Like if we search for “Jaems” the result should return the document containing “James”. <br>With the fuzzy query we can handle such situation. In the below query, the query is given as “Mael” against the field “gender”. Since we have used the fuzzy query, it will fetch all the document which contain the word “Male” in the field “gender”</p><pre>#fuzzy query<br>POST testindexterms/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;fuzzy&quot;: {<br>            &quot;gender&quot;: {<br>                &quot;value&quot;: &quot;Mael&quot;<br>            }<br>        }<br>    }<br>}</pre><p>Basically the fuzzy query is designed to find out the mispellings for only one character as shown above. If we use the same query and search for “Meal”, it would return no documents. This can be configured using advanced parameters made available in the fuzzy query. We can use the parameter , fuzziness set to two for achieving the same as below:</p><pre>POST testindexterms/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;fuzzy&quot;: {<br>            &quot;gender&quot;: {<br>                &quot;value&quot;: &quot;Meal&quot;,<br>                &quot;fuzziness&quot;: &quot;2&quot;<br>            }<br>        }<br>    }<br>}</pre><h3>Conclusion</h3><p>In this blog, we have seen some of the important term level queries, with examples. In the next blog we can see another section of full text queries which was postponed as it required some background in term level queries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2df2dbbe7477" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/elasticsearch-queries-part-03-term-level-queries-2df2dbbe7477">Elasticsearch queries — part 03 — term level queries</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch queries — part 02 — full text queries]]></title>
            <link>https://medium.com/elasticsearch/elasticsearch-queries-part-02-full-text-queries-e5ffe8fb3110?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/e5ffe8fb3110</guid>
            <category><![CDATA[elasticsearch-queries]]></category>
            <category><![CDATA[querydsl]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 19 Oct 2019 02:01:01 GMT</pubDate>
            <atom:updated>2019-10-19T02:01:01.547Z</atom:updated>
            <content:encoded><![CDATA[<h3>Elasticsearch queries — part 02 — full text queries</h3><h3>Phase 03 — Elasticsearch Full text queries — Blog 12</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAGbkO2JPnaWwB94HUEIbw.png" /></figure><p>In my previous <a href="https://medium.com/elasticsearch/introduction-to-elasticsearch-queries-b5ea254bf455">blog</a>, we have seen the basic classification of the Elasticsearch queries (leaf and compound), the basics of both the categories, and also the query/filter contexts. In this blog, the purpose is to get introduced to the common full-text queries in the Elasticsearch world.</p><p>As a preparation to this blog, let us index some data which predominantly consists of some texts. For the purpose of simplicity, I have taken a CSV of a trimmed version of Facebook posts and their description and details, which are available in <a href="https://data.world/martinchek/2012-2016-facebook-posts">public</a>. You can download the trimmed version of the same from <a href="https://github.com/2arunpmohan/elasticsearch_blogs/tree/master/blog-12">here</a>. You can index these tweets to Elasticsearch as we have discussed in my earlier blog discussing <a href="https://medium.com/elasticsearch/introduction-to-kibana-as-a-dev-tool-2937bcb35aee">Kibana as a dev tool</a>.</p><p>I have indexed the above tweets to an index named <strong>fb-post</strong>. A sample data document after indexing would look like below:</p><pre>{<br>        &quot;_index&quot; : &quot;fb-post&quot;,<br>        &quot;_type&quot; : &quot;_doc&quot;,<br>        &quot;_id&quot; : &quot;TszxwG0Bm6hFGbtHjVCC&quot;,<br>        &quot;_score&quot; : 1.0,<br>        &quot;_source&quot; : {<br>          &quot;status_type&quot; : &quot;shared_story&quot;,<br>          &quot;link&quot; : &quot;<a href="http://abcnews.go.com/blogs/headlines/2011/12/chief-justice-roberts-responds-to-judicial-ethics-critics/">http://abcnews.go.com/blogs/headlines/2011/12/chief-justice-roberts-responds-to-judicial-ethics-critics/</a>&quot;,<br>          &quot;description&quot; : &quot;PAUL J. RICHARDS/AFP/Getty Images Chief Justice John Roberts issued a ringing endorsement Saturday night of his colleagues’ ability to determine when they should step down from a case because of a conflict of interest. “I have complete confidence in the capability of my colleagues to determine when ...&quot;,<br>          &quot;caption&quot; : &quot;abcnews.go.com&quot;,<br>          &quot;love_count&quot; : 0,<br>          &quot;shares_count&quot; : 12,<br>          &quot;page_id&quot; : 86680728811,<br>          &quot;wow_count&quot; : 0,<br>          &quot;post_type&quot; : &quot;link&quot;,<br>          &quot;id&quot; : &quot;86680728811_272953252761568&quot;,<br>          &quot;posted_at&quot; : &quot;2012-01-01 00:30:26&quot;,<br>          &quot;sad_count&quot; : 0,<br>          &quot;angry_count&quot; : 0,<br>          &quot;message&quot; : &quot;Roberts took the unusual step of devoting the majority of  his annual  report to the issue of judicial ethics.&quot;,<br>          &quot;picture&quot; : &quot;<a href="https://external.xx.fbcdn.net/safe_image.php?d=AQAPXteeHLT2K7Rb&amp;w=130&amp;h=130&amp;url=http%3A%2F%2Fabcnews.go.com%2Fimages%2FPolitics%2Fgty_chief_justice_john_roberts_jt_111231_wblog.jpg&amp;cfs=1&amp;sx=108&amp;sy=0&amp;sw=269&amp;sh=269">https://external.xx.fbcdn.net/safe_image.php?d=AQAPXteeHLT2K7Rb&amp;w=130&amp;h=130&amp;url=http%3A%2F%2Fabcnews.go.com%2Fimages%2FPolitics%2Fgty_chief_justice_john_roberts_jt_111231_wblog.jpg&amp;cfs=1&amp;sx=108&amp;sy=0&amp;sw=269&amp;sh=269</a>&quot;,<br>          &quot;likes_count&quot; : 61,<br>          &quot;thankful_count&quot; : 0,<br>          &quot;<a href="http://twitter.com/timestamp">@timestamp</a>&quot; : &quot;2012-01-01T00:30:26.000+05:30&quot;,<br>          &quot;comments_count&quot; : 27,<br>          &quot;name&quot; : &quot;Chief Justice Roberts Responds to Judicial Ethics Critics&quot;,<br>          &quot;haha_count&quot; : 0<br>        }<br>      }</pre><p>In the above document, our fields of interests are the text fields like “name”, “message” and “description”.</p><p>Now let us move to each of the full-text query one by one.</p><h4>1. Match query</h4><p>We have discussed the <strong>match</strong> query in the previous blog, but haven’t mentioned the normal use cases for the match query. The most common use case for the <strong>match</strong> query is when we have a large data set and we need to find some near-exact matches quickly.</p><p>For example, in our twitter data set, we need to find if there is a word “confidence” in the entire tweet collection. This might be done using the simple <strong>match </strong>query against the field “text” as below:</p><pre>POST fb-post/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;description&quot;: {<br>        &quot;query&quot;:&quot;confidence&quot;<br>      }<br>    }<br>  }<br>}</pre><p>The results for this would show the tweets with the text “confidence”. <br>Now in the above example, we have seen given only a single word. What happens when we give multiple words?. Let us try the following query, and here the query we are going to give is “<strong>confidence buildings</strong>”</p><pre>POST fb-post/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;description&quot;: {<br>        &quot;query&quot;:&quot;confidence buildings&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Now, this would return documents matching either “confidence” <strong>OR</strong> “buildings” . The default behavior of the <strong>match </strong>query is of <strong>OR</strong>. This can be changed. If we want to match both “confidence” <strong>AND “</strong>buildings”, we can specify the “operator” parameter in the query as below:</p><pre>POST fb-post/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;description&quot;: {<br>        &quot;query&quot;:&quot;confidence buildings&quot;,<br>        <strong>&quot;operator&quot;:&quot;AND&quot;</strong><br>      }<br>    }<br>  }<br>}</pre><p>The above query would return the documents which contain both “confidence” AND “buildings” (which is zero in our data set)</p><p><strong>2. multi-match query</strong></p><p>The multi match query, as the name suggests, would search for the search keywords in more than one field. Suppose we have a search keyword “Giffords family”, to be searched in both the “name” and “description” field, we can make use of the multi-match query.</p><pre>POST fb-post/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;multi_match&quot; : {<br>      &quot;query&quot;:    &quot;Giffords family&quot;, <br>     <strong> &quot;fields&quot;: [ &quot;name&quot;, &quot;description&quot; ] </strong><br>    }<br>  }<br>}</pre><p>Here the words “Giffords” OR “family” is searched against the fields “name” and “description”, and the matching documents are returned.</p><p>We can also give custom scoring against the specific fields. In the below query, a boost of 5 is given to all the documents which match the keywords against the field “name”</p><pre>POST fb-post/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;multi_match&quot; : {<br>      &quot;query&quot;:    &quot;Giffords family&quot;, <br>     <strong> &quot;</strong>fields<strong>&quot;: [ &quot;name^5&quot;, </strong>&quot;description&quot;<strong> ] </strong><br>    }<br>  }<br>}</pre><p><strong>3. query_string query</strong></p><p>Another useful query is the query_string query. It is similar to the match query, but here, the format of the search keywords matter. It expects a specific format and if the search keyword is not of the same format, it returns an error.</p><p>Consider the following query:</p><pre>POST fb-post/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;query_string&quot; : {<br>            &quot;query&quot; : &quot;(step down) OR (official act)&quot;<br>        }<br>    }<br>}</pre><p>Here, the search keyword is first split in to two parts, to the left of the OR and to the right of the OR condition. That is operators in the search query serves as the de-limiters. Then each part will be analyzed (according to the field which it is being queried, in the above example which queries all the fields, it would do a standard analying) and then queried.</p><p>Querying can be done against a particular field/fields also as below:</p><pre>POST fb-post/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;query_string&quot; : {<br>            &quot;query&quot; : &quot;(step down) OR (official act)&quot;,<br>            <strong>&quot;fields&quot; : [&quot;description&quot;,&quot;name&quot;]</strong><br>        }<br>    }<br>}</pre><p><strong>4. match_phrase query</strong></p><p>Match_phrase query is a particularly useful piece of query which looks for matching the phrase rather than individual words. Here in the example given below, the match_phrase query fetches documents which matches the words “deeply concerned” in the same order.</p><pre>POST fb-post/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;match_phrase&quot; : {<br>            &quot;description&quot; : &quot;<strong>deeply concerned</strong>&quot;<br>        }<br>    }<br>}</pre><p>A very useful customization for the match_phrase query would be matching even if the order of the words are changed. For example, if we want “deeply concerned” and “concerned deeply” to get matched, we can use the help of the slop parameter with the <strong>match_phrase </strong>query as below:</p><pre>POST fb-post/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;match_phrase&quot;: {<br>            &quot;description&quot; : {<br>                &quot;query&quot; : &quot;deeply concerned&quot;,<br>                &quot;slop&quot;: 2<br>            }<br>        }<br>    }<br>}</pre><p>The slop value defaults to 0 and ranges to a maximum of 50. In the above example, the slop value 2 indicates, how far the terms can be allowed to consider as a match.</p><p>Now consider the below query with an incomplete keyword “ab” at the end of the phrase. The match_phrase query does not give matches for this query even if there is document having “deeply concerned <strong>about</strong>”<strong> </strong>phrase in the “description ”field</p><pre>POST fb-post/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;match_phrase&quot; : {<br>            &quot;description&quot; : &quot;<strong>deeply concerned ab</strong>&quot;<br>        }<br>    }<br>}</pre><p><strong>5. match_phrase_prefix query</strong></p><p>In the above example, we saw that the match_phrase query needs exact phrases to do the matches. But sometimes it would be handy if we could match partial matches too with the match_phrase_prefix query. The “match_phrase_prefix” query helps us to achieve this type of matches.</p><pre>POST fb-post/_search<br>{<br>    &quot;query&quot;: {<br>        &quot;match_phrase_prefix&quot;: {<br>            &quot;description&quot; : &quot;<strong>deeply concerned ab</strong>&quot;<br>        }<br>    }<br>}</pre><p>The above query can match phrases like the below: <br> “deeply concerned ab<strong>out</strong>”<br>“deeply concerned ab<strong>ove”</strong></p><p>A practical use case would be the autocomplete implementation of zip codes where users type in partial phrases.</p><h3>Conclusion</h3><p>In this blog, we have seen a few important full-text queries in Elasticsearch query world. I will be writing about the <strong>term level queries</strong> in the next blog and then coming back to some of the special full-text queries, which will aid in a better understanding.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5ffe8fb3110" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/elasticsearch-queries-part-02-full-text-queries-e5ffe8fb3110">Elasticsearch queries — part 02 — full text queries</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Elasticsearch Queries]]></title>
            <link>https://medium.com/elasticsearch/introduction-to-elasticsearch-queries-b5ea254bf455?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/b5ea254bf455</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[elasticsearch-queries]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 12 Oct 2019 06:11:01 GMT</pubDate>
            <atom:updated>2019-10-18T09:45:12.187Z</atom:updated>
            <content:encoded><![CDATA[<h3>Phase 03 — Elasticsearch queries intro — Blog 11</h3><p>In the <a href="https://medium.com/elasticsearch/introduction-to-elasticsearch-queries-b5ea254bf455">last blog</a>, we have seen how Kibana can be used as a dev tool and how sample data can be loaded using Kibana. <br>From this blog, we will be looking in to the query DSL of Elasticsearch, which is quite powerful and an indispensable field of knowledge for any Elasticsearch user.</p><h3>Elasticsearch query types</h3><p>The queries in Elasticsearch can be broadly classified into two categories,</p><h4>1. The leaf queries</h4><p>Leaf queries look for specific values in certain field/fields. These queries can be used independently. Some of these queries include match, term, range queries.</p><h4>2. The compound queries</h4><p>Compound queries uses the combination of leaf/compound queries. Basically, they combine multiple queries together to achieve their target results.</p><p>The broad classification of the two queries is roughly shown in the below diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*B2CWVPrA2EwqANIU13xe5w.png" /></figure><p>As you can see in the above picture, there are still many categories inside the Leaf and the compound classification. We will visit most of the queries/query types in the above figure in much more detail in the coming blogs.</p><h3>Basic query samples</h3><p>Now let us familiarize ourselves with 2 basic queries of the leaf and one queriy from the compound query type to get things started.</p><h4>1. The simple “match” query</h4><p>Suppose we consider the <a href="https://github.com/2arunpmohan/elasticsearch_blogs/tree/master/blog-10">documents indexed</a> in the <a href="https://medium.com/elasticsearch/introduction-to-kibana-as-a-dev-tool-2937bcb35aee">previous blog</a>, let us try a simple match query on the field “first_name” for the search keyword “Dany”. The query will look like below:</p><pre>POST employees/_search<br>{<br> “query”: {<br> “match”: {<br> “country”: “China”<br> }<br> }<br>}</pre><p>The above query will return us all the documents which has the country given as China</p><h4>2. Range query</h4><p>Now let us fire another query, this one too a leaf query. This query should return us all the employees who have salary greater than or equal to 500,000. This can be achieved using a range query as below:</p><pre>POST employees/_search<br>{<br>&quot;query&quot;: {<br>  &quot;range&quot;: {<br>    &quot;salary&quot;: {<br>      &quot;gte&quot;: 500000<br>    }<br>  }<br>}<br>}</pre><h4>3. Bool query</h4><p>Now comes the interesting part. How can we compare the above queries?. That is, I need all the employees who are from China, but earning more than 500,000. <br>This requires a combination of the above two leaf queries. Now Elasticsearch provides the facility for combining these queries using the bool query. Let us discuss the general structure of the bool query and then get back to the problem.</p><p><strong>Bool query general structure:</strong></p><pre>POST _search<br>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [...],<br>      &quot;filter&quot;: [...],<br>      &quot;must_not&quot;: [...],<br>      &quot;should&quot;: [...]<br>    }<br>  }<br>}</pre><p><strong>must</strong>: The clause (query) must appear in matching documents and will contribute to the score.</p><p><strong>filter: </strong>The clause (query) must appear in matching documents. However unlike must the score of the query will be ignored.</p><p><strong>should</strong>: The clause (query) should appear in the matching document.</p><p><strong>must_not:</strong> The clause (query) must not appear in the matching documents.</p><p>Now coming back to our problem, our bool query for returing all the employees from China and earning more than 500,000 would look like below:</p><pre>POST employees/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;match&quot;: {<br>            &quot;country&quot;: &quot;China&quot;<br>          }<br>        },<br>        {<br>          &quot;range&quot;: {<br>            &quot;salary&quot;: {<br>              &quot;gte&quot;: 500000<br>            }<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>Now, let us consider if we want to filter out all the male employees from the list. What should we do?. Just adding a must_not section in the above query with condition gender: “Male” would do the job, as below:</p><pre>POST employees/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;match&quot;: {<br>            &quot;country&quot;: &quot;China&quot;<br>          }<br>        },<br>        {<br>          &quot;range&quot;: {<br>            &quot;salary&quot;: {<br>              &quot;gte&quot;: 500000<br>            }<br>          }<br>        }<br>      ],<br>      &quot;must_not&quot;: [<br>        {<br>          &quot;match&quot;: {<br>            &quot;gender&quot;: &quot;Male&quot;<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><h3>The query context and the filter context</h3><p>Elasticsearch, by default, while returning the search results, would sort them based on their relevance <strong>score, </strong>which indicates how well the document matches the query. This relevance score is computed and returned in the <strong>_score </strong>parameter of the metadata with each result.</p><p>This is by default a positive floating point number.</p><p>The _score computation techniques might be different for different type of queries. That is the score computation for “match” query might be different from that of the “span” query.</p><p>But most importantly, the score computation depends on what context the query clause was ran. That is the query clauses can be ran in the “query” context or in the “filter” context.</p><h4>Query context</h4><p>When a clause is executed in the query context, it looks for “how well the document matched to the query”. The more the match, the higher the score would be.</p><p>An example would be as given in the screenshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*UF-Myxdi891P6D8XSTtX0A.png" /></figure><p>Here in the above example, I have searched for “Director of” in the field “title”. This query has returned me a few results, looking the result 1, the title has matched exactly the query clause (meaning, the keywords “Director” and “of”) was present in that. The score of the 1st document is therefore, higher <strong>7.363</strong></p><p>Where as in the 2nd document only one of the keywords in the query clauses matched (only the “Director” has matched in the 2nd document) and hence the score is less (<strong>5.305</strong> ) compared to the score of the first document.</p><p>So the first document has matched more well compared to the second document and that is clearly reflected in the <strong>_score </strong>metadata of both the documents.</p><p>This is what happens when the query clauses are given in the query context.</p><h3>Filter context</h3><p>When the query clause is given in the filter context, it just looks if the documents contains the clause of not. That is effectively a true/false returning. Suppose we are querying the data in the filter context, by asking whether the document field gender matches “Male”, we will get only the matching documents, with no score.</p><p>Unlike the query context, the filter context does not use time to compute the scores, and hence filter context return faster results.</p><p>The above example of filter context involving the filtering by gender can be shown in the below picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*pKE5TUSQRDgS4auqc2Qh8Q.png" /></figure><p>In the above example, you can see that when applied in the filter context, the scores for the resulting documents returned 0.</p><h3>Revisiting the bool query.</h3><p>Considering the above context, it is time to revisit the bool query.</p><p>In the bool query, the must and should sections, executes in the query context, meaning the clauses in the must section computes scores.</p><p>Where as the must_not and the should sections in the bool query executes the query clauses in the filter contexts, and will not affect scoring.</p><p>To demonstrate, let us try to same set of query clauses in <strong>must</strong> section first and then apply one clause in must and then one in the filter section and then see how the score variations.</p><h4>Case 1 : Both clauses inside the “must” section</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*xOGKANpdiZsIR_wxVfZp2w.png" /></figure><p>As you can see, in the above query, both the clauses are in the same <strong>must </strong>condition and the score returned by the 1st result’s document is <strong>2.4333658 </strong>(in the right panel)</p><h4>Case 2 : One clause is moved to the filter section</h4><p>Let us move the gender clause the filter section in the bool query as below and then run the query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*jP2PNCJftKSoC3s6D_-HTg.png" /></figure><p>Now in the right panel, looking at the scores, you can see that the score has dropped to <strong>1.7261622, </strong>which means that only the clausein the <strong>must </strong>section was computed for scoring and the clause in the <strong>filter </strong>section was not used for scoring.</p><p>To confirm this we can run the above query, only with the <strong>must </strong>section clause and see if it is returning the same score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*tQ5lLCTVi8hE1lMSgoAYkA.png" /></figure><p>As predicted, you can see from the above picture, the score remains the same, even with the filter clauses removed from the query.</p><h3>Conclusion</h3><p>In this blog, we have just familiarised with the classification of Elasticsearch queries, the context of the queries and some of the most basic queries.</p><p>From the next blog on wards we will look in to detail, on each type of the query with more examples and data sets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5ea254bf455" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/introduction-to-elasticsearch-queries-b5ea254bf455">Introduction to Elasticsearch Queries</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Kibana — as a dev tool]]></title>
            <link>https://medium.com/elasticsearch/introduction-to-kibana-as-a-dev-tool-2937bcb35aee?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/2937bcb35aee</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[elk]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 05 Oct 2019 13:27:20 GMT</pubDate>
            <atom:updated>2020-01-17T15:29:38.078Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to Kibana — as a dev tool</h3><p>Phase 03 — Kibana 7.x, installation, basics — Blog 10</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lpG2ynLX9Upc4j6jYuORYQ.png" /></figure><p>Its almost 2 years that I have written a blog in medium on the ELK stack, and a lot has changed during this time . Right from its name of ELK stack to the Elastic stack to almost every member of the stack has got an update. <br>But the one change it mattered the most was my shift from using the elasticsearhc-head plugin for day to day operations in Elasticsearch. I have completely transformed myself from using the head plugin to the world of Kibana. <br>In the last 2 years, from Kibana 5.x to Kibana 7.x, a lot has been changed and that too, the changes the users and developers were dying to see. Those changes range from the simple, JSON prettifying in the dev tools to the bulk data indexing facility via Kibana (either by supplying a JSON or CSV file).</p><p>So let me introduce the basic and most useful features of the power packed Kibana 7.x in this series, starting right from the installation and setup.</p><h3>Installation</h3><p><em>The installation mentioned here is only for the Ubuntu flavor of Linux.<br></em><strong>Step 1 — Download the setup file</strong> <br>you can download the Debian installation file from this <a href="https://www.elastic.co/downloads/past-releases/kibana-7-2-0">link</a>. <br>Here we are using the version 7.2.0 of Kibana. <br><em>Also, please make sure you have the version 7.2.0 or above Elasticsearch installed in your system. The installation is pretty much the same as we have seen in my earlier </em><a href="https://medium.com/elasticsearch/installing-elasticsearch-and-first-steps-to-the-api-dbc8e00c7ac4"><em>blog</em></a><br>Via command line you can just type in the following:</p><pre>wget <a href="https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-amd64.deb">https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-amd64.deb</a></pre><p><strong>Step 2— Install the debian package</strong></p><pre>sudo dpkg -i <a href="https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-amd64.deb">kibana-7.2.0-amd64.deb</a></pre><p><strong>Step 3 — Configuration of the Kibana.yml file<br></strong>By default the configuration file of the Kibana would be in the</p><pre>/etc/kibana/config/kibana.yml</pre><p>In the above file, we can specify many things, like the Elasticsearch host and the port on which Kibana is to run. <br>The default settings for the host adress for Elasticsearch would be the localhost and the port for Kibana running is the 5601. So ideally, if you are running the Elasticsearch in the same machine as that of Kibana, you should change nothing.</p><p><strong>Step 4— Start Kibana<br></strong>You can start the Kibana service using the command below:</p><pre>sudo service kibana start</pre><p>Now after this, go to the browser and visit, localhost:5601,Kibana would be loading and the landing page would appear as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b1gF-a7rVoepQAd5O9bmdQ.png" /></figure><p>In the above page, as of now, we are interested in only two sections.<br>1. The Data Loading Section<br>2. Dev tools section</p><p>We are restricting our areas of interest to only the above sections, because in the Phase 03 and Phase 04 of this blogs, we are not dwelling into the details of how visualizations and dashboards are created. Rather, we will be working on sample data indexing/loading to Elasticsearch and also to query the indexed data.<br>The other sections of the Kibana will be explored after Phase 04 in much more detail.</p><h3>The Dev Tools Section</h3><p>The dev tools section in Kibana, serves pretty much the same purpose as that of the elasticsearhc-head plugin we have seen before, but with greater flexibility and addendum. Let us create an index, named <a href="https://github.com/2arunpmohan/elasticsearch_blogs/blob/master/MOCK_DATA.csv"><strong>testindex</strong></a><strong> </strong>from the dev console. Open the dev console (by clicking the box 2) and then type in the following</p><pre>PUT testindex</pre><p>This will look as below in the dev console</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJIeJTKT-FD0JCvhRfXn8w.png" /></figure><p>Now after typing in the PUT request, press the play button (indicated by box 1) in the above image. The index will be created and the response will be coming in the right section , marked as the red box 2.</p><p>Like wise in the console, we can try most of the REST APIs for Elasticsearch. We will be dealing with the queries APIs, primarily in the next two phases.</p><h3>The data loading section</h3><p>In the above picture, click on the box 1, which says “<em> Import a CSV, NDJSON, or log file</em>” and now a screen would appear like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nh-hq-qGNWROayk8gUsBLw.png" /></figure><p>Now download the sample data from <a href="https://github.com/2arunpmohan/elasticsearch_blogs/blob/master/blog-10/blog-10_sampleData.csv">here</a> and upload it using the above screen. After that, the next screen would look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fzaDmeKx-cQoF3vvTdO7Pw.png" /></figure><p>After pressing the “import” from the above screen, the next screen would be asking for the index name where the data should be loaded as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*t-UvgctzKFwC4qyUhfLuDQ.png" /></figure><p>In the screen shown in the left, select the “Advanced” tab, in order to edit the mapping. <br>Since the sample data, we have contains a date, field, it would be helpful, if we change the mapping of the same, which can be done in the “advanced” section. Upon clicking the “advanced” tab, the screen would show the following screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3ShOBYTexsJija6V4ngcg.png" /></figure><p>In the above screen, in the 1st section (red box 01), I have provided a unique index name (testindex-01), then in the “mappings” section, I have edited the mapping type of the field “joiningDate” to the type “date”. After this click on the 3rd box, namely the “import”. This will start the uploading the file and indexing the data. A progress bar like below will indicate the completion of the data indexing process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jaX34JnyrjElXYrhLZa6DA.png" /></figure><p>Now, from the left navigation bar, click the “Dev tools” and type like below and press the play button to run the query and in the response we can see the indexed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*CIo8do4r-2IAyvB4JCWmkw.png" /></figure><p>In the left panel, where we run the queries, is a simple search request, which will return 10 documents from the index. The response in the right panel shows the documents indexed. The red box marked in the red shows one such document. The indexed document starts from the object under the “_source” field. The “_index”,”_type”,”_id”,”_score” fields are called as the meta data.</p><h3>Conclusion</h3><p>In this blog we have seen how to install, run and run simple requests using Kibana. Also we have seen how to load the data in CSV using the Kibana console. Now I iterate, these two make only 10–20% of the Kibana usage, the other 80% is building visualizations/dashboards, which will be covered later.</p><p>The 2 sections we saw, will be helpful for the blogs relating to the queries and many other APIs which will be coming in the next phases. So lets gear up for the next lessons in queries and aggregations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2937bcb35aee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/introduction-to-kibana-as-a-dev-tool-2937bcb35aee">Introduction to Kibana — as a dev tool</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom analyzer building in Elasticsearch]]></title>
            <link>https://medium.com/elasticsearch/custom-analyzer-building-in-elasticsearch-4e86f7c9c3be?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/4e86f7c9c3be</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Thu, 14 Dec 2017 17:08:51 GMT</pubDate>
            <atom:updated>2017-12-16T17:25:00.138Z</atom:updated>
            <content:encoded><![CDATA[<h4>Phase 02 — indexing, mapping and analysis — Blog 09</h4><h3>Introduction</h3><p>In the previous blog in this phase, I have explained more about the general analyzer structure and components. Also I have explained what the function of each component is. In this blog we will see the implementation side, by building a custom analyzer and then querying and seeing the difference.</p><h3>Case for a custom analyzer</h3><p>So let us consider a case for our custom analyzer. Suppose the text we are inputting to Elasticsearch contains the following</p><p><strong>1. html tags</strong><br>html tags can occur in our text while indexing, and this is actually not needed in most cases. So we need to remove those. <br>2. <strong>stopwords<br></strong>Words like the,and,or etc, are less significant when it comes to searching the content and are generally called as stopwords<br><strong>3. upper case characters.</strong><br><strong>4. short forms</strong> like H2O, $,% <br>In some cases, the short forms like these should be replaced with their original English words.</p><p>A sample case for text include inf the above cases will be like below:<br>Arun has 100 $ which accounts to 3 % of the total &lt;h2&gt; money &lt;/h2&gt;</p><p>So let us see how this text is analyzed , parsed and indexed in Elasticsearch using custom analyzers.</p><h3>Applying the custom analyzer</h3><p>In the above sample text, the operations that need to be done and the corresponding components for the custom analyzer are given in the below table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*SpdkJnHLgWQZE9Q8." /></figure><p>Custom analyzers are built in based on the requirements like above. Here we have a total of 5 things to be taken care of as shown in the above table.</p><p>In the index settings of the elasticsearch, there is a section called “analysis”,under the “settings” object, where we can specify the individual components, which we are going to use. Now under another section called “analyzers” under the “analysis”,we name our custom analyzer/analysers and inside each custom analyzer, we pick the components we want to be used.</p><p>After this, during mapping, we give the name of this custom analyzer under the “analyzer” field under the required field to make the text in that field undergo required analysis.</p><p>The hierarchy in the “settings” would look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/0*XBxncPBvp44XsOrA." /></figure><h4>Applying all the components</h4><p>Now applying all the above components to create a custom analyzer, will look like below:</p><pre>curl -XPUT localhost:9200/testindex_0204 -d &#39;{<br>  &quot;settings&quot;: {<br>    &quot;analysis&quot;: {<br>      &quot;char_filter&quot;: {<br>        &quot;subsitute&quot;: {<br>          &quot;type&quot;: &quot;mapping&quot;,<br>          &quot;mappings&quot;: [<br>            &quot;$=&gt; dollar&quot;,<br>            &quot;%=&gt; percentage&quot;<br>          ]<br>        },<br>        &quot;html-strip&quot;: {<br>          &quot;type&quot;: &quot;html_strip&quot;<br>        }<br>      },<br>      &quot;tokenizer&quot;: &quot;standard&quot;,<br>      &quot;filter&quot;: {<br>        &quot;stopwords_removal&quot;: {<br>          &quot;type&quot;: &quot;stop&quot;,<br>          &quot;stopwords&quot;: [<br>            &quot;has&quot;,<br>            &quot;which&quot;,<br>            &quot;to&quot;,<br>            &quot;of&quot;,<br>            &quot;the&quot;<br>          ]<br>        }<br>      },<br>      &quot;analyzer&quot;: {<br>        &quot;custom_analyzer_type_01&quot;: {<br>          &quot;type&quot;: &quot;custom&quot;,<br>          &quot;char_filter&quot;: [<br>            &quot;subsitute&quot;,<br>            &quot;html_strip&quot;<br>          ],<br>          &quot;tokenizer&quot;: &quot;standard&quot;,<br>          &quot;filter&quot;: [<br>            &quot;stopwords_removal&quot;,<br>            &quot;lowercase&quot;<br>          ]<br>        }<br>      }<br>    }<br>  },<br>  &quot;mappings&quot;: {<br>    &quot;test_type&quot;: {<br>      &quot;properties&quot;: {<br>        &quot;text&quot;: {<br>          &quot;type&quot;: &quot;string&quot;,<br>          &quot;analyzer&quot;: &quot;custom_analyzer_type_01&quot;<br>        }<br>      }<br>    }<br>  }<br>}&#39;</pre><p>This will create the index with the custom analyzer, named “custom_analyzer_01”. <br> This mapping is explained in detail, with each section explained in the below figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/0*-Px_GJgH2xw3yLNn." /></figure><h3>Token generation using the custom analyzer</h3><p>The tokens generated by using this analyzer can be seen by using the analyze API, like below:</p><pre>curl -XGET &quot;localhost:9200/testindex_0204/_analyze?analyzer=custom_analyzer_type_01&amp;pretty=true&quot; -d &#39;Arun has 100 $ which accounts to 3 % of the total &lt;h2&gt; money &lt;/h2&gt;&#39;</pre><p>The token list is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/0*DWAwDrLzy2N1mT3O." /></figure><p>Here you can make some observations:</p><ol><li>The token number 3 and 6 were originally $ and %, but it was then replaced by &quot;dollar&quot; and &quot;percentage&quot; as we specified in the char_filter section.</li><li>Also the html tags &lt;h2&gt; and &lt;/h2&gt; are also removed from the token list by the html_strip filter</li><li>The terms &quot;to&quot;,&quot;the&quot;,&quot;which&quot;,&quot;has&quot; etc which was mentioned in the stopwords filter has been removed from the token list.</li><li>The token number 1 should originally look like &quot;Arun&quot;, but has been lowercased by our filter applied.</li></ol><h3>Conclusion</h3><p>In this blog we have seen how a custom analyzer is built and applied to a field in Elasticsearch. With this blog, I intend to conclude the phase 2 of the blog series (indexing,mapping and analysis). This phase is one of the foundational pieces in understanding Elasticsearch as from now on we might be using the inputs from this phase for many purposes. From Phase 03, I will be introducing you to the query DSL world of elasticsearch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e86f7c9c3be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/custom-analyzer-building-in-elasticsearch-4e86f7c9c3be">Custom analyzer building in Elasticsearch</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Analysis and analyzers in Elasticsearch]]></title>
            <link>https://medium.com/elasticsearch/introduction-to-analysis-and-analyzers-in-elasticsearch-4cf24d49ddab?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/4cf24d49ddab</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 09 Dec 2017 10:23:11 GMT</pubDate>
            <atom:updated>2017-12-16T18:31:44.775Z</atom:updated>
            <content:encoded><![CDATA[<h4>Phase 02 — indexing, mapping and analysis — Blog 08</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmvPfSSm2G8C-khX1rhCGg.png" /></figure><h3>Introduction</h3><p>In the first blog of this series we have seen the inverted index computation when a document is indexed in Elasticsearch, and in the second blog we have seen the basics of mappings in Elasticsearch. Now in this blog we will see in detail about the analysis part of Elasticsearch, how it is done and how can we customize the analysis.</p><h3>1. Analyzing Process Explain</h3><p>In order to understand the analyzing process and the need for it in Elasticsearch, we need to have a little more deeper look into the inverted index creation in Elasticsearch. What we have discussed in the blog 01 of phase 02, about the inverted index creation was a basic version, here let me add some intricacies to the inverted index creation scenario.</p><p>When we index these documents to Elasticsearch the flow of process is like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*UfMsqzCkTAkQdlR8." /></figure><p>Now let me explain each stage befor the inverted index creation:</p><h4>1.1 Character filters</h4><p>The character filter has the ability to perform addition,removal or replacement actions on the input text given to them. To understand it more clearly, if the input strings contains a misspelled word recurring and we need to replace it with the correct one, we can use character filters for the same. One of the most common applications of this filter is to strip down the html tags from the input text.</p><p>Let us see the working of the character filter using the Analyze API of Elasticsearch. Here we are going to remove the html tags from a text with the character filter named “html_strip” . The curl request for that is as below:</p><pre>curl -XPOST &#39;localhost:9200/_analyze?pretty&#39; -H &#39;Content-Type: application/json&#39; -d &#39;{<br>  &quot;tokenizer&quot;: &quot;standard&quot;,<br>  &quot;char_filter&quot;: [<br>    &quot;html_strip&quot;<br>  ],<br>  &quot;text&quot;: &quot;The &lt;b&gt; Auto-generation &lt;/b&gt; is a success&quot;<br>}&#39;</pre><p>The resulting tokens are like below:</p><p>“The”,”Auto”,”generation”,”is”,”a”,”success”</p><p>Here we can see there are no html tags in the tokens. Likewise try out the above curl request without the “char_filter”:[“html_strip”] and see the difference.</p><h4>1.2 Tokenizer</h4><p>The input text after its transformation from the Character filter is passed to the tokeniser. The tokenizer would split this input text into individual tokens (or terms) at specific characters. The default tokenizer in elasticsearch is the “standard tokeniser”, which uses the grammar based tokenisation technique, which can be extended not only to English but also many other languages.</p><p>Let us see one example for a standard tokeniser below:</p><pre>curl -XPOST ‘localhost:9200/_analyze?pretty’ -H ‘Content-Type: application/json’ -d &#39;{<br> “tokenizer”: “standard”,<br> “text”: “The Auto-generation is a success”<br>}&#39;</pre><p>In the response, you can see the text is broken into below tokens:</p><pre>“The”,”Auto”,”generation”,”is”,”a”,”success” </pre><p>Here the words are split whenever there is a white-space and also a hyphen (-).</p><p>Note: There are different types of tokenizers for different purposes. In some use cases we might not need splitting at special characters, like in the case of an email id or a url, so in order to cater such needs tokenizers like “UAX URL Email Tokenizer” are available for our disposal. The list of tokenizers supplied by Elasticsearch can be found<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenizers.html"> here</a></p><h4>1.3. Token filter</h4><p>After the input text is split into tokens/terms, it is handed over to the final stage of analysis, the token filtering. Token filters can act on the tokens generated from the tokenizers and modify,add, or remove them. Let us try a token filter with the above example. The token filter we are going to try out here is the lowercase token filter, which will lower case all the tokens coming into it. The following curl request uses the analyze API for demonstrating the same:</p><pre>curl -XPOST &#39;localhost:9200/_analyze?pretty&#39; -H &#39;Content-Type: application/json&#39; -d&#39;{<br>  &quot;tokenizer&quot;: &quot;standard&quot;,<br>  &quot;filter&quot;: [<br>    &quot;lowercase&quot;<br>  ],<br>  &quot;text&quot;: &quot;The Auto-generation is a success&quot;<br>}&#39;</pre><p>The tokens generated in the response is seen as below:</p><pre>“the”,”auto”,”generation”,”is”,”a”,”success”</pre><p>Notice that every token in now lowercased. This is what the lowercase token filter does to the tokens.</p><p>For a list of the token filters that comes with Elasticsearch, visit the link<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenfilters.html"> here</a></p><p>One of the most common use cases of token filters in Elasticsearch is to add synonyms to a word. This essentially means, a word can be mapped to its synonyms by using this filter and whenever we search for the synonyms, the document containing the base word will appear. We will see the application of this in our future blogs.</p><h3>2. Analyzers</h3><p>The process of analyzing the contents of fields in documents by Elasticsearch was explained in the above section. As it is mentioned in the above section, there are several kinds of Character filters, tokenizers and token filters available and we are supposed to choose them wisely according to the use case we are encountering. The combination of these three components (character filters,tokenizers and token filters) are called as Analyzers. There are several types of built in Analysers available in Elasticsearch for dealing with the most common use cases. For example, the Standard Analyzer, the default analyser of Elasticsearch, is a combination of a standard tokenizer and two token filters (standard token filter, lowercase and stop token filter). Likewise a wide variety of analyzers are possible depending on the combinations of the char filters, tokenizers and token filters.</p><p>The general structure of an analyser is represented below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/0*j90hAftsL47MGivN." /></figure><p>We can also make custom analyzers by selecting the required filters and the tokenizer. We will see the making of a custom analyzer in the next blog to this series.</p><h3>3. Analysis Phases</h3><p>Now that we have a clear picture of what is analysis and what are analyzers, let us move in to the two phases of analysis that happens in Elasticsearch, the index time analysis and the search time analysis.</p><h4>3.1 Index time analysis</h4><p>Let us consider the following document for indexing</p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/1 -d &#39;{<br>   &quot;text&quot;: &quot;My name is Arun&quot;<br>}&#39;</pre><p>Since we are not applying an analyzers, Elasticsearch applies the default analyzer “Standard analyzer” on this. Let us see the final tokens of the above document, when applied with Standard Analyser, with the help of analyze API</p><pre>curl -XPOST &#39;localhost:9200/_analyze?pretty&#39; -H &#39;Content-Type: application/json&#39; -d&#39;{<br>   &quot;analyzer&quot;: &quot;standard&quot;,<br>   &quot;text&quot;: &quot;My name is Arun&quot;<br>}&#39;</pre><p>The tokens generated for storing in the inverted index are:</p><pre>“my”,”name”,”is”,”arun”</pre><p>The inverted index will look like the below table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/111/0*C-F7ZuWmilNM4SZq." /></figure><p>This entire process happens in the index time and hence the name index time analysis.</p><h4>3.2 Search time analysis</h4><p>Search time analysis, as the name indicates will happen at search time. But there is a difference, that is this analysis occur on the query depending up on which query is used.</p><p><strong>3.2.1 Term query — Case 1</strong></p><p>Consider the following query:</p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/_search -d &#39;{<br>  “query”: {<br>    “term”: {<br>      “text”: “name”<br>    }<br>  }<br>}&#39;</pre><p>If we run this query against the index “testindex-0203”, it will return the document indexed as the result. The token “name” is there in the inverted index and is mapped agains the document 1. So when we search for the term “name”, it will look up the inverted index and since the term was found there the corresponding document was fetched as results.</p><p><strong>3.2.2 Term query — Case 2</strong></p><p>Now consider another case with the same “term” query, like below:</p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/_search -d &#39;{<br>  “query”: {<br>    “term”: {<br>      “text”: “Name”<br>    }<br>  }<br>}&#39;</pre><p>Here we query with the same term query, but with a different casing for the search keyword, its now “Name” instead of “name”. Now something intersting happens, this search will not yied us any documents. The reason for this strange behaviour is that “Name” is not existing in the inverted index and hence no documents to show.</p><p>So, with the “term” query no analysis is allowed to be done on the search keyword.</p><p><strong>3.2.3 Term query — Case 3</strong></p><p>Let us consider another case with the term query to see this behaviour, here is the query</p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/_search -d &#39;{<br>  “query”: {<br>    “term”: {<br>      “text”: “My name”<br>    }<br>  }<br>}&#39;</pre><p>In the above case , the search keyword is not analyzed and hence, Elasticsearch is looking for the token “My name” in the inverted index. And since such a term does not exists, elasticsearch will return zero results for the above query too.</p><p>This was the case of “term” query in elasticsearch. Let us try a different query called match query and check for the outputs.</p><p><strong>3.2.4 Match query — Case 1</strong></p><p>Consider the following query:</p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/_search -d &#39;{<br>  “query”: {<br>    “match”: {<br>      “text”: “name”<br>    }<br>  }<br>}&#39;</pre><p>This will give return the response with the indexed document as “name” token exists in the inverted index.</p><p><strong>3.2.5 Match query — Case 2</strong></p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/_search -d &#39;{<br>  “query”: {<br>    “match”: {<br>      “text”: “Name”<br>    }<br>  }<br>}&#39;</pre><p>Here, when we used the “term” query for the case 2, it yielded no response. But for match query, whatever analysis was applied to the field that is queried (text) at the time of indexing, the exact same analysis would done on the search keyword (“Name”)also . This makes the search keyword to undergo the “standard analysis” and the search keyword “Name” is changed to “name” (due to lowercase token filter in standard analyzer). This new search keyword “name” exists in the inverted index and the response will have the corresponding document too.</p><p><strong>3.2.6 Match query — Case 3</strong></p><pre>curl -XPOST localhost:9200/testindex-0203/testtype/_search -d &#39;{<br>  “query”: {<br>    “match”: {<br>      “text”: “My name”<br>    }<br>  }<br>}&#39;</pre><p>Here the search keyword given is “My name” which will transform in to the keywords “my” and “name” upon undergoing the standard analysis. Both of these keywords exists in the inverted index and hence return the document as the response.</p><p>So, depending on the query type, the search keywords are undergoing the analysis (same as that of the field which is queried up on) during the search time. This is called as search time analysis.</p><h3>Conclusion</h3><p>In this blog, I have introduced the very basic components of Analyzers and the types of analysis happening in Elasticsearch. In the next blog we wil see how to build our own custom analyzer for a very specific use case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cf24d49ddab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/introduction-to-analysis-and-analyzers-in-elasticsearch-4cf24d49ddab">Introduction to Analysis and analyzers in Elasticsearch</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Contents]]></title>
            <link>https://medium.com/elasticsearch/contents-cebdc419c8c9?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/cebdc419c8c9</guid>
            <category><![CDATA[about]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 09 Dec 2017 05:10:11 GMT</pubDate>
            <atom:updated>2020-02-16T04:15:04.202Z</atom:updated>
            <content:encoded><![CDATA[<p>The structure and road-plan of the Elasticsearch tutorial blogs.<br>Note: Those with the hyperlinks are done and published, where as those not with hyperlinks are to be published.</p><h3><strong>Phase 01</strong></h3><pre>Blog 01 — <a href="https://medium.com/elasticsearch/introduction-to-elasticsearch-5d9e9f3f769e">Introduction To Elasticsearch</a><br>Blog 02 — <a href="https://medium.com/elasticsearch/elastic-stack-familiarization-c330f605160b">Elasticsearch Stack Familiarisation</a><br>Blog 03 — <a href="https://medium.com/elasticsearch/installing-elasticsearch-and-first-steps-to-the-api-dbc8e00c7ac4">Installing Elasticsearch and first steps to the API</a><br>Blog 04 — <a href="https://medium.com/elasticsearch/elasticsearch-crud-operations-using-head-plugin-a18fc7b5d01e">Elasticsearch CRUD operations using head plugin</a><br>Blog 05 — <a href="https://medium.com/elasticsearch/elasticsearch-multiple-instances-and-head-plugin-familiarization-11a2a0d0b5d6">Elasticsearch Multiple Instances and head familiarisation</a></pre><h3>Phase 02</h3><pre>Blog 06 — <a href="https://medium.com/elasticsearch/what-happens-when-a-document-is-indexed-in-elasticsearch-16b7ae3415bc">What happens when a document is indexed in Elasticsearch</a><br>Blog 07 — <a href="https://medium.com/elasticsearch/mappings-in-elasticsearch-a-short-introduction-478faa1f40da">Mappings in Elasticsearch, a short introductio</a>n<br>Blog 08 — <a href="https://medium.com/elasticsearch/introduction-to-analysis-and-analyzers-in-elasticsearch-4cf24d49ddab">Introduction to Analysis and analyzers in Elasticsearch</a><br>Blog 09 — <a href="https://medium.com/elasticsearch/custom-analyzer-building-in-elasticsearch-4e86f7c9c3be">Custom analyzer building in Elasticsearch</a></pre><h3>Phase 03</h3><pre>Blog 10 - <a href="https://medium.com/elasticsearch/introduction-to-kibana-as-a-dev-tool-2937bcb35aee">Introduction to Kibana as a dev tool</a><br>Blog 11 - <a href="https://medium.com/elasticsearch/introduction-to-elasticsearch-queries-b5ea254bf455">Elasticsearch queries - part 01</a> <br>Blog 12 - Elasticsearch queries - part 02 - full text queries<br>Blog 13 - Elasticsearch queries - part 03 - term level queries<br>Blog 14 - Elasticsearch queries - part 04 - Geo queries<br>Blog 15 - Elasticsearch queries - part 05 - Span queries <br>Blog 16 - Elasticsearch queries - part 06 - Specialised queries<br>Blog 17 - Elasticsearch queries - part 07 - Nested queries<br>Blog 18 - Elasticsearch queries - part 08 - Parent child queries<br>Blog 19 - Elasticsearch queries - part 09 - Compound queries 01<br>Blog 20 - Elasticsearch queries - part 10 - Compound queries 02<br>Blog 21 - Introduction to Scoring<br>Blog 22 - Advanced scoring</pre><h3>Phase 04</h3><pre>Blog 23 - Introduction to Elasticsearch aggregations<br>Blog 24 - Aggregations - part 01 - bucket aggregations 01<br>Blog 25 - Aggregations - part 02 - bucket aggregations 01<br>Blog 26 - Aggregations - part 03 - pipeline aggregations 01<br>Blog 27 - Aggregations - part 04 - bucket aggregations 02<br>Blog 28 - Aggregations - part 05 - metric aggregations<br>Blog 29 - Aggregations - part 06 - matrix aggregations</pre><h3>Phase 05</h3><pre>Blog 30 - Introduction to Elasticsearch scripting<br>Blog 31 - Scripting - part 01<br>Blog 32 - Scripting - part 02 </pre><h3>Phase 06</h3><pre>Blog 33- Introduction to Elastic stack components (Logstash,Beats)<br>Blog 34- Logstash familiarisation 01 - Reading file and indexing to Elasticsearch <br>Blog 35- Logstash familiarisation 02 - JDBC to Elasticsearch <br>Blog 36- Logstash familiarisation 03 - Re-indexing using logstash<br>Blog 37- Logstash familiarisation 04 - Apache log parsing using Elasticsearch<br>Blog 38- Filebeat - Reading syslog and indexing to Elasticsearch (new fields introduction,Kibana dashboards)<br>Blog 38- Auditbeat - Audit log parsing and indexing to Elasticsearch (audit rule,file integrity,Kibana dashboards)<br>Blog 39- Metricbeat - Collecting system metrics (Default dashboards)<br>Blog 40- Packetbeat - Collecting network data (default dashboards)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cebdc419c8c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/contents-cebdc419c8c9">Contents</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mappings in Elasticsearch — A short introduction]]></title>
            <link>https://medium.com/elasticsearch/mappings-in-elasticsearch-a-short-introduction-478faa1f40da?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/478faa1f40da</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 09 Dec 2017 04:17:54 GMT</pubDate>
            <atom:updated>2017-12-16T18:30:20.231Z</atom:updated>
            <content:encoded><![CDATA[<h4>Phase 02 — indexing, mapping and analysis — Blog 07</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*3-6A9nJFXMmU34IEaDc4dg.png" /></figure><h3>Introduction</h3><p>In this short blog, I will explain what is mapping in elasticsearch along with some common useful best practices. A good understanding of mapping will be handy, when we learn analysing/analyzers in Elasticsearch later in this blog series.</p><h3>Mapping</h3><p>Mapping is nothing but the schema for the documents in Elasticsearch. As I have mentioned before in one of the blogs in Phase 01, Elasticsearch is schema less. This means, unlike other databases like MongoDb or MySql, there is no need to pre-define the schema for the document before we are indexing it to Elasticsearch. So how does this work?. It is simply that Elasticsearch has the built in ability to detect the type of the fields of a document and generate a schema and apply it throughout the index. Of Course there are some things to be taken care of here, but we will come back to it later in this blog. First let us see how the mapping comes into existence when we index a sample document.</p><h4>Step 1</h4><p>Create an index without any documents.</p><pre>curl -XPUT localhost:9200/testindex-0202</pre><p>We will get the following response for the above command:</p><pre>{<br>  “acknowledged”: true,<br>  ”shards_acknowledged”: true,<br>  ”index”: ”testindex-0202&quot;<br>}</pre><p>Now let us see if there are any mapping applied to it by making use of the “get mapping” API like below:</p><pre>curl -XGET localhost:9200/testindex-0202/_mapping</pre><p>The above command will result in the following response:</p><pre>{<br>  “testindex-0202”: {<br>    “mappings”: {<br>    }<br>  }<br>}</pre><p>As you can see from the response, the “mappings” section is empty, which implies there is no mappings applied at this stage.</p><h4>Step 2</h4><p>Now index a document to the created index, like below:</p><pre>curl -XPUT localhost:9200/testindex-0202/testtype/1 — d ‘{<br> “name”: ”ArunPMohan”,<br> ”age”: 31,<br> ”married”: true<br>}’</pre><p>As you can see in the above document, I have three fields and the types of the fields are string (for the “name” field), integer (for the “age” field) and boolean (for the “married” field). When we type in this command, it results in no error, which simply means that Elasticsearch has auto detected the field types. The above command for indexing the document, yields the following output from Elasticsearch:</p><pre>{<br>  “_index”: ”testindex-0202&quot;,”_type”:”testtype”,”_id”:”1&quot;,<br>  ”_version”: 1,<br>  ”result”: ”created”,<br>  ”_shards”: {<br>    “total”: 2,<br>    ”successful”: 1,<br>    ”failed”: 0<br>  },<br>  ”created”: true<br>}</pre><p>Now as we have created the document, we can use the “GET mapping” API to see the mapping changes. So apply the following command:</p><pre>curl -XGET localhost:9200/testindex-0202/_mapping</pre><p>The response is as given below:</p><pre>{<br>  “testindex-0202”: {<br>    “mappings”: {<br>      “testtype”: {<br>        “properties”: {<br>          “age”: {<br>            “type”: “long”<br>          },<br>          “married”: {<br>            “type”: “boolean”<br>          },<br>          “name”: {<br>            “type”: “text”,<br>            “fields”: {<br>              “keyword”: {<br>                “type”: “keyword”,<br>                “ignore_above”: 256<br>              }<br>            }<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>In the above response, please note the “mappings” section. Here the first object under the mappings is the “testtype”. The “testtype” indicates the type of the index. Below the “testtype”, we have the “properties” object, which holds the list of fields and their type. You can see the “age” field is of the type “long”, the “married” field of the type “boolean” and the “name” field of the type “text”. The “text” type indicates the field is of the type “string”. Under the name field we have another section called as “fields”. We will see more about the “fields” section in the later blogs, for now, it is enough that it comes with the string/text field as default.</p><h3>Things to know about Mappings</h3><p>Let us have a look at some of the important things that would help us to understand more about mappings.</p><h4>1. Updating mappings.</h4><p>In an Elasticsearch there would be multiple types and for each type has its own mappings. Once a mapping is generated for the number of fields in under a type, it cannot be modified. That is if a document consists of 10 fields, 10 mappings are generated and we cannot modify it afterwards. In case if we want to modify it, we need to delete the index and then apply the modified mapping manually and then re-index the data.</p><p>Another important thing to be noted here is that, the above is true if we if try to modify the existing mapping, but for a new field, we are able to update the mapping. That is, if we add a 11th field to the document, in between, the mapping will auto-update accordingly. To make things much more clear, let us index a new document to the “testindex-0202”. Here in the new document, we are including a new field called “country”, apart from the “name”,”age” and the “married” fields. Let us do the same like below:</p><pre>curl -XPUT localhost:9200/testindex-0202/testtype/2 -d ‘{<br> “name”: ”AlexJohn”,<br> ”age”: 32,<br> ”married”: true,<br> ”country”: ”Italy”<br>}’</pre><p>Now hit the GET mapping API as we have done before. In the results you can see there is another field added called “country”.</p><h4>2. Common Error</h4><p>One of the most common error when dealing with mmappings is that the changing of the existing types of a field. Let us see this error for ourselves and have a better understanding. In the index “testindex-0202”, so far we have indexed 2 documents. Both the documents had the same type for the field “age” and the type was “long”. Now let us try to index a document like below:</p><pre>curl -XPUT localhost:9200/testindex-0202/testtype/3 -d ‘{<br> “name”: ”LexaMathew”,<br> ”age”: ”twenty”,<br> ”married”: false,<br> ”country”: ”Italy”<br>}’</pre><p>Note that here the age is given in string type, which indicates that it is a string field. The response for the above request will be like below:</p><pre>{<br>  “error”: {<br>    “root_cause”: [<br>      {<br>        “type”: ”mapper_parsing_exception”,<br>        ”reason”: ”failedtoparse[<br>          age<br>        ]”<br>      }<br>    ],<br>    ”type”: ”mapper_parsing_exception”,<br>    ”reason”: ”failedtoparse[<br>      age<br>    ]”,<br>    ”caused_by”: {<br>      “type”: ”number_format_exception”,<br>      ”reason”: ”Forinputstring: \”twenty\””<br>    }<br>  },<br>  ”status”: 400<br>}</pre><p>In the above response we can see the error “mapper_parsing_exception” on the field “age”. This indicates that the the expected field here was of another type and not string. In such cases re-index the document with the appropriate type or check for the data and see if there is any data modifications to be done and change data format/mapping accordingly.</p><h4>3. Applying Custom mapping</h4><p>So far we have seen the auto generation of mapping, that is mapping was generated by elasticsearch only. Now the most common use case is to apply our own mapping. This can be done by using the PUT mapping API in Elasticsearch like below:</p><p>First create an index of the name “testindex-0202a”</p><pre>curl -XPUT localhost:9200/testindex-0202a</pre><p>Now use the PUT mapping API like below:</p><pre>curl -XPUT localhost:9200/testindex-0202a/_mapping/testtype -d ‘{<br>  “properties”: {<br>    “name”: {<br>      “type”: ”text”<br>    }<br>  }<br>}’</pre><p>This would create a mapping with “name” as a string/text field. Now any documents indexed in the index “testindex-0202a” and the type “testtype” would be expected to have the “name” field of the type “text”.</p><h4>4. Best practice in applying mappings</h4><p>In practice what happens is that the data types for a field might vary in different points of time. So it takes us a couple of iterations for us to fix on the structure/type of the document fields. So, we might need to constantly change the data type/structure or mapping for the index. What usually is a recommended practice is to create a shell script, which will first delete the existing index and then apply mapping to the index. The advantage of this practice being, it will be easy to delete the existing index and then modify the mapping by creating a new index, thus saving us a lot of time.</p><h3>Conclusion</h3><p>In this blog we have covered the basics of Elasticsearch mappings like the application of mapping by Elasticsearch, some best practices and also how to apply custom mapping to an Elasticsearch index. In the next blogs we will see in detail about the Elasticsearch analysis and analyzers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=478faa1f40da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/mappings-in-elasticsearch-a-short-introduction-478faa1f40da">Mappings in Elasticsearch — A short introduction</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What happens when a document is indexed in Elasticsearch?]]></title>
            <link>https://medium.com/elasticsearch/what-happens-when-a-document-is-indexed-in-elasticsearch-16b7ae3415bc?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/16b7ae3415bc</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 09 Dec 2017 04:16:04 GMT</pubDate>
            <atom:updated>2018-09-16T17:53:09.188Z</atom:updated>
            <content:encoded><![CDATA[<h4>Phase 02 — indexing, mapping and analysis — Blog 06</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmvPfSSm2G8C-khX1rhCGg.png" /></figure><h3>Introduction</h3><p>From this blog, we are entering the phase 02 of this blog series,named “indexing, analysis and mapping”. In this phase you will learn more in detail about the process of document indexing and the internal process during an document indexing like the analysis, mapping etc. This short first blog to the phase 02 series will introduce you to the general process that is happening when a document is indexed in Elasticsearch.</p><h3>Indexing a document and beyond</h3><p>Let us index a document like below to Elasticsearch</p><pre>curl -XPUT localhost:9200/testindex0201/testtype/1 -d ‘{<br> “name”: “Arun Mohan”,<br> “age”: 31<br>}’</pre><p>Now when we can take the head plugin (more about the head plugin here), and view the indexed file in the index name “testindex0201”. So we indexed the document in elasticsearch in a snap, but there are lot many questions relating to this simple process. Some of these questions are:</p><ol><li>where does the document I just created resides on the disk?</li><li>if it resides on disk, can I change the location?</li><li>is Elasticsearch storing the data as such or modifying it during the indexing process?</li><li>how come the search on documents faster in Elasticsearch?</li></ol><p>Let us see what the answers for these questions are in the coming sections.<br>1. Where do documents get stored in the disk?</p><p>Here in this blog we are talking about the .deb file installation of elasticsearch . The default data path for elasticsearch is the folder “/var/lib/elasticsearch/”. Here one thing to be noted is that, before the vesions 5.x, under the above path, a folder in the name of the cluster to which the node belongs to was created and the data was stored inside it. From 5.x this was deprecated and no cluster name is stored. So in effective, the data path for 5.x and the versions before would look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*6EChYu211XYYNFny" /></figure><h3>2. Changing the location of the data</h3><p>We have seen where in disk the data is stored by Elasticsearch. This brings to a basic question,that whether we can change the location of the data?. The need for such customisation occurs in situations like the default path of data containing less storage and we decide to mount another volume exclusively for the data storage.</p><p>Elasticsearch definitely provides the option for customising the data paths. The data path can be configured in the <em>elasticsearch.yml </em>file located in</p><pre>/etc/elasticsearch/elasticsearch.yml</pre><p>Here the custom path is to be applied against the “path.data” field.</p><h3>3. Data indexing process</h3><p>The below diagram shows a high level flow of the data indexing process in Elasticsearch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*48ggEosWDsQpvgS8" /></figure><p>As you can see from the above picture, the document is not as such indexed in the Elasticsearch, rather it is being performed some actions on it by the Analyzer component and split into tokens/terms. These terms are then stored in the disk as the inverted index. So let us brief a bit up on the Analyzer part of the above diagram (we will be revisiting Analyzers in detail in the next blog).</p><p>To start on Analyzers, it is good to have a small recap on how and what the input documents look like. Documents are JSON objects having their own set of key value pairs. In the example given above, we have two keys named “name” and “age”, also their values too. So when a document is to be indexed to Elasticsearch, the Analyzers part of the Elasticsearch takes each key, split them at certain delimiters (there are default delimiters such as space, full stop etc). The output of this splitting is called as tokens. Each token is then applied certain filters (standard filtering procedure includes the lower casing of all the splitted tokens). So in effective, a key consists of a series of tokens after the Analyzer is done analysing. These tokens after analysis is called as terms. These terms are then stored against that field (key) in an inverted index. Let us see what is this inverted index in detail in the coming section.</p><h3>4. Elasticsearch speed and Inverted Index</h3><p>As discussed in the above section, the “terms” generated by the Analyzer is sent to an inverted index. Now its time to have a detailed look at this term, the “inverted index”.</p><p>Inverted index is the primary reason for the robustness and speed of the Elasticsearch’s search. It is best explained with examples. Consider there are two documents like below:</p><p><strong>Document 1</strong></p><pre>{<br> “name” : “this is a cat”<br>}</pre><p>Document 2</p><pre>{<br> “name”: “there is a cat and a dog”<br>}</pre><p>After analysis, the terms in the documents would be as follows</p><p><strong>Document 1</strong></p><p>Field Terms</p><pre>“name”: “this”,”is”,”a”,”cat”</pre><p><strong>document 2</strong></p><p>Field Terms</p><pre>“name” : “there”,”is”,”a”,”cat”,”and”,”dog”</pre><p>Now let us combine the above two table of terms for the field “name” and make it a single one like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*axltigQzCXdd838jOrw1dw.png" /></figure><p>Now the above table is called the inverted index for the field “name”. The advantage of using inverted index is that, the search term can be looked up in the “terms” column and then, if there is a match, finding the documents where the search term is present is a very simple job of a look up at the corresponding column. For example, if in this case there are say one million documents. In the conventional method, we have to traverse each document and also each field ‘s value to retrieve the matching search results. With inverted index, we are only searching a selected set of terms and then since there is no duplication of terms, if we find a match, we look up which all documents have the terms in the “documents” column and then pass these documents as the result. Thus a lot amount of search time is saved compared to the conventional method.</p><p>If a data set contains n documents and if the number of fields in these documents are m, then the total number of inverted indices generated for the data set is equal to n*m.</p><p>And each inverted index is further optimised by avoiding common stopwords (like “the”,”is” etc) and thus creating a very short list of terms. This makes the search super faster.</p><h3>Conclusion</h3><p>Here in this blog, I have provided a brief picture of the indexing process in Elasticsearch. Now in the next blog to the phase two series, we will look in greater depths of the analysis process in Elasticsearch. We will see the components of Analyzers, the types of Analyzers and tokenisers and many more with practical examples.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16b7ae3415bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/what-happens-when-a-document-is-indexed-in-elasticsearch-16b7ae3415bc">What happens when a document is indexed in Elasticsearch?</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch multiple instances and head (plugin) familiarization]]></title>
            <link>https://medium.com/elasticsearch/elasticsearch-multiple-instances-and-head-plugin-familiarization-11a2a0d0b5d6?source=rss----7d515a650486---4</link>
            <guid isPermaLink="false">https://medium.com/p/11a2a0d0b5d6</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Arun Mohan]]></dc:creator>
            <pubDate>Sat, 09 Dec 2017 04:14:27 GMT</pubDate>
            <atom:updated>2017-12-09T10:25:08.809Z</atom:updated>
            <content:encoded><![CDATA[<h4>Phase 01 — Introduction to Elasticsearch — Blog 05</h4><h3>Introduction</h3><p>This is a short, optional blog to the elasticsearch introduction series. This blog contains two parts</p><ol><li>Installation of multiple elasticsearch instances in your system</li><li>video blog to various operations, using the plugin elasticsearch-head</li></ol><h3>1. Running Multiple instances</h3><p>Elasticsearch comes in different versions, from the latest 6.0 to the oldest ones. So it is a common scenario that we use different versions for different porjects. It might be very handy to know how to use different versions of elasticsearch in a single system, so that we can try and test the queries and other for different versions simultaneously and thus save a lot of time.</p><h3>Sample setup</h3><p>In this blog we will set up two instances of Elasticsearch, the first one being version 5.4.0 and the second one being 2.4.0, and run them simultaneously. I will be assigning the default port 9540 to the 5.4.0 version and for the version 2.4.0 let us assign the port 9240.</p><h4>Step 1 — download the zip file installations</h4><p>So for these kind of developer friendly installation of mulitple instances, the better practice is two download the zip file installations of elasticsearch.</p><p>For 5.4.0 the zip file for installation can be downloaded from<a href="https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.zip"> here</a></p><p>For 2.4.0 the zip file for installation can be downloaded from<a href="https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.4.0/elasticsearch-2.4.0.zip"> here</a></p><p>After downloading the zip files for both the versions, extract the zip files to corresponding folders.</p><h4>Step 2 — Edit the config files</h4><p>In the zip file installations the config file (elasticsearch.yml) is located inside the config folder like below:</p><p>For 2.4.0 /elasticsearch-2.4.0/config/elasticsearch.yml</p><p>Similarly, for 5.4.0 /elasticsearch-5.4.0/config/elasticsearch.yml</p><p>Now edit the config file for both like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DhPksttbG73GFGES." /></figure><p>In the above picture, you can see the port information is changed in the parameter “http.port” (boxes labelled 1).</p><p>The boxes labelled 2 are general settings which will prevent the CORS issue from occurring when we run the elasticsearch-head or point any other UI elements to this instance.</p><h4>Step — 3 Running the separate instances</h4><p>Now type in the following commands from each of the extracted folders as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/0*dWQpHDMkaFnNw4Gv." /></figure><p>Now this will start both elasticsearch versions on the respective ports mentioned in the config file. To verify once more, we can type in the sudo netstat -ntlp command in the terminal and see the process running in the ports. The result would be as shown in the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/0*WIdww2PO2yEpkyum." /></figure><h4>Step 4 — Accessing</h4><p>In order to access these separate instances, we can just make calls to the respective calls as shown in the below image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E56sQdM3tHAwnshB." /></figure><h3>2. Basic Operations in Elasticsearch head</h3><p>Embedded below is a short video blog on how to perform basic CRUD operations using Elasticsearch.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FN1qJZ4BqFnQ%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DN1qJZ4BqFnQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FN1qJZ4BqFnQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/65d31cf3fec696a3867dce6d9c8c2d4c/href">https://medium.com/media/65d31cf3fec696a3867dce6d9c8c2d4c/href</a></iframe><h3>Conclusion</h3><p>In this blog we have seen how to run multiple elasticsearch instances on the same machine and also the basic elasticsearch CRUD operations using the plugin elasticsearch-head.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11a2a0d0b5d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elasticsearch/elasticsearch-multiple-instances-and-head-plugin-familiarization-11a2a0d0b5d6">Elasticsearch multiple instances and head (plugin) familiarization</a> was originally published in <a href="https://medium.com/elasticsearch">elasticsearch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>