<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shira Elitzur on Medium]]></title>
        <description><![CDATA[Stories by Shira Elitzur on Medium]]></description>
        <link>https://medium.com/@shira_26754?source=rss-9f43e4b0efe2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GSZsiCoUJbtfggKjmoQkkg.jpeg</url>
            <title>Stories by Shira Elitzur on Medium</title>
            <link>https://medium.com/@shira_26754?source=rss-9f43e4b0efe2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:57:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shira_26754/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Elasticsearch —  Basic Concepts]]></title>
            <link>https://medium.com/@shira_26754/elasticsearch-basic-concepts-5fe9663edda1?source=rss-9f43e4b0efe2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fe9663edda1</guid>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Shira Elitzur]]></dc:creator>
            <pubDate>Mon, 14 Nov 2022 19:09:26 GMT</pubDate>
            <atom:updated>2022-11-14T19:09:26.494Z</atom:updated>
            <content:encoded><![CDATA[<h3>Elasticsearch — Basic Concepts</h3><p>In this post, I would like to explain some of the commonly used terminology related to Elasticsearch.<br>While I found the documentation sufficient in explaining how to work with the API, I found it a little confusing when it comes to the explanation of the different components of Elasticsearch.<br>So if you just started to work with Elasticsearch or even if you are working with Elasticsearch for a while but still confused about the terminology (cluster, node, index, shard, etc.) this post is for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtnpEG4FWr1z8k0taYdrEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*R2hjJ3goj5RcVgc8G0F9fA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtnpEG4FWr1z8k0taYdrEw.png" /><figcaption>Looking for a thing…</figcaption></figure><p>Elasticsearch is an open-source search engine, written in Java and based on the Apache Lucene search engine. It is also a document-based database and even though it is usually used exclusively for search it can also be your application&#39;s main database (depending on your application needs).</p><p>One of the advantages of Elasticsearch — and you probably know that if you are already working with the API - is that you really don’t have to fully understand each of the concepts I will describe later to start and use the API. Elasticsearch takes care of these things behind the scenes for you. <br>It just works.<br>Of course, the more you understand the more you will be able to use Elasticsearch effectively, give better search results to your application users and use your hardware resources wisely.</p><h3>Cluster and Node</h3><p><strong>A cluster</strong> is a collection of one or more <em>nodes</em> that together hold our data.<br><strong>Node </strong>is a single <strong>server</strong> in a <em>cluster</em>.</p><p>The Cluster is recognized by its name, and a Node belongs to the <em>cluster</em> that is specified in the <em>node’s</em> configuration under <em>cluster.name</em>.<br>The default name for a <em>cluster</em> is ‘elasticsearch’ the default value for <em>cluster.name</em> of a <em>node</em> is ‘elasticsearch’. <br>So if you start an Elasticsearch server using the default configurations (which you probably do if you are in your first steps of using it) — then you have a <em>cluster</em> named ‘elasticsearch’ with one <em>node</em> in it.</p><p>The <em>nodes</em> under one <em>cluster</em> are all accessible for you to index your data and search it in all the other <em>nodes</em> in the <em>cluster</em> (meaning that if one <em>node</em> receives your search request, it can forward it to the <em>node</em> that contains the data being queried).<br>You don’t specify a <em>node</em> when you send a request to Elastic, and you don’t have to know in which <em>node</em> the data you are looking for actually is.<br>You just send your request to index or to search and Elasticsearch will do its thing and give you the result.</p><h3>Document</h3><p>The basic unit. The actual data that we save in Elasticsearch. Actually, a more ‘basic’ unit would be a <strong>Field(s)</strong>, the key-value pairs that the document is a collection of them. If you are not familiar with document-based databases let’s say that document is the equivalent of a record in a relational database (and the fields would be the values for that record in each column).</p><h3>Index</h3><p>A collection of documents with similar attributes/structure. For example: a ‘users’ <em>index</em> or an ‘orders’ <em>index</em>.<br>There is no limit on the number of documents within an <em>index</em>, and there is no limit on the number of <em>nodes</em> within a <em>cluster</em>.</p><h3>Type</h3><p>Before I will explain <em>types</em> I must say that it is on its way out — <em>types</em> are deprecated in version 6+ and are not supported from version 8. <br>I include types here in this post for two reasons:<br>1. In versions lower than 6, you <strong>must</strong> use types.<br>2. The idea of this post is to discuss the concepts, and understanding what <em>type</em> was supposed to do in the first place and why it was cut out is part of this discussion.</p><p>So <em>Type</em> is used to create logical categories inside the <em>index</em>. Each <em>type</em> within the <em>index</em> has its own <strong>mapping</strong>— which is a list of the existing <em>fields</em> for documents of this type.</p><p>This role of <em>type</em> came from the original decision of Elastic developers that <em>index</em> is the equivalent of a <em>database</em> and <em>type</em> is the equivalent of a <em>table</em> in a relational database. They decided to change it because this analogy caused confusion since in Lucene’s implementation there is a connection between fields over the <em>index</em> (potentially, fields with the same name in the mapping of different <em>types</em>), a connection that does not exist in a relational database. <br>To continue with the comparison to a relational database — it is more accurate to think of this now as if the <em>index</em> is the equivalent of a <em>table</em>.<br>So if you are using Elasticsearch with a version that uses <em>type</em>, it would be better if you don’t use the <em>type</em> for logical categories inside the <em>index</em>, and use one <em>type</em> (and one <em>mapping</em>) to all the documents ‘of the same kind’.</p><p><strong>Let’s summarize what we said so far-</strong></p><p>We have a <strong>Cluster</strong> which is a collection of <strong>Nodes</strong> that are servers that receive and process our index and search requests.<br>Inside the <em>nodes</em>, there are <strong>Indices</strong> (<em>index</em> is equivalent to a <em>table</em>).<br>Inside the <em>Index</em>, we have the data. The data is the <strong>Document</strong> (equivalent to a <em>record</em>).</p><p>Now let’s dive <strong>into</strong> the <em>index</em>.</p><h3>Shard and ‘Sharding’-</h3><p>The <em>index</em> is divided into <em>shards</em>. In practical terms, our data is <strong>in</strong> the <em>shards</em>.<br>The ‘sharding’ mechanism is very important. It is what allows Elastic to distribute the data and increase the performance. <br>It allows Elasticsearch (in its inner implementation) to make sure that all the data is accessible all the time (handle hardware failure), <strong>and</strong> it allows you who manages the <em>cluster</em> to scale it to your needs easily.<br>As the <em>cluster</em> grows or shrinks Elasticsearch will re-allocate the shards so that the <em>cluster</em> will remain balanced.</p><p>You can scale by buying <strong>bigger</strong> servers (called vertical scale), or you can scale by buying <strong>more</strong> servers (horizontal scale).<br>While clearly Elastic will benefit from bigger and stronger servers, the scalability that the sharding mechanism is geared to is <strong>horizontal scale </strong>— the ability to add <strong>more</strong> <em>nodes</em>.<br>Horizontal scaling is simple in Elasticsearch because it is distributed by its definition and it knows how to handle the load of the shards between the <em>nodes</em>, and most important — our application doesn’t care how or when.</p><p>So what is this sharding you ask?</p><p>When we create an <em>index</em> we decide and set how many <strong>primary shards </strong>and how many <strong>replicas </strong>it will have.<br>By default, there are 5 primary shards and 1 replica (1 replica means that <strong>each</strong> primary shard will have one replica).<br>The replica is a full copy of all the data in the primary shard and a replica shard will never sit in the same <em>node</em> as its primary shard — so if a <em>node</em> crashes, the data is <strong>still available</strong>.<br>After you create the <em>index</em>, the number of the primary shards is fixed but you can <strong>always</strong> add replica shards.</p><p>If you are not convinced yet how great this is -<br>Every shard is a Lucene <em>index</em> that runs as a separate process. So we have parallel processing, and obviously better performance.<br>The replica shards are <strong>not only</strong> a backup, but they can also respond to search requests — so we are increasing search performance.</p><p>How the shards are distributed between the <em>nodes</em> and how the documents are distributed between the shards of one <em>index</em> is beyond the scope of this post, and while it is always good to understand, it is not at all necessary for you to know even for advanced work with Elasticsearch.</p><h4>Cluster health</h4><p>A <em>cluster</em> can be green, yellow, or red.<br><strong>Green</strong> means that all the shards are up, running, and allocated to <em>nodes</em><br><strong>Yellow</strong> means that all the <strong>primary</strong> shards are allocated.<br><strong>Red</strong> means that there are primary shards that are not allocated.<br>While it is clear what is wrong with the ‘red’ health status, notice that if you have one <em>node</em> in your <em>cluster</em> and you define replica shards in your <em>index</em> — your <em>cluster’s</em> health will be yellow. There is no point in storing copies of the same data in one <em>node</em>, if we lose that <em>node</em>, we will lose all copies of our data.<br>So for the protection of your data, just create more <em>nodes</em> (= Elasticsearch servers).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtnpEG4FWr1z8k0taYdrEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*VZ_9tSosmOMerNwNqksOmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtnpEG4FWr1z8k0taYdrEw.png" /><figcaption>Ha! Found it!</figcaption></figure><p>It took time and reading different sources for me to feel comfortable with the concepts I described above, and I hope you found my explanation clear and helpful.</p><p>You know, for search…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fe9663edda1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>