<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arif Yılmaz on Medium]]></title>
        <description><![CDATA[Stories by Arif Yılmaz on Medium]]></description>
        <link>https://medium.com/@arifyilmaz?source=rss-ae3e714d158------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cMD_aH8z9R6tcwSIEimg-A@2x.jpeg</url>
            <title>Stories by Arif Yılmaz on Medium</title>
            <link>https://medium.com/@arifyilmaz?source=rss-ae3e714d158------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:15:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arifyilmaz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Elasticsearch Scroll API vs Search After with PIT?]]></title>
            <link>https://medium.com/trendyol-tech/elasticsearch-scroll-api-vs-search-after-with-pit-1e0ec52e47ef?source=rss-ae3e714d158------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e0ec52e47ef</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Arif Yılmaz]]></dc:creator>
            <pubDate>Wed, 16 Nov 2022 07:46:01 GMT</pubDate>
            <atom:updated>2022-11-16T12:42:08.352Z</atom:updated>
            <content:encoded><![CDATA[<h3>Elasticsearch Scroll API vs Search After with PIT</h3><p>As Trendyol technology team, we have some common values. One of them is “trust in data”. We make our decisions with data. We question the ideas with data, no matter who they come from. We clarify our goals and results with numerical data. In this article, I will give our real problem and our approach to solve it while adhering our values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*wWXt1LFO92tggWSiz_VYjA.jpeg" /></figure><p><a href="https://medium.com/trendyol-tech/evolution-of-our-invalidation-system-bdfee456498b">In the last article of our invalidation system series</a>, we have mentioned some future works. One of them is to use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after">search_after</a> parameter with <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html">point in time api (pit)</a> instead of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html">scroll api</a> to use pagination in our Elasticsearch queries. In this article, I will give details about the steps of the migration from scroll api to search_after with pit and and I will show some load test results.</p><p><strong>Our Main Motivation</strong></p><p>As Trendyol Indexing Team, we have to apply all changes to products as soon as possible. Otherwise, we could display wrong information about a product and even we could sell a product with a wrong price. As a toy example, let’s consider that there is a change in the Adidas brand. We have to find all the Adidas products and apply this change to these products’ data.</p><p>The number of products can vary. Some brands may have thousands of products whereas others may have hundreds. We need to fetch these products as fast as possible. To do that, we run Elasticsearch scroll api queries and sometimes face queries that match millions of products like multiple brand, category updates etc. These queries take a lot of time to finish. When we searched about how we can speed up the running of these queries, we found that Elasticsearch does not recommend using scroll api for deep pagination.</p><blockquote>We no longer recommend using the scroll API for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (pit).</blockquote><p>To use the search_after parameter with pit we need to change our queries.</p><p><strong>search_after Parameter</strong></p><p>We need to initially give a query and sort fields.</p><pre>{<br>  &quot;size&quot;: 1,<br>  &quot;_source&quot;: false, <br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;category.id&quot;: 408<br>    }<br>  },<br>  &quot;sort&quot;: [<br>    {<br>      &quot;id&quot;: {<br>        &quot;order&quot;: &quot;desc&quot;<br>      }<br>    }<br>  ]<br>}</pre><p>The search response includes an array of sort values for each hit.</p><pre>{<br>  &quot;took&quot; : 8,<br>  &quot;timed_out&quot; : false,<br>  &quot;hits&quot; : {<br>    &quot;total&quot; : {<br>      &quot;value&quot; : 10000,<br>      &quot;relation&quot; : &quot;gte&quot;<br>    },<br>    &quot;hits&quot; : [<br>      {<br>        &quot;_index&quot; : ...,<br>        &quot;_type&quot; : &quot;_doc&quot;,<br>        &quot;_id&quot; : &quot;356550942&quot;,<br>        &quot;_score&quot; : null,<br>        <strong>&quot;sort&quot; : [<br>          356550942<br>        ]</strong><br>      }<br>    ]<br>  }<br>}</pre><p>To retrieve the next page of results, we repeat the request, take the sort values from the last hit, and insert those into the search_after array.</p><pre>{<br>  &quot;size&quot;: 1,<br>  &quot;_source&quot;: false, <br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;category.id&quot;: 408<br>    }<br>  },<br>  &quot;sort&quot;: [<br>    {<br>      &quot;id&quot;: {<br>        &quot;order&quot;: &quot;desc&quot;<br>      }<br>    }<br>  ],<br>  <strong>&quot;search_after&quot;: [356550942]</strong><br>}</pre><p>The details can be found <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after">here</a>. If an update occurs between these requests, the order of our results may change, causing inconsistent results across pages. To prevent this, we can create a point in time (pit) to preserve the current index state over our searches.</p><p><strong>Point in Time API (pit)</strong></p><p>A search request by default executes against the most recent visible data of the target indices, which is called point in time. Elasticsearch pit is a lightweight view into the state of the data as it existed when initiated. In some cases, it’s preferred to perform multiple search requests using the same point in time.</p><p>For example, if updates happen between search_after requests, then the results of those requests might not be consistent as changes happening between searches are only visible to the more recent point in time.</p><p>Before handling a query, we should create a pit with the following request:</p><pre>POST /my-index-000001/_pit?keep_alive=1m</pre><p>Here, the keep_alive parameter tells Elasticsearch how long it should keep a point in time alive. The response of this request includes an id. We should add this id to our queries with the following format:</p><pre>{<br>  &quot;size&quot;: 2,<br>  &quot;_source&quot;: false, <br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;category.id&quot;: 408<br>    }<br>  },<br><strong>&quot;pit&quot;: {<br>	    &quot;id&quot;:  &quot;46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==&quot;, <br>	    &quot;keep_alive&quot;: &quot;1m&quot;  <br>    }</strong>,<br>&quot;sort&quot;: [<br>    {<br>      &quot;id&quot;: {<br>        &quot;order&quot;: &quot;desc&quot;<br>      }<br>    }<br>  ],<br>  &quot;search_after&quot;: [356550942],</pre><pre>}</pre><p>After handling our request, we should close our pit by sending the following request:</p><pre>DELETE /_pit<br>{<br>    &quot;id&quot; : &quot;46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==&quot;<br>}</pre><p>After that point, we have run a load test to compare the performance of search_after with pit and scroll api. In order to do that, we create three simple rest endpoints. All the endpoints basically take a query string and just search all the documents. The first endpoint uses scroll api and the others use search_after with pit.</p><p>We are currently using <a href="https://github.com/searchbox-io/Jest">jest</a> client in our production environment. However, this library is deprecated. Our plan is to use <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html">Elasticsearch Java High Level Client</a> instead of jest client. In this load test, we also wanted to test the performance of these two libraries. To do that, we implemented following endpoints:</p><ul><li>scroll api endpoint(/scroll) by using jest client</li><li>search_after endpoint(/search-after) by using jest client</li><li>search_after endpoint(/high-level-search-after) by using high level client</li></ul><p><strong>Test Setup</strong></p><p>In our previous article, we have mentioned the search partitioning algorithm. This algorithm splits query using searchPartition field to get multiple queries, each hitting approximately 10k documents. You can read the details of this algorithm <a href="https://medium.com/trendyol-tech/evolution-of-our-invalidation-system-bdfee456498b">here</a>. In the test environment, we found a merchant having about 10k products. Our test query is the following:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;listings.merchantId&quot;: &quot;123&quot;<br>    }<br>  }<br>}</pre><p>We have sent this request to each endpoint 1000 times and compared each individual request response time and all 1000 requests response time for each endpoint.</p><p><strong>Results</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVfSFjMSmdbB5DpqJGW0zw.png" /></figure><p>When we compare two search after endpoints, we can see that the high level implementation’s performance is <strong>two times better</strong> than the jest client implementation. Currently, we are using the same implementation of the scroll endpoint in the production environment. When we compare scroll endpoint and high level search after endpoint, we can see that the latter decreases response time by <strong>about</strong> <strong>30%</strong>. High level search after endpoint has the best response time for almost all of the cases. At the end of the load test, we can conclude that <strong>high level search after implementation is the best</strong> among these three options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_HzdjZ4bsWTSSqSFCCRIg.jpeg" /></figure><p>In this article, I tried to explain what “trust in data” really means by giving details about our real time problem and solution. For the Elasticsearch deep pagination problem, we did not believe in even Elasticsearch documentation. Instead, we run load tests for each possible solution and choose the best one according to the test results.</p><p>Thanks for reading this far. All feedback is welcome. If you like to “<em>try”</em> with us, you could apply for our <a href="https://jobs.lever.co/trendyol?lever-via=q48u5LZ7k7">open positions</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e0ec52e47ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/elasticsearch-scroll-api-vs-search-after-with-pit-1e0ec52e47ef">Elasticsearch Scroll API vs Search After with PIT?</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of Our Invalidation System]]></title>
            <link>https://medium.com/trendyol-tech/evolution-of-our-invalidation-system-bdfee456498b?source=rss-ae3e714d158------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdfee456498b</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Arif Yılmaz]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 10:17:56 GMT</pubDate>
            <atom:updated>2022-01-18T13:33:53.707Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lsIr7TbcVxONXxD8Tm2tbQ.jpeg" /></figure><p>As Trendyol Indexing Team, we are mainly responsible to apply any changes including price, stock, promotion etc. to the content as soon as possible. We have named these changes as invalidations and we described the invalidation process <a href="https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidations-part-1-7989227cf0f2">here</a>. In this article, I will dive into Content Invalidation Api problems and how we solved them, but before I strongly recommend you to read <a href="https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidations-part-1-7989227cf0f2">this article</a> if you have not read it.</p><h4>Timeout Problem</h4><p>Content Invalidation Api (CIA) basically gets an invalidation request and generates an Elasticsearch query from this request. This query gives the contents that should be updated. Running an Elasticsearch query may take long time and this may cause timeout errors in the APIs that send invalidation request to CIA. We should had somehow prevent this errors. We have solved this problem by enqueuing request in memory and returning HTTP OK immediately.</p><h4>Terminating Process Problem</h4><p>This approach solved the timeout problem but it caused a different problem. When a CIA process terminates, requests which are currently in this process are not properly processed. It means that requests are lost forever. This was also an important problem, because we could have display wrong information about a content and this may have mislead our users.</p><p>This time solution was not tricky. We decided to separate our handling process and storing requests. In this scenario, CIA is responsible to just send incoming invalidation request to a Kafka topic. We introduced a new service Content Invalidation Dispatcher (CID) that is responsible to read invalidation requests from this Kafka topic and process these requests. This approach solved the terminating process problem, because a terminating process does not send acknowledgement to Kafka and requests that are in this terminating process will be read from one of the other precesses properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xUfg_ZYda0tVXlPr_HXBBQ.png" /><figcaption>Content Invalidation Dispatcher with Kafka topic</figcaption></figure><h4>Kafka Poll Interval Problem</h4><p>One of the most important Kafka consumer config for our application is max.poll.interval.ms. Kafka documentation explains this config as follows:</p><blockquote>The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. For consumers using a non-null group.instance.id which reach this timeout, partitions will not be immediately reassigned. Instead, the consumer will stop sending heartbeats and partitions will be reassigned after expiration of session.timeout.ms. This mirrors the behavior of a static consumer which has shutdown.</blockquote><p>The default value of this config is 300000 ms (5 min) and we used its default value. Some invalidation queries may result in millions of contents to be updated. It means that we need to fetch millions of documents from Elasticsearch and it may take more than our poll interval (5 minutes) and cause rebalancing in our Kafka topic. Unfortunately, rebalancing causes bigger problem because this <em>heavy </em>invalidation request will be processed again and again and we will not be able to process remaining requests in the same Kafka partition.</p><p>For a quick solution, we decide to change queue strategy to RabbitMQ. This gives us a temporary solution. But we could able to fight our main problem. RabbitMQ does not work like Kafka. It does not have partitions. A RabbitMQ consumer may process a message in a long time and other consumers can continue to process other messages. We replace Kafka topics with RabbitMQ queues and this really solved this problem.</p><h4>Long Running Elasticsearch Queries</h4><p>Content Invalidation Dispatcher with RabbitMQ queues solution solved most of our problems. However, we still have long running queries. This queries indirectly affect other invalidation request and we have solved it programatically. Firstly, we used <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results">Elasticsearch scroll api</a> to implement pagination in CID. However, Elasticsearch documentation does not recommend scroll api, if we are paging more than 10.000 hits. As I said before, we occasionally scroll millions of hits and scroll api is not suitable for these cases. Additionally, deep pagination causes increase in Elasticsearch search latency and this indirectly affects other invalidation requests.</p><p>In order to solve this problem, we should have split queries in a way that the result of the queries have approximately 10.000 hits. We defined a new field <em>searchPartition</em> to our Elasticsearch mapping and set its value by using modularity of ID that spread our contents. (ID % 1000) This assures that each document will have a searchPartition value between 0 and 999. We used searchPartition field to split queries. Our split algorithm can be explained briefly as follows:</p><ol><li>If the total number is greater than 10.000 in the query result, do not continue to invalidation process.</li><li>Run an aggregation query over searchPartition field with size 1000 and order key.</li><li>Group aggregation bucket in a way that each group should have approximately 10.000 elements.</li><li>Add range filter over searchPartition field to the original query for each group and send it to our partition RabbitMQ queue.</li><li>Read these partitioned requests and apply them properly.</li></ol><p>Lets give a toy example:</p><p>Assume that we have following query:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;category.id&quot;: &quot;100&quot;<br>    }<br>  }<br>}</pre><p>The total value is 20.000 for this query. Now, we will run this aggregation query:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;category.id&quot;: &quot;100&quot;<br>    }<br>  }, <br>  &quot;aggs&quot;: {<br>    &quot;searchPartitionAggs&quot;: {<br>      &quot;terms&quot;: {<br>        &quot;field&quot;: &quot;searchPartition&quot;,<br>        &quot;size&quot;: 1000,<br>        &quot;order&quot;: {<br>          &quot;_term&quot;: &quot;asc&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>Let’s say the aggregation result is as following:</p><pre>&quot;buckets&quot; : [<br>        {<br>          &quot;key&quot; : 0,<br>          &quot;doc_count&quot; : 2600<br>        },<br>        {<br>          &quot;key&quot; : 1,<br>          &quot;doc_count&quot; : 7000<br>        },<br>        {<br>          &quot;key&quot; : 2,<br>          &quot;doc_count&quot; : 500<br>        },<br>        {<br>          &quot;key&quot; : 3,<br>          &quot;doc_count&quot; : 1000<br>        },<br>        {<br>          &quot;key&quot; : 4,<br>          &quot;doc_count&quot; : 8900<br>        }<br>]</pre><p>Now, we group these buckets:</p><pre>group1 =&gt; keys: 0, 1, 2 and total count:10.100<br>group2 =&gt; keys: 3, 4 and total count: 9.900</pre><p>The partitioned queries are now as follows:</p><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;match&quot;: {<br>            &quot;category.id&quot;: &quot;100&quot;<br>          }<br>        },<br>        {<br>          &quot;range&quot;: {<br>            &quot;searchPartition&quot;: {<br>              &quot;gte&quot;: 0,<br>              &quot;lte&quot;: 2<br>            }<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><pre>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        {<br>          &quot;match&quot;: {<br>            &quot;category.id&quot;: &quot;100&quot;<br>          }<br>        },<br>        {<br>          &quot;range&quot;: {<br>            &quot;searchPartition&quot;: {<br>              &quot;gte&quot;: 3,<br>              &quot;lte&quot;: 4<br>            }<br>          }<br>        }<br>      ]<br>    }<br>  }<br>}</pre><p>As you can see, original query is split into two queries and these queries will hit 10.000 documents which is recommended limit for scroll api.</p><p><strong>What is Next?</strong></p><p>At this point, we have actually solved all problems. However, we have other considerations. As Trendyol tech team, we prefer using Kafka topics rather than RabbitMQ queues because of our future multi data center scenarios. We are planning to use active-active Kafka clusters in different data centers. Firstly we will give a chance to CID with Kafka topics solution since we solved long running Elasticsearch query problem with searchPartition strategy.</p><p>Elasticsearch documentation recommends <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after">search_after</a> parameter for pagination with <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html">point in time api (PIT)</a>. We will use this parameter and api in the implementation of CID. Currently, we are using Elasticsearch version 6.8.6. However this version does not support PIT feature. We will also upgrade our Elasticsearch clusters to version 7.</p><h4><strong>Conclusion</strong></h4><p>In this article, I tried to explain the problem we faced in our invalidation environment and our step by step solutions. Our company grows fast and we always face new problems. We sometimes solved them by writing some code and sometimes by changing the system. Each problem gives us new experiences and we want to share our experiences with the community.</p><p>Thanks for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdfee456498b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/evolution-of-our-invalidation-system-bdfee456498b">Evolution of Our Invalidation System</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Handle Millions of Invalidation — Part 2]]></title>
            <link>https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidation-part-2-7f7cd3fb7373?source=rss-ae3e714d158------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f7cd3fb7373</guid>
            <category><![CDATA[trendyol]]></category>
            <category><![CDATA[e-commerce-solution]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Arif Yılmaz]]></dc:creator>
            <pubDate>Mon, 21 Dec 2020 09:57:20 GMT</pubDate>
            <atom:updated>2020-12-28T10:26:24.409Z</atom:updated>
            <content:encoded><![CDATA[<h3>How We Handle Millions of Invalidation — Part 2</h3><p>Previously, I explained the invalidation process from Rabbitmq queues to Kafka topics. I highly recommend you to read <a href="https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidations-part-1-7989227cf0f2">Part-1</a> of this series, if you have not read it. In this article, I am going to describe how we create content data and sync our ElasticSearch and Couchbase data sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpiWJLry0r9fsG-AzRZxUw.png" /><figcaption>Figure 1: Invalidation Process</figcaption></figure><p>From previous part of this series, we know that Content Invalidation Api(CIA) sends content ids to different Kafka topics according to the invalidation type of the request. This strategy brings two advantages to us.</p><ol><li>While we are immediately applying some invalidations like price or stock changes, we can postpone other invalidations like content meta data changes.</li><li>Invalidations with different reasons are processed separately. Therefore, they do not affect each others. For example, if there are large amount of content meta changes we can still apply stock changes or any other type of changes. None of the invalidation type waits others to be applied.</li></ol><h3>Perses</h3><p>Perses was the Titan God of destruction. We have named this service as Perses, because it destroys old content data and adds the new. Perses takes three environment variables. These variables directly affect how Perses works.</p><ul><li><strong>application</strong>: This variable indicates which type of invalidation will be processed. Possible values are <em>Immediately, PriceImmediately, PromotionImmediately, StockImmediately, SupplierImmediately</em>,<em> Hourly, Translation, PartialUpdate, Reindex</em></li><li><strong>channel</strong>: Currently, you can also do grocery shopping at Trendyol. Market(grocery) contents are processed separately from other contents. This variable indicates whether market contents will be processed. Possible values are <em>trendyol, market</em>.</li><li><strong>culture</strong>: At the moment, Trendyol website is available at 30 countries with three languages: Turkish, English and German. For each country, we just have one available language. The website is Turkish in Turkey, German in Germany and English in the remaining 28 countries. Culture parameter indicates both language and country. Possible values are <em>tr-Tr, international</em>. When Perses starts with <em>international </em>culture, it handles all cultures except <em>tr-Tr</em>.</li></ul><p>When Perses starts, it checks these three variables and determine which Kafka topic will be listened accordingly.</p><p>Some of the invalidations are applied immediately and price, stock, supplier and promotion invalidation are applied immediately and separately. When application variable is <em>Immediately, PriceImmediately, PromotionImmediately, SupplierImmediately </em>or<em> StockImmediately </em>Perses starts to listens its Kafka topic immediately.</p><p>Some of the invalidation types are not need to be applied immediately. In this case CIA sends them to a special Kafka topic. This Kafka topic is determined by the hour of the day. For example, if this kind of invalidation is send to CIA at between 10.00 A.M and 11.00 A.M, CIA send it to <em>productcontent_11_tr-TR</em> topic.(assume that culture is tr-TR) If the application is <em>Hourly, </em>Perses starts listening to previous hour’s topic. For example, at 10.00 A.M it starts listening to <em>productcontent_9_tr-TR topic. </em>A content can be send to hourly Kafka topics for different reason. We are applying Kafka compaction strategy to hourly topics’ partitions. Compaction enabled partitions holds only one message for a specific key. We mark content id as the key of the message thus hourly topics do not hold a content more than one. This is important because each content in hourly topics will be invalidated just one time. We eliminate unnecessary invalidations. You can find more detail about Kafka compaction in <a href="https://kafka.apache.org/documentation/#compaction">here</a>.</p><p>A content may be translated to German or English. According to the which language content is translated to, CIA sends the content id to either <em>productcontent-translation_de-DE</em> or <em>productcontent-translation_en-GLB </em>topic. When application is <em>Translation, </em>Perses starts listening these two Kafka topics<em>.</em></p><p><em>PartialUpdate</em> and <em>Reindex</em> application types will be explained later in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yxlc0EZxHwotW6cfrGX1Sg.png" /><figcaption>Figure 2: Invalidation Process in Perses</figcaption></figure><p>In Figure 2 you can see the details of invalidation process in Perses. When Perses reads a content id from Kafka, it directly sends it to Product Content Api Gateway (will be called gateway here after) with culture parameters given as environment variable. Gateway constructs the content data and sends it back to Perses. Finally, Perses saves the content data to Product Content Couchbase.</p><p>If an exception occurs for a message in Perses, Perses sends this message to the exception topic of it’s main topic. This exception topic is listened at every 20th minute of each hour. If an exception occurs for the same message again, this time Perses sends this message to another exception topic. This topic is listened at every 50th minute of each hour. If an exception occurs again, Perses will not try to apply this invalidation and send an error message instead.</p><p>In Figure 2 you can also see that Perses reads three Kafka topic. In order to do that, we use Actor Systems of Akka. For each topic we run a different actor called Content Invalidation Actor and each actor listens one topic separately. You can find more details about Actor systems in<a href="https://doc.akka.io/docs/akka/current/general/actor-systems.html"> here</a>.</p><h3>Product Content Api Gateway</h3><p>A product content data consists of various fields like name, price, stock, merchants, promotions, size etc. Product Content Api Gateway collects all of the fields from different APIs and datasources and creates product content data. All of the dependencies of gateway is briefly described below:</p><ul><li><strong>Inventory Api: </strong>Content’s default and scheduled prices are given by this api.</li><li><strong>Stock Api</strong>: Quantities of content’s all sizes are given by this api.</li><li><strong>Pim Read Api</strong>: Content meta data like name, description, brand, category, all sizes, all attributes and all listings are given by this api.</li><li><strong>Web Brand Category Api: </strong>Content brand, category and gender details are given by this api.</li><li><strong>Assortment Api: </strong>This api is used to determine whether a brand/category is available in a country or not.</li><li><strong>Promotion Api: </strong>All available promotions of a content are given by this api.</li><li><strong>Tag Api: </strong>We can add a tag to a content to display it in different ways. This api returns all of the tags of the given content.</li><li><strong>Translation Api: </strong>Gateway uses this api to get translated content information when it is called for countries other than Turkey.</li><li><strong>Delivery Date Api: </strong>This api returns all of the sellers’ delivery dates.</li></ul><p>Other than these APIs, gateway also fetch some fields from different Couchbase buckets. Suppliers, content score and rating score are taken from these buckets.</p><p>As you can see, gateway has a lot of external dependencies. We must try to parallelise these api calls and data fetching process from Couchbase buckets to decrease the response time. As the title of this article says, we handle millions of invalidations every day and we do that without affecting user experience. Therefore, decreasing a millisecond from the response time of gateway really matters a lot. However, we cannot parallelise each job, because sometimes a field taken from an api call may be a parameter for a different api call and similarly we may fetch data from Couchbase bucket by using a field taken by an api call. We first do independent jobs and do dependent ones by using the result of these finished jobs.</p><p>A content may be sold by multiple sellers. When a user search this content in the Trendyol application or website, we show this content for just one seller. Otherwise, this content would be shown again and again. In order to choose which seller will be shown, we apply an algorithm called buybox algorithm. After fetching all of the data it needs, gateway apply buybox algorithm for each supplier and generate buybox score. The details of buybox algorithm is out of context for this article.</p><h3>Couchbase ElasticSearch Connector</h3><p>As we explained in Part-1 of this series, we use ElasticSearch in two services. Firstly, our search services use ElasticSearch and secondly Content Invalidation Service(CIS) and Content Invalidation Api(CIA) use ElasticSearch to query and find content id from other fields like listing id, brand id etc.</p><p>When Perses updates a document in Couchbase, we should also update ElasticSearch so that Couchbase and ElasticSearch datasources remains sync. In order to update documents in ElasticSearch we use Couchbase ElasticSearch Connector (CBES). CBES replicates your documents from Couchbase Server to ElasticSearch in near real time. The connector uses the high-performance Database Change Protocol (DCP) to receive notifications when documents change in Couchbase. You can find details of the Couchbase ElasticSearch Connector from this <a href="https://docs.couchbase.com/elasticsearch-connector/current/index.html">link</a>.</p><p>Product detail services, search services, CIA and CIS have different needs. When CBES receives an updated content data, we transform it into a different models to meet search services, CIA and CIS needs.</p><h4>Partial Update</h4><p>For most of the invalidation types, we build content data from scratch because some of the fields are related and if we update a field we should update it’s related fields. For example, if a seller makes a discount on his product his buybox score for that product increases. Buybox score is also related with other fields like stock. Thus, we should also get the current stock of product for that seller to recalculate the buybox score. However, some fields are not related with any others. For example, content score or content tags are not related with any other fields. If these fields are updated, we can just update these fields on the content data.</p><p>When Perses is started with <em>PartialUpdate </em>application type. It starts listening to partial update Kafka topic. When Perses reads a message from this topic, it firstly fetch whole document from Couchbase and just update the changed field and then upsert this document to Couchbase. Fetching content data from Couchbase is much faster than getting it from gateway. Thus, partial updates take less amount of time from full updates.</p><h4>Reindex</h4><p>As we offer new features at Trendyol application, new fields may be added to content data. In this case we should update all contents to add these new fields. Sometimes, we made a mistake in our development phase and set wrong value to a field at all of the contents. In this case, we firstly should fix our mistake and update all fields.</p><p>In these two cases, we need to update all contents’ data and we named it reindex. When we need reindex, we firstly fetch all content ids from ElasticSearch and send them to a special reindex topic. When Perses is started with <em>Reindex </em>application type, it starts listening to this reindex topic. The rest is the same with the invalidation process. When Perses reads a content id from this topic, it gets content data from gateway and insert this content data to Couchbase. When an update is occurred in Couchbase, CBES takes this updated content, converts it to our ElasticSearch model and send it to ElasticSearch.</p><h3>Conclusion</h3><p>In this article, I tried to explain how we build content data and sync our two datasources. As we growing rapidly our total number of products and the number of invalidation at each day are increasing. Our system should be able to handle these rapid increase. We are still working to decrease invalidation time. In the future, I may write an article about our changes and fixes on our invalidation process.</p><p>Thanks for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f7cd3fb7373" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidation-part-2-7f7cd3fb7373">How We Handle Millions of Invalidation — Part 2</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Handle Millions of Invalidations — Part 1]]></title>
            <link>https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidations-part-1-7989227cf0f2?source=rss-ae3e714d158------2</link>
            <guid isPermaLink="false">https://medium.com/p/7989227cf0f2</guid>
            <category><![CDATA[e-commerce-solution]]></category>
            <category><![CDATA[trendyol]]></category>
            <dc:creator><![CDATA[Arif Yılmaz]]></dc:creator>
            <pubDate>Wed, 09 Sep 2020 07:09:21 GMT</pubDate>
            <atom:updated>2020-12-20T13:39:26.700Z</atom:updated>
            <content:encoded><![CDATA[<h3>How We Handle Millions of Invalidations-Part 1</h3><p>At the moment, we have more than 20 millions different sellable products in Trendyol. At any time, any product can be sold, new products can become sellable, new campaigns may be opened, new promotions can be applied. Therefore, product data can be changed at any time. We have to apply these changes to products as soon as possible. Otherwise, we could display wrong information about a product and even we could sell a product with a wrong price. As Trendyol tech team, we call these product data changes as “content invalidation” and we are able to handle millions of invalidations every day and in these article series we are going to describe how we handle these millions of invalidations without affecting user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpiWJLry0r9fsG-AzRZxUw.png" /><figcaption>Figure 1: Invalidation Process</figcaption></figure><p>We can describe content as product metadata like name, description etc. and all of its related information like price, stock, promotions etc. As Trendyol technology team, we store content data in two different data sources for different purposes. We use ElasticSearch for our search services and campaign detail service which provides search functionality inside a boutique. We also use Couchbase buckets for our product detail service. We should apply each invalidation to both our ElasticSearch and Couchbase data sources in order to be consistent at each page of Trendyol website and application.</p><h3>Content Invalidation Service</h3><p>Content invalidation can be in different ways. For example, the price or the stock of a product may change etc. Every invalidation type has different significance. We have to apply price invalidation as soon as possible whereas we can postpone stock invalidation.</p><p>Each invalidation starts with an invalidation message. These messages are sent to different Rabbitmq queues based on their invalidation types. Invalidation messages can be in different formats but all of them have either a content id or a listing id. Content id is the unique identifier of each content. Listing can be basically described as the combination of merchant, size and fulfillment type of a content. Each product can be sold by multiple merchants and each listing contains product size and merchant information. For example, if a merchant with id 100 sells S and M size of a product and another merchant with id 200 sells the same product’s M and L size, we store four listings and each of them contains merchant and size information. The details of the listing is out of context for this article. Listing id is the unique identifier of each listing. We can find the content id of the content which has a listing with the given listing id. We are sure that a listing can be in only one content thus an invalidation message with a listing id will invalidate only one content. How we find the content id from a listing id is again out of context for this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R5dVpjYt3bOU0KQ7cBn7YA.png" /><figcaption>Figure 2: Invalidation flow from Rabbitmq queues to Kafka topics</figcaption></figure><p>Content Invalidation Service (CIS) listens each Rabbitmq queues to get invalidation messages. Each invalidation message is transformed to a different type of message and sent to a different api called Content Invalidation Api (CIA). The details of each queue is described below.</p><p><strong>Management of Assortment Changes</strong></p><p>Currently, Trendyol website is available in both Turkey and Germany and we are still working for other countries. However, every product is not available in every country. The availability of a product in a country is determined according to the product’s brand and category. If a brand/category becomes available for a country, all of the products of that brand/category become available for that country. Similarly, if a brand/category becomes unavailable for a country, all of the products of that brand/category become unavailable for that country. As Trendyol tech team, we called this brand and category availability as “brand category assortment”. CIS listens this assortment changes and sends them to CIA with reason as “assortment”.</p><p><strong>Management of Price Changes</strong></p><p>Merchants may set different prices for specific products and specific time intervals. With this way, they can easily apply their discounts early. Our inventory service holds these scheduled prices and default price. If a content does not have any scheduled price, the default price is applied to the product. CIS listens to these inventory changes and sends them to CIA with reason as “inventory”.</p><p><strong>Management of Content Metadata Changes</strong></p><p>When there is a change in the metadata of the content like name, description etc. an invalidation message is sent to this queue. CIS listens to these changes and sends them to CIA with reason as “product”.</p><p><strong>Management of Content Translation</strong></p><p>At the moment, Trendyol has two different languages: Turkish and German. We are still working on English website. All of the contents are initially Turkish and they are translated to German and English. Each translated content is sent to this queue with the language information. CIS listens to this queue and sends them to CIA with reason as “translation” and additionally with language information.</p><p><strong>Management of Promotion Changes</strong></p><p>Merchants are able to define promotions to their products. Whenever a product is created or updated we should apply it to the defined products. CIS listens to all of the promotion change events and sends them to CIA with reason as “promotion”.</p><p><strong>Management of Stock Changes</strong></p><p>CIS listen to new stock creation events and stock update events. Stock creation events are sent to CIA with reason as “stock”. However, each stock update event may not have the same importance. For example, a stock change from 100 to 99 is not critical, thus it is not necessary to invalidate. By doing this, we aim to decrease the number of invalidations we have to handle. It may be asked how we decide that a stock change is critical. In order to make this decision, we set a critical stock quantity (At that moment its value is 3). When the new stock of a content after the change is less than or equal to this critical quantity, it is more possible to be out of stock and it is decided as critical. We have to display out of stock information about contents to users thus we have to handle these invalidations. We send each critical stock update event to CIA with reason as “stock”.</p><h3><strong>Content Invalidation Api</strong></h3><p>Content Invalidation Api (CIA) basically sends content ids to Kafka topics based on the invalidation type. Before diving into the details let’s discuss the invalidation request parameters of CIA.</p><ol><li><em>parameterIds</em>: List&lt;Long&gt;</li><li><em>parameterName</em>: String</li><li><em>isNestedType</em>: boolean</li><li><em>nestedPathName</em>: String</li><li><em>invalidationType</em>: InvalidationType</li><li><em>indexKeys</em>: List&lt;IndexKey&gt;</li><li><em>invalidationTime</em>: Long</li></ol><p>CIA sends only content ids along with invalidation reason to Kafka topics. However the invalidation request may not have content ids. It can have for example brand ids, category ids, listing ids etc. In order to get content ids, CIA sends ElasticSearch queries and fetches corresponding content ids.</p><p>First four parameters are used to create ElasticSearch queries. <em>parameterIds</em> can be content ids, brand ids, category ids etc. <em>parameterName</em> indicates the type of parameter ids. This should be exactly the same name in the ElasticSearch mapping. For example, content ids are stored in “id” field in ElasticSearch index, therefore <em>parameterName</em> should be “id”. Some of the fields are nested fields in ElasticSearch mapping. <em>isNestedType</em> indicates whether this parameter is nested or not. If it is nested, CIA sends a nested ElasticSearch query. <em>nestedPathName</em> gives the nested path. It is used only if <em>isNestedType</em> field is true. The details of nested types and nested query can be found at this <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html">link</a>.</p><p>InvalidationType is an enum used to get the reason of the invalidation. This enum also holds topicGeneratorKey which is used to find the corresponding Kafka topic. The possible values are listed below:</p><ul><li>PROMOTION</li><li>ASSORTMENT</li><li>BRAND_CATEGORY</li><li>SUPPLIER</li><li>TAG</li><li>LABEL</li><li>CONTENT_SCORE</li><li>RATING_SCORE</li><li>FIT_RATIO</li><li>STOCK</li><li>PRICE</li><li>INVENTORY</li><li>PRODUCT</li><li>CAMPAIGN</li><li>CONTENT</li><li>IMMEDIATELY</li><li>TRANSLATION</li></ul><p>Currently, Trendyol website is available in Turkey and Germany. We are still working for different countries with English language. IndexKey is an enum used to indicate language and country of the invalidation request. Turkish contents and other contents (called international content) are stored in different ElasticSearch indexes. This enum also holds this index information. Currently available values are listed below:</p><ul><li>trTR</li><li>deDE</li><li>enAT</li><li>enBE</li><li>enBG</li><li>enHR</li><li>enCZ</li></ul><p>Finally <em>invalidationTime</em> is a unix timestamp of the request time. It is used when an exception occurs. The details will be given later.</p><p>When a request is sent to CIA, it first finds the content ids which will be invalidated. To do that, it checks <em>parameterName</em> parameter. If it is “id” then it means that <em>parameterIds</em> parameter stores these content ids. <em>parameterName</em> can be also “brand.id”, “category.id”, “listing.id” etc. In this case, CIA should find the corresponding content ids. To do that, it sends ElasticSearch query by using <em>parameterIds</em>, <em>parameterName</em>, <em>isNestedType</em> and <em>nestedPathName</em> parameters. To which index this query will send is found by <em>indexKey</em> parameter.</p><p>After finding content ids, CIA should find the correct Kafka topic to send content ids and invalidation reasons. In order to find the correct topic name, CIA uses <em>invalidationType</em> parameter and <em>indexKey</em> parameter. For example, a request with <em>invalidationType</em> as PRICE and <em>indexKey</em> as trTR is sent to <em>productcontent_price_tr_tr</em> topic, whereas a request with <em>invalidationType</em> as STOCK and <em>indexKey</em> as deDE is sent to <em>productcontent_stock_de_de</em> topic. All of the combinations of <em>invalidationType</em> and <em>indexKey</em> can be sent with a request and each of them are sent to their corresponding Kafka topic.</p><h3>Exception Handling</h3><p>It is clear that we should apply each invalidation as soon as possible. However, there can be an exception at any time. We should handle exceptions and still apply invalidations. In order to do that, we follow the steps described below.</p><ol><li>If there is an exception, CIA takes the request and checks whether the request has invalidation time. If it does not have the invalidation time, CIA set the current time as invalidation time and sends it to Rabbitmq queue.</li><li>This queue has a ten minutes TTL. After ten minutes, the requests in this queue are sent to another Rabbitmq queue. This queue is listened by our service called invalidation request exception handler.</li><li>Invalidation request exception handler listens request from this queue and checks whether the listened request is sent in the last 2 hours. If the request is sent in the last 2 hours, it sends it to CIA. With this way, we try to apply these exceptional invalidation requests after 10 minutes from exception time.</li></ol><p>Each Rabbitmq queue which is listened by Content Invalidation Service has dead-letter queue. When an exception occurs for a request in CIS, this request is send to corresponding dead-letter queue. We periodically move messages from dead-letter queue to the listened queues. Thus we are still able to handle all invalidations.</p><h3>Conclusion</h3><p>In this article, I tried to explain the invalidation process from Rabbitmq queues to Kafka topics. I will continue by explaining how we create content data and how we sync ElasticSearch and Couchbase datasources in Part 2.</p><p>Thanks for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7989227cf0f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/how-we-handle-millions-of-invalidations-part-1-7989227cf0f2">How We Handle Millions of Invalidations — Part 1</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>