<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oktay Koçer on Medium]]></title>
        <description><![CDATA[Stories by Oktay Koçer on Medium]]></description>
        <link>https://medium.com/@oktaykocer?source=rss-328fb2d36dd7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NrU7OgVpKf6zIGh5a7emOw.jpeg</url>
            <title>Stories by Oktay Koçer on Medium</title>
            <link>https://medium.com/@oktaykocer?source=rss-328fb2d36dd7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:52:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@oktaykocer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Taking Seller Product Indexing Team to the Future: Re-platforming and Beyond at Trendyol]]></title>
            <link>https://medium.com/trendyol-tech/taking-seller-product-indexing-team-to-the-future-re-platforming-and-beyond-at-trendyol-081c9f948d1a?source=rss-328fb2d36dd7------2</link>
            <guid isPermaLink="false">https://medium.com/p/081c9f948d1a</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[replatforming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[e-commerce-solution]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Oktay Koçer]]></dc:creator>
            <pubDate>Wed, 19 Mar 2025 05:53:52 GMT</pubDate>
            <atom:updated>2025-03-19T05:53:52.122Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uqBRAZytEiJqcjZWh1hpBA.png" /></figure><p>The Seller Product Indexing (SPI) team at Trendyol embarked on an ambitious journey to re-platform our core systems responsible for managing and displaying sellable product data. This complex undertaking involved addressing numerous challenges, including outdated architecture, performance bottlenecks, and intricate dependencies. We invite you to delve into the details of our re-platforming process, exploring the obstacles we faced, the strategies we employed, and the ultimate impact of our efforts. We extend our sincere gratitude to the entire SPI team the author of this article, and other key contributors, such as <a href="https://medium.com/u/7b7e307ee151">Elif Seray Dönmez Çelik</a>, <a href="https://medium.com/u/cab7a906f51e">Burhan Günaydın</a>, <a href="https://medium.com/u/a18b67d1309d">Ahmet Tarık Güler</a>, <a href="https://medium.com/u/7ebbb29834d">Mert Bulut</a>, <a href="https://medium.com/u/77397c9e27">Abdulsamet İLERİ</a>, <a href="https://medium.com/u/328fb2d36dd7">Oktay Koçer</a>, <a href="https://medium.com/u/ebfb17edcea6">Kerem Can Kabadayı</a> and <a href="https://medium.com/u/d9ca6ef0c6dd">Emre Turgut</a> for their invaluable contributions and dedication throughout this transformative project.</p><h3>Introduction</h3><p>The Seller Product Indexing (SPI) team operates under Product Offering and Catalog. Its primary function is to <strong>index sellable Product</strong> data, enabling the display and management of products for sellers through the Seller Center platform. In addition, it provides sellable Product data to various stakeholders such as Global Platforms, Seller Campaign, Buyer Indexing, and others.</p><p>To achieve this, the team collaborates with multiple teams, including Inventory, Seller Store, Seller Base, Listing, Product Core, Rule Management, Productivity, Data Science, and Data Engineering, ensuring that updates are reflected in <strong>near real-time</strong>.</p><p>The SPI team is dedicated to developing innovative solutions that enhance efficiency, improve user experience, and deliver business requirements with the highest performance.</p><p>Currently, <strong>more than one billion</strong> ProductContent-Culture documents are available for sale or can be made available for sale on Trendyol. The high volume of updates occurring within this structure is referred to as invalidation, and the SPI team is responsible for managing these invalidations. Essentially, all the necessary fields that should be present on a seller-facing Product document (such as stock, price, variant details, and scoring) are included. As a result, Product data is highly dynamic and subject to continuous updates, handling <strong>hundreds of millions of events</strong> per day.</p><p>The team operates entirely on an <strong>event-driven architecture</strong>. One of its primary challenges is ensuring that updates from various teams affecting product documents are reflected in near real-time. Moreover, since the final aggregated version of a product’s data is first established by this team, it assumes responsibility when business operations need to be built on top of this data and incorporates various business features accordingly.</p><h3>Before Replatforming</h3><h4>Our Pre-Replatform Applications</h4><p>In the following sections of the article, to avoid confusion regarding the application name, we explained our applications and their functions before the replatform process under the headings below. Our overall architecture before the replatform process is illustrated in the visual below.</p><figure><img alt="General diagram of our Pre-Replatform Applications" src="https://cdn-images-1.medium.com/max/853/0*HttZeleQqFDDg3Uc" /><figcaption>General diagram of our Pre-Replatform Applications</figcaption></figure><h4>Enricher API</h4><p>It was a monolithic application that listened to data from 64 different topics, such as listing, price, stock, content, category, brand, business unit, and campaign, and wrote this data to Couchbase. After writing the data related to content into Couchbase, it also sent the relevant updates to the topics listened to by other teams.</p><p>The application processed part of the messages from various topics by listening to fat payloads and writing them directly to Couchbase. For others, it used the ID information from the event to retrieve the updated data via the relevant team’s API and then wrote it to Couchbase.</p><p>In scenarios where certain topics had a high load, we used an in-memory cache (Redis) and a Couchbase-based cache structure to manage the traffic.</p><p>In summary, the application performed multiple tasks: listening to events, making API calls to the relevant teams based on the incoming event, writing documents to Couchbase, and sending update messages to other teams that listened for our data changes.</p><h4>Invalidation API</h4><p>In updates that affected multiple content types, such as brand, category, or campaign, the enricher API sent the relevant ID to the invalidation API. Using Elasticsearch, it retrieved the content ID of the document related to the brand, category, or campaign ID and sent it back to the enricher API. The enricher API then partially updated the Couchbase document (instead of rewriting the entire document, it only wrote the modified sections with the source data).</p><p>The updated data in Couchbase was written to Elasticsearch through our couchbase-to-elastic (cbes) application, which listens to Couchbase DCP events (cbes — <a href="https://github.com/Trendyol/go-dcp-elasticsearch">GitHub link</a>).</p><p>We stored our Couchbase documents based on content IDs, allowing us to handle created, updated, and deleted content and write these changes to Elasticsearch. Additionally, in cases where incorrect data was written to Couchbase, we utilized specific endpoints to send invalidation requests for the desired documents, enabling corrections with the accurate data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/0*dGx-Jf96e9aF1LMy" /><figcaption>Invalidation API working structure</figcaption></figure><h4>Supplier API</h4><p>This application listened to seller information and updated the Couchbase document with fields such as supplier badges, account, logos, and permissions. After collecting seller-related information from different topics, we stored it in an in-memory cache to avoid creating unnecessary load on the seller team’s endpoints.</p><h4>Comparator API</h4><p>We developed the comparator API application to quickly identify missing or incorrect data in our documents. This application featured an endpoint that accepted a content ID, fetched the existing document from Couchbase, and compared it with the correct data retrieved from the source endpoints of various teams. It returned the detected discrepancies as a response.</p><p>Additionally, the comparator API had an <strong>invalidate</strong> parameter. When set to <strong>true</strong>, it triggered invalidations for the missing data. These invalidations were listened to by the enricher API, which then queried the relevant source for the data, mapped it, and updated the document accordingly. This application allowed us to respond swiftly to many support requests but did not directly contribute to resolving the root problems.</p><p>The comparator API could also find related content based on a given Elasticsearch query, compare it with the source data, identify discrepancies or errors, and send them to the enricher API to write the correct data. However, this approach often fell short in achieving the desired throughput (TP). Furthermore, the process created additional load on the teams’ endpoints since both the comparator and enricher applications queried them twice.</p><h4>Challenges We Faced Prior To the Replatforming</h4><p>Our pre-replatform applications had numerous flawed processes that hindered the team’s ability to develop new features and required the entire team to focus on resolving issues. The challenges we faced before the replatform process included:</p><ul><li><strong>Team Motivation and Integration of New Features<br>- </strong>Before the replatform process, one of the main challenges we faced was the overwhelming number of support requests from different teams. Resolving these issues required us to put aside all ongoing work and focus entirely on troubleshooting.<br>- This situation led to delays in completing sprint tasks and often resulted in sprint failures. To adapt, we had to abandon the sprint model and switch to a Kanban workflow, allowing us to manage the constant influx of support tasks more effectively. However, this shift also impacted team motivation and hindered the integration of new features.</li><li><strong>Lack of Unit and Integration Tests<br>- </strong>Almost none of our applications had any tests in place. When new code was developed, we relied entirely on manual testing. This was primarily because the existing code in our applications was not designed to be testable.<br>- As a result, even after manual testing, we occasionally encountered errors post-deployment. Moreover, we couldn’t bring up our applications end-to-end in a local environment to debug issues effectively. Instead, manual tests were conducted in the staging environment, requiring us to temporarily shut down the staging applications.<br>- Often, after completing tests, we forgot to restart the staging applications, causing disruptions and preventing other teams from progressing with their own tests.</li><li><strong>Inadequate Application Performance<br>- </strong>The performance of our applications was insufficient, particularly when dealing with data from different teams that needed to be written to a single document. This often resulted in race conditions due to events arriving out of order. Since we couldn’t identify the root cause of these errors, we resorted to creating scheduled jobs within our applications to correct the incorrect data in the documents and rewrite it to Couchbase — a process we referred to as reindexing.<br>- Additionally, our Kafka consumers were not designed to process data concurrently. This limitation made it difficult to handle load spikes, as the lag in our consumers would grow quickly under high traffic. Even when we scaled our applications to match the number of topic partitions, we still couldn’t achieve the desired throughput (TP) during heavy load scenarios.</li><li><strong>Inability to Recreate Incorrect Documents from Scratch<br>- </strong>When we had incorrect documents, we needed to fix the data within them. However, we were unable to recreate the entire document from scratch with all its data (full reindex). For instance, if we discovered that the variantId within the content was incorrect, the listing document with that variantId would continue to persist with outdated information.<br>- To correct the data, we had to initiate separate processes for the content and listing data. Of course, since changing the listing information also affected stock and price data, additional reindexing operations were triggered. In practice, we were sending partial reindex requests one after the other in an attempt to correct the document. This approach proved inefficient and difficult to manage.</li></ul><figure><img alt="Partial document structure" src="https://cdn-images-1.medium.com/max/190/0*ZNvbdDLmFU9SY5Sa" /><figcaption>Partial document structure</figcaption></figure><ul><li><strong>Infrastructure Problems in Our Applications<br>- </strong>Each of our consumers had retry and exception topics, but this setup wasn’t working as expected. Messages arriving in the retry topics were retried within seconds, only to be immediately moved to the exception topic. For reasons we couldn’t fully understand, the exception topic either wasn’t functioning properly or was significantly delayed. This led to unprocessed data and generated support requests from other teams.<br>- These problems were exacerbated as new developments were added, and the number of issues continued to grow. Before the replatform process, we had 400 million documents, and this number was steadily increasing (currently 1.5 billion). There were also errors and slow performance in the invalidation processes.</li><li><strong>Presence of Couchbase-Based Cache Structures<br>- </strong>One of the frequent sources of issues in our applications was the cache structure used in our enricher API. We had decided to set up a structure that would help us later on by listening to events from domain topics. We created Couchbase buckets to handle mappings based on the varying event types for different domains. The key reason behind this decision was our reluctance to put additional load on the APIs of different teams, as those teams often struggled to handle the load.<br>- Each time an update occurred, we needed to retrieve the source data by calling the respective API. However, the technological infrastructure used by the teams could not support this additional load, and migrating to a more suitable system would take time. We stored a variety of data, such as ListingId-VariantId, VariantId-ContentId, ContentId-CategoryId &amp; BrandId, and ContentId-Translated, in a key-value format. As new content was added, the volume of data grew.<br>- When relevant events arrived, we would perform upsert or delete requests to the Couchbase bucket. However, this setup caused events to be processed slowly. Additionally, for reasons we couldn’t pinpoint, it led to incorrect data being stored and inaccurate updates being made to content.</li><li><strong>Insufficient Alerting and Monitoring<br>- </strong>Before the replatform process, our applications had weak alerting, monitoring, and logging capabilities. We had set up alerts through Elasticsearch Kibana error logs for applications that exceeded certain thresholds, ensuring that messages were sent to our alert channels. However, many of these alerts did not contain actionable messages, and the alert channel became overwhelmed, making it easy for important issues to be overlooked without taking appropriate action.<br>- We didn’t have Grafana dashboards in place, so we couldn’t quickly view topic or exception consumer group lags. Instead, we had to check through Kafka UI applications. The Kafka consumer group lag alerts were the same across all topics (one million), which meant we could miss situations that required immediate action.<br>- Additionally, to identify issues in our applications, we had to debug manually. Kibana logs were not sufficient for us to detect errors. We lacked an end-to-end system monitoring solution, preventing us from effectively tracking and diagnosing problems across the entire system.</li><li><strong>Insufficient Logging<br>- </strong>In addition to the frequent support requests, we had developed an audit API application. This application listened to invalidated messages that we produced via Kafka for different teams and wrote them to an Elasticsearch audit index. This allowed us to track when and with which parameters we sent the events.<br>- However, this application used the same Elasticsearch cluster as our search and invalidation applications. As a result, it created a significant load on Elasticsearch, particularly during reindexing scenarios, which could cause performance issues. Furthermore, the data in Elasticsearch was being provided to the team working on the seller screens. Errors in Elasticsearch would directly lead to complaints from sellers. In hindsight, if our logging infrastructure had been better, we might not have needed such an application at all.</li><li><strong>Pod-Based Cache Structure<br>- </strong>To avoid creating load on other teams’ APIs or because some of our endpoint needs were not being met, we used a memory cache structure within our enricher API application. We developed this system for domain topics such as category, brand, campaign, and attributes, which could cause frequent updates to multiple content types. We utilized Redis pub-sub features for this.<br>- Instead of directly processing incoming events, we stored the relevant IDs and necessary information as metadata in our Couchbase bucket. A scheduler was used to process any unprocessed metadata in this bucket and send Kafka messages to our invalidation API. Once the record was processed, we marked it as complete in the metadata bucket. The invalidation API then handled the incoming message and, using Elasticsearch, found the content IDs related to the category, brand, or campaign IDs. It would send Kafka messages with the content IDs to the enricher API.<br>- The enricher API would fetch the relevant category, brand, or campaign data from the memory cache and update the content document accordingly. Of course, to ensure that the cache remained up-to-date, we had a cache invalidation scheduler that ran every 10 seconds. It would pull data from the respective team’s API or from Elasticsearch indexes and update the cache. If no data was found, it would remove the cache entry.<br>- Maintaining this structure was challenging. The cache system could lead to outdated data being written to documents, and it also caused events to be processed slowly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QvlJTOovvNQiDDnP" /><figcaption><strong>Pod-Based Cache Structure</strong></figcaption></figure><ul><li><strong>A lack of a solution-oriented approach and ignoring problems<br>- </strong>We recorded the events where we experienced race conditions (which we referred to as problematic events) into an Elasticsearch index. A scheduled job within the enricher-api would fetch the relevant data from this index and send a message to the topic consumed by the comparator API. The comparator API would process the message and, if any discrepancies were found, send it back to the enricher-api. The enricher-api would then query the source data, map it, and update the document with the correct information. While this approach allowed us to resolve the errors, we were never able to identify the root cause.</li><li><strong>Inability to Address Cases<br>- </strong>We were wasting a lot of time on support requests due to insufficient logs, overly complex designs making applications untraceable, and the lack of local environment setups. Beyond just wasting time, our investigations were not reaching any conclusions.</li><li><strong>Inability to Perform Full Reindexing, Invalidation Load<br>- </strong>Whether we address our cases or not, the data needs to be corrected in any case. The processes we refer to as reindexing or invalidation essentially express similar concepts. While reindexing involves updating all document data from the sources, invalidation can occur for specific partial reasons, such as PRICE or STOCK, etc. During this process, we did not have a process we could call reindexing. When we needed to invalidate multiple data points on a document, we had to initiate multiple partial invalidations. This meant an enormous load for both our source teams and Kafka. To illustrate, if there are N suppliers on a document and I invalidate the document, it creates 1 invalidation load. However, if I invalidate the N suppliers, it generates N events. Thus, the number of invalidations increased to Nx based on the amount of data we needed to correct. I would also like to share an incident story related to this. One day, when we initiated around 68 million invalidations, we had produced 8 billion events and put the entire Trendyol at risk. In such cases, we were forced to reduce the retention of the topics. This could cause the listening teams to miss events if they couldn’t consume the topics at the expected speed.</li></ul><h3>Beginning of Replatforming</h3><p>The team’s workload has been significantly impacted by planning efforts and the high demands of Lucid Business, with half of the team members focused on these tasks. Additionally, we have been engaged in discussions with other domain teams regarding endpoint requests needed for our replatforming efforts, requiring multiple meetings.</p><p>Another major challenge arose due to other teams’ inability to efficiently handle high-traffic product data loads, such as brand and category information. As a result, we had to maintain some of this data within our own cache.</p><p>The primary reason behind the replatforming decision was the inability of our existing applications to scale effectively. However, several other <strong>critical factors</strong> also influenced this decision:</p><ul><li>High case support workload on the team</li><li>Case tracing complexities</li><li>Weaknesses in monitoring processes</li><li>Legacy monolithic codebase</li><li>Quick-fix solutions implemented in a hacky manner</li><li>Lack of unit, integration, and automation tests</li><li>Insufficient documentation</li><li>Caching difficulties</li><li>High resource consumption</li><li>Absence of a proper reindexing process</li><li>Event duplication issues (where a single event is propagated as multiple events to different teams)</li><li>Direct access to other teams’ databases</li></ul><p>These structural problems made it necessary for us to transition to a <strong>modern microservices architecture</strong> and an <strong>event-driven</strong> model. Since each of these issues is highly detailed, we will not go into further depth here.</p><h4>Initial Steps and Approach</h4><p>Before starting the replatforming process, our first task was to conduct a cost analysis and define a weekly effort plan. We carefully documented the new architecture and design after a thorough analysis. Both the current (“as-is”) and future (“to-be”) structures were mapped out using Lucid diagrams, and all documentation was prepared accordingly.</p><p>Following this, we completed an in-depth analysis of the monolithic application’s business features and database-level dependencies. Based on these findings, we devised a <strong>step-by-step release</strong> and <strong>rollback plan</strong>. To ensure a smoother and safer transition with minimal effort, we decided to deploy a fully parallel second system alongside the existing one.</p><p>Given the continuous influx of business requirements and the company’s vision for this domain, a business freeze was not an option. Therefore, we split the team into two subgroups, working on replatforming tasks within the same sprint but under separate workgroups. With this approach, three developers remained focused on ongoing tasks, while the other three worked exclusively on the replatforming efforts.</p><h4>Challenges in Replatforming</h4><p>One of the most complex aspects of the replatforming process was dealing with the existing application’s in-memory cache solutions and its deep integration with other teams at the database level. To address this, as well as to ensure service-based access to specific data during indexing, we engaged with dependent teams for required changes.</p><p>After conducting a detailed requirements analysis, we shared our findings with the relevant teams. To ensure an asynchronous yet effective tracking process, we established weekly meetings.</p><p>With each newly deployed feature in the replatformed system, we gradually eliminated legacy structures such as direct database access and legacy caching mechanisms. Instead, we integrated with the relevant teams’ event-driven architectures or services.</p><p>For cases where certain teams could not provide <strong>high-throughput</strong>, <strong>low-latency services</strong>, we implemented an <strong>event-sourcing architecture</strong> to maintain efficiency. Additionally, we introduced new endpoints and services, incorporating event notification mechanisms where necessary to enhance system <strong>reliability</strong> and <strong>maintainability</strong>.</p><h3>Through Replatforming</h3><p>During our re-platforming journey, we followed the below strategies.</p><p><strong>Strategy 1: Use Event Notification Over Event Carried State Transfer</strong> <br>+ Notification is easy to implement because no need to check before or after state change or entity version. We expect to get the latest data from the API provided.<br>+ Initial migration and reindex scenarios are easy.<br>+ Storage is low.<br>- Synchronous Read lookup required.</p><p><strong>Learnings: </strong>When we got production for some parts of the system, we had some issues. When we listened to the incoming event and went to the API, we realized that the API was not returning the most up-to-date data. It gives stale data. <em>(API gives the data read replica in SQL DB).</em> Therefore, before the implementation, it is necessary to shake hands on this subject.</p><p><strong>Strategy 2: Merge events wherever you can<br></strong>As the SPI team, we have processed millions of events daily. To reduce the write rate to our database, we applied merging based on key, or some value within the event. For example, if we look at the below figure, we reduced the processed number of events from 5000 to 1504.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y92OiHyYoqxwvHtK" /><figcaption>Figure: Example merging events</figcaption></figure><p><strong>Strategy 3: Use the Same Key Compaction technique to reduce the number of processed messages</strong></p><p>We are processing some types of events at night. (<em>Because it creates pressure on the system, night is more safe for us.</em>) For example, in the below figure, we see category events. <em>A category can hit millions of events so compaction is crucial here!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0XlRG8d2BgVdfbev" /><figcaption>Figure: Example Category events</figcaption></figure><p>For the same number of categories, let’s look at how compaction affected</p><ul><li>Before merging: (we had to stop the consumer :()</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YsKfSZ31neLDWbST" /><figcaption>Figure: Category messages without the same key compaction</figcaption></figure><ul><li>After merging:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*So7-Y86DJcSm71Tv" /><figcaption>Figure: Category messages with the same key compaction</figcaption></figure><p><strong>Strategy 4: Implement worker pool pattern on consumers</strong></p><p>We can scale horizontally in Kafka based on the topic’s partition count. We also wanted to increase our application performance. We used a worker pool pattern and it gives a really good performance (CPU and memory are also low). Here are the<a href="https://gist.github.com/Abdulsametileri/2b2a142f1ff9d85e31b07598ecf6dcf3"> implementation</a> and results 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rMb1FXOX8orfA5Rp" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TxU1pJbTk7uD3D0i" /><figcaption>Figure: 26 pods (26 partitions) + have two IO calls</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZBN73trOMNghgLIK" /><figcaption>Figure: Above consumer CPU and Memory metrics</figcaption></figure><p><em>Memory seems too much, but this app connects five different couchbase clusters. Every cluster has a 15mb x node count allocated heap for the connection buffer. For example, if we have 20 nodes on the cluster, it means a 300 MB buffer allocated on startup. For five clusters 1500mb allocated.</em></p><p><strong>Strategy 5: For the other strategies (as a part of it shown below) and real use cases, please look at</strong><a href="https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3"><strong> the this article</strong></a><strong>.</strong></p><ul><li><a href="https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3#b122">Pause/Resume consumer any time</a></li><li><a href="https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3#e97b">Error-Only Retry for batch consuming</a></li><li><a href="https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3#f3c0">Skip Message By Header Feature</a></li></ul><p>Let’s see these strategies result</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*HAi-hYhnWFYSePXQ" /><figcaption>Figure: 30 pods (30 partitions) + have one IO operation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*cavxCrqpq_LDpEvk" /><figcaption>Figure: 15 pods (15 partitions) + have two IO operations</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YesY2agvMDhfkmk9" /><figcaption>Figure: 16 pods (16 partitions) + have two IO operations</figcaption></figure><h4>Dynamic config, secret, MultiDC, TBP adaptation (service maturity)</h4><p>Our company adheres to service maturity policies, with application-specific maturity scores that every team strives to elevate to “Gold.” One of our replatforming goals was to enhance our applications by aligning them with these policies.<br>Initially, all our configurations were statically embedded within the application. Making changes required a new commit and deployment — a tedious process, especially for changes needing immediate impact. To address this, we began storing frequently updated feature flags and business configurations in a dynamic config system. This allowed us to quickly and easily update configurations via a user interface while retaining the history of changes, akin to Git commits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*2UNF17-6G98u8Dmo" /><figcaption>Example Business Config</figcaption></figure><p>We created application-specific dynamic config files to define rules such as whether to skip processing an event based on its Kafka message type, or whether to process it immediately or through scheduled jobs. These settings could now be adjusted on the fly without requiring new deployments.</p><p>Security was another critical aspect. Sensitive information like usernames and passwords, previously stored in application code, was moved to a dynamic vault. This approach allowed us to update secrets, such as Kafka or Couchbase credentials, directly through an interface without restarting the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*E1jQZsM6UEDstqlZ" /><figcaption>Example Secret Config</figcaption></figure><p>Availability is paramount for a system like Trendyol, which handles massive loads. Ensuring minimal disruption during unexpected issues required robust multi-data center (MultiDC) support.</p><p>Our in-house Trendyol Builder Platform (TBP) made achieving MultiDC functionality straightforward. It facilitated automatic rollouts, ensuring smoother transitions. We migrated all our Argo-based applications to TBP and began deploying them across multiple data centers and clusters.</p><p>Additionally, we configured new Couchbase clusters as server group clusters, enabling automatic load redirection between data centers during issues. For Kafka, we leveraged rack awareness to ensure each application connected to brokers in its local data center.</p><h4>Unit Test / Integration Test</h4><p>At the start of the replatforming process, the average test coverage of our projects was below 10%. Truthfully, testing had not been a priority. The absence of a dedicated developer-in-test, insufficient unit tests, and a lack of integration and automation testing habits made it increasingly difficult to validate new features. Deploying changes to production became riskier and more stressful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*yE6KXHDJ0rFMefKo" /><figcaption>Sonar Analysis of a Legacy Project</figcaption></figure><p>Determined not to repeat these mistakes, we established strict rules for replatformed projects. For every new consumer and API, we committed to writing integration tests and maintaining at least 85% unit test coverage.</p><p>Initially, this approach slowed down development due to the team’s unfamiliarity with such practices. It felt like reinventing the wheel, as we lacked prior examples. However, once we established a rhythm, everything became significantly easier. Writing tests allowed us to identify and address our own errors, which proved invaluable.</p><p>We began holding monthly meetings to review test coverage, code smells, and critical violations in each project. These discussions highlighted areas for improvement and celebrated successes. Additionally, we integrated Sonar into our GitLab pipelines, ensuring that all code adhered to predefined standards and exceeded the minimum coverage threshold before deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/0*WnaFun33Hmhukwbm" /><figcaption>Sonar Analysis of a Replatformed Project</figcaption></figure><p>One of our replatformed applications, Enricher, was particularly large, involving nine consumers listening to business topics. These consumers processed events to create product data, which was exposed via a Gateway API. The Gateway handled complex business logic, interacted with various APIs and Couchbase clusters, and generated critical product data visible to buyers and sellers.</p><p>Given the high stakes, we took extra care to write detailed integration tests for all data sources and comprehensive unit tests for every piece of business logic. The smallest mistake could have had significant repercussions for both buyers and sellers, so this meticulous approach was non-negotiable.</p><h4><strong>Functional Test</strong></h4><p><strong>Automation Test</strong></p><p>Before starting to write test automation, we collaborated with <a href="/service/mailto:onur.erdemiroglu@trendyol.com">Onur Erdemiroğlu</a> to determine the structure we needed to establish. The Gateway product has a lot of data, and with the strategies (reasons) we use, it can also return specific data from the product. Therefore, we needed to create automation that could cover all scenarios here. To prepare the necessary data by GW as realistically as possible, we mocked all the services the API interacts with during startup and data fetching. Similarly, we set up the databases the API depends on and added the necessary data to these databases. We achieved all this in isolated test environments developed within the company. Additionally, we built a data structure that is automatically generated alongside the tests, aligned with our business flow within the automation. This approach ensured that each test could run independently, and the data produced in one test would not affect the others.</p><p><strong>Differ Tool</strong></p><p>We realised that the automation tests we wrote could not cover all scenarios. The product data is quite extensive, and we couldn’t find data that covered every case. In our normal process, we write the products to a Couchbase bucket. To address this limitation, we created a test bucket alongside the original one and ensured that the data provided by the newly developed GW application was written into this test bucket. In this way, we were able to generate both the “as-is” product data and the product data created by the replatformed application. We expect these two datasets to be identical. To compare these datasets, we used a differ tool.</p><p>The <strong>DifferTool</strong> is an API response comparison tool developed by our teams at Trendyol. It takes two different requests as input, sorts their JSON responses, and compares them. If there are differences in the responses, it generates a UI to highlight the differing points. This allows us to quickly identify the reasons behind any discrepancies. For the DifferTool, we provided the endpoints that fetched the products from both the old and the new structures as inputs. We made the parameter values for these two endpoints programmatically adjustable. Consequently, we were able to quickly check thousands of products this way.</p><h4>Performance Test</h4><p>After completing the tests in the new flow we established with the newly developed GW application, it was time to evaluate the performance of this flow. For performance testing, we used the <strong>Ares tool</strong>, developed by our Test Platform team. Ares provides a UI that enables us to run performance tests. Through this UI, we were able to run our performance tests quickly and easily.</p><p>Once we achieved the expected metrics, we proceeded with the release plan.</p><h4>How Did We Deploy Our New Architecture to Production?</h4><p>When creating our replatform plan, we analyzed the domain of our team and divided it into smaller components (price, stock, supplier, listing, etc.). The order in which these components were deployed to production was crucial. For example, we prioritized developments in price, stock, or supplier over listing because they were dependent on the listing data. Price, stock, and supplier information could be stored in separate fields within documents in Couchbase, but when a listing is created for the first time, the price, stock, and supplier data also need to be written to the document. This is because stock and price data are created with the listing, while supplier information can exist without the listing.</p><p>As part of the replatform plan, we decided to continue using one of our legacy applications, the <strong>invalidation-api</strong>. This application connects to Elasticsearch and allows us to find the content ID of our documents by using domain-specific IDs such as category, brand, and campaign. The found content IDs are then sent to <strong>indexer</strong>, one of the replatform applications, for writing into Couchbase. In the previous structure, we were sending data to Couchbase via <strong>enricher-api</strong>. However, the tasks of fetching source data, writing to Couchbase, and sending updates to other teams were handled in this project. Therefore, as we developed replatform components, we replaced the parts in <strong>invalidation-api</strong> that sent data to <strong>enricher-api</strong> with those that sent it to <strong>indexer</strong>, using feature flags. In case of any issues, we could switch the feature flag back to false to restore the old functionality. We also conducted safe testing using these flags in our staging and production tests. Additionally, with replatform, we deployed the newly developed code directly to production and only enabled the feature flag portion of the task. This helped us avoid complexity in MR and deployment sequences.</p><p>While deploying new applications to production as part of the replatform, we focused on the following:</p><ul><li><strong>Feature Flags</strong>: We used feature flags in our new applications to ensure that newly deployed applications wouldn’t affect the live environment or create additional load.</li><li><strong>Indexer Configuration</strong>: To make testing easier, we added certain Couchbase features to our <strong>indexer</strong> application (such as UpsertSpecOptions, StoreSemantic, and test buckets) or disabled some (such as CAS).</li><li><strong>Branch Strategy</strong>: Some of our applications proceeded directly via feature branches without merging into the development branch.</li><li><strong>Deployment Order</strong>: We decided on the order in which applications should be deployed to production to prevent double processing and to ensure the proper shutdown of applications.</li><li><strong>Post-Deployment Actions</strong>: After deployment, we carefully planned actions for potential unwanted behaviors, such as incorrect data being written to documents or lag in new applications. We needed to be prepared to scale if necessary and quickly switch to the old application if there were issues that could impact customers.</li><li><strong>Monitoring and Alerts</strong>: We defined alerts for our applications before deploying to production. This helped us identify problematic situations quickly. We also disabled old application alerts to avoid false positives, ensuring that only the critical issues were flagged.</li><li><strong>Release Planning</strong>: To ensure all these aspects were covered, we decided to create detailed release plans before deployment. Using the lessons we learned from deploying components, we improved our plans for future deployments.</li></ul><p>Before going live, we had an example release plan, as illustrated below, which helped us ensure smooth transitions to production.</p><h4><strong>Example Supplier Release Plan</strong></h4><ul><li><strong>Invalidate Message Consumer:<br>- </strong>Remove “x” from supplier config (replace “x” with the actual value).</li><li><strong>Indexer:<br>- </strong>Enable caching (cas).<br>- Remove diff bucket.<br>- Remove UpsertSpecOptions{CreatePath: true}.<br>- Remove StoreSemantic.</li><li><strong>Supplier Change Consumer:<br>- </strong>Set businessconfig invalidation to true.<br>- Verify that messages are arriving in the Invalidation supplier topic.</li><li><strong>Enricher API:<br>- </strong>Merge the SEL-1141 branch.<br>- Verify that messages are arriving in the Indexer supplier topic.</li><li><strong>Invalidation API:<br>- </strong>Enable the sendSupplierInvalidationsToIndexer flag.<br>- Verify that SupplierAccountUpdated and PermissionUpdate messages are being sent to the Indexer topic.<br>- Shut down the supplier-api.<br>- Verify that updates are reflected in content documents.<br>- If no issues are encountered, remove the parts of the enricher that feed data to the supplier-api topics. (This is not critical and can be done later.)</li></ul><p><strong>Failure Plan</strong></p><ul><li><strong>If data is incorrect:<br>- </strong>Enable the indexer to write to the diff bucket.<br>- Revert the supplier-api topics to the release time and reprocess them.</li><li><strong>In case of lag:<br>- </strong>Scale pods.<br>- Investigate parts that can be parallelized.<br>- Adjust concurrency.</li></ul><h4>Incidents After Migration</h4><p>Even with a controlled replatforming process, large-scale developments can still lead to significant incidents. Unfortunately, despite our best efforts, we couldn’t avoid some incidents during our migration.</p><p><strong>Price Stock Incident</strong></p><p>One of the first incidents we encountered during the replatform process occurred when we were migrating the price and stock domains. In our legacy system, <strong>enricher-api</strong> was listening for updates and performing partial updates on our documents, particularly for price and stock information. These values were tied to listings, so they were directly associated with listing data.</p><p>In our new structure, we created an application called <strong>price-stock-consumer</strong> that listens to topics from the inventory team. This application sends the <strong>listing ID</strong> to our <strong>invalidation-api</strong>. The <strong>content ID</strong> corresponding to the listing ID is then fetched from Elasticsearch and forwarded to the <strong>indexer</strong> application for writing to Couchbase.</p><p>After the information reaches <strong>indexer</strong>, we query the <strong>inventory team’s API</strong> to get the latest price or stock data associated with that listing. The data is then written to Couchbase, ensuring it’s up-to-date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/0*4VfrFfhYv-hAxSUD" /><figcaption>Replatorfom price stock consumer general diagram</figcaption></figure><p>After implementing this structure, the first issue we encountered was a backlog in the invalidation-api topics. Due to this topic lag, the seller screens that were listening to our updates were affected. Initially, we had created only one topic each for TR and international. We didn’t anticipate the load that could occur here. After noticing this issue, we rolled back the changes to mitigate the crisis. Later, we created separate topics for price and stock for both TR and international, and added the necessary applications.</p><p>The second issue was that we didn’t test the end-to-end flow thoroughly in the staging environment. We couldn’t fully test the logic we implemented for price and stock in the gateway. In addition to the API calls for price and stock, the other teams also needed information such as whether the price/stock data was already present in the document or if it was being created for the first time. Since we didn’t test this part adequately and hadn’t written integration tests, we encountered issues like price/stock data being deleted or not updated in the document.</p><p>Finally, since price and stock data are linked to the listing, and at that time, we hadn’t started the replatform flow for listings yet, the enricher-api continued to handle the listing flow. When a listing was created for the first time, we should have triggered the price and stock invalidation for that listing as well. However, we hadn’t implemented this business logic in the gateway. As a result, we couldn’t reflect the price and stock data for newly created listings. To solve this issue, we had developed an endpoint in the invalidation-api to find the content IDs using the required Elasticsearch query and send them to the indexer. We then wrote back all content without price and stock information to our documents.</p><p><strong>Listing Incident</strong></p><p>The most comprehensive transition during our replatforming process was undoubtedly the migration of the Listing flow. Beyond updating the metadata for listings, we also had to ensure seamless updates to stock, price, and supplier flows whenever listings were added or removed from a product. The criticality and complexity of this flow led us to introduce integration tests at this stage.</p><p>However, things didn’t go as smoothly as planned. Despite updating the listings, we accidentally indexed some stock and price data as zero, preventing sellers from tracking their updates properly.</p><p>When we deployed the new Listing flow to production, the first issue we encountered was the inability of external services to handle the increased load. Shifting to new endpoints after replatforming introduced unforeseen challenges. For example, we discovered that our backend, powered by Elasticsearch, now required chunked requests to operate efficiently. Initially, our chunks were too large, which caused the Elasticsearch cluster to crash. Another issue arose from pagination inconsistencies, leading to missing listings in our responses. To resolve these problems, we reduced the chunk sizes.</p><p>Our system relied on two separate endpoints, each owned by a different team and fetching data from distinct sources. This caused discrepancies: some listings were successfully returned by one endpoint but not the other, resulting in missing listings.</p><p>A scenario we overlooked during planning involved listings being written onto content for the first time. Normally, when a Listing Create event arrived, we would retrieve the related content directly from the event. However, since we missed this detail, we treated these events like others and queried Elasticsearch to locate the content. Unfortunately, as no listings existed on the product yet, these queries returned no results, causing us to miss Listing Create events.</p><h3>After Replatforming</h3><p>Enricher is not only API, it has Kafka consumer and producer. Direct comparison with gthe ateway is not appropriate but by decomposing multiple consumers and a single gateway API, this is what we achieved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*unS891-S_gwaZfOy" /><figcaption>Enricher TP</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nY3q5xvKSSlM3Xar" /><figcaption>Gateway TP</figcaption></figure><p>The enricher API shows a lower overall throughput (tens of thousands of requests/minute) with occasional spikes, consistent with its Kafka-driven consumer/producer architecture. Meanwhile, the gateway API handles significantly more traffic (in the millions of requests/minute).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QN5OSfnDrCRE3c3j" /><figcaption>Enricher CPU</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NUhptS6-FUasf-ii" /><figcaption>Gateway CPU</figcaption></figure><p>You can see a broader CPU range (up to ~4 cores) with multiple pods contributing to frequent spikes, but settling back to a lower baseline. In the second, usage generally stays around or below ~1.5 cores, with clearer bursts from certain pods but an overall steadier pattern than in the first time window.</p><h4><strong>Results</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IfCKrQviorGQEGHf" /><figcaption>Enricher CPU</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ulcPRzZYuzPTPRQr" /><figcaption>Gateway CPU</figcaption></figure><ul><li>You can see a broader CPU range (up to ~4 cores) with multiple pods contributing to frequent spikes, but settling back to a lower baseline. In the second, usage generally stays around or below ~1.5 cores, with clearer bursts from certain pods but an overall steadier pattern than in the first time window.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zYlIdTqoLpWaxxU_" /><figcaption>Enricher Memory</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PjzD9cX-UBQA4V0K" /><figcaption>Gateway Memory</figcaption></figure><ul><li>Memory usage often sits in the 4–6 GiB range with some pods pushing up to around 10 GiB, reflecting multiple services each holding their memory overhead. In the second, the gateway baseline is closer to 3–4 GiB, with occasional spikes but generally lower overall usage. The fact that the gateway connects to 11 Couchbase buckets (each allocating its default heap allocation) helps explain some of that baseline memory consumption.</li></ul><h4><strong>Overall Impact</strong></h4><p>By decomposing the single enricher API service into separate Kafka consumers and a dedicated gateway API, we’ve achieved:</p><ul><li>Clearer resource usage and scaling boundaries</li><li>More consistent CPU and memory consumption</li><li>Lower latency and fewer hotspots on the gateway side</li><li>Stable (and expectedly higher) resource usage on the enricher side, matching its expanded responsibilities</li></ul><p>This architecture not only makes the system simpler to reason about and scale but also keeps each component focused on a smaller set of tasks — leading to improved reliability and performance across the board.</p><h4><strong>Increasing Test Coverage</strong></h4><p>During the re-platforming process, we paid close attention to certain aspects while developing. Integration tests helped cover the gaps left by the lack of automation tests and made manual testing much easier.</p><p>However, we also needed to increase the automation test coverage on the business code. To achieve this, we utilised the Code Coverage integration developed within Trendyol. We incorporated goals aimed at improving this coverage into our objectives.</p><h4>Addressing Cases and Simplifying the Invalidation Process</h4><p>With the replatforming, we started to easily track events between applications with a structure suitable for the microservice we built. Detailed consume and produce logs, debuggable applications, and local setups have been key factors that facilitated tracing during this process. By improving the invalidation and reindexing processes, we were able to easily correct support-requiring and erroneous cases. Previously, fixing the invalidation of 1 content produced 20–30 events, but now we’ve reduced that to 1 event, and we have roughly accelerated our invalidation processes by 20x.</p><p><strong>The problems experienced in external services we use have come to light. Their inability to keep up with the new TP. The emergence of errors that went unnoticed in the old system.</strong></p><p>The replatforming process resolved some architectural design flaws in the old system. These changes allowed the system to operate more efficiently and enabled faster processing. However, these improvements led to an increased load on other services. While bottlenecks in the old system limited the load, the new system’s faster and more efficient structure resulted in other services handling more operations.</p><p>In the face of this increased load, other teams had to update their technologies and code. Although replatforming initially solved a set of issues that only needed to be addressed by the SPI team, the increased efficiency of the system caused similar problems to arise in other services due to the higher load. This also brought to light structural flaws that had previously gone unnoticed…</p><h4>Automation Tests</h4><p>After the replatforming process, our goal was to ensure that the replatformed applications could be tested quickly and reliably, allowing potential bugs to be identified before deployment. In this section, we will share the step-by-step approach we followed and the strategies we adopted.</p><p>1. <strong>Creating a Roadmap and Identifying Critical Applications</strong></p><p>The team had a wide range of applications under development. Therefore, prioritizing testing after the replatforming process was the first step. We quickly reviewed all applications to identify the most critical ones and those posing the highest functional risks. This was an essential step to efficiently allocate resources and optimize the timeline as much as possible.</p><p>2. <strong>Setting Up the Automation Infrastructure</strong></p><p>At this stage, we established the automation infrastructure to accelerate the development process. Burhan Günaydın and Onur Erdemiroğlu had already developed a framework for automation tests. In this setup, tests ran in an isolated test environment we built in-house, with test data being automatically generated. By maintaining this structure, we simplified the tests further using the Spring and JUnit frameworks.</p><p>3. <strong>Preparing and Prioritizing Test Cases</strong></p><p>Once critical applications were identified and the automation infrastructure was set up, it was time to start coding test cases. We first pinpointed the critical flows within the critical applications and outlined test scenarios for these flows. This approach aimed to achieve maximum test coverage in the shortest possible time.</p><p>4. <strong>Writing Tests and Integrating Them into the CI/CD Pipeline</strong></p><p>After defining the test cases, we began automating each one step by step. We integrated our test projects into the CI/CD pipelines, ensuring that automation tests were executed with every code change. This enabled the detection of potential issues at an early stage, significantly improving the overall development process.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fK7KLl0Ak9QWjOC8" /></figure><p>The re-platforming of the Seller Product Indexing system at Trendyol has been a transformative endeavor, requiring meticulous planning, cross-team collaboration, and a deep understanding of our intricate systems. By successfully navigating the challenges and implementing innovative solutions, we have not only improved the performance, scalability, and maintainability of our systems but also fostered a culture of continuous improvement and learning within the SPI team. The re-platformed SPI system now stands as a robust foundation for Trendyol’s continued growth and success in the dynamic e-commerce landscape.</p><p><strong>Join Us</strong><br>We’re building a team of the brightest minds in our industry. Interested in joining us? Visit the pages below to learn more about our open positions.</p><p><a href="https://careers.trendyol.com/tr?source=post_page-----35584eb2b733--------------------------------">Home - Trendyol Careers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=081c9f948d1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/taking-seller-product-indexing-team-to-the-future-re-platforming-and-beyond-at-trendyol-081c9f948d1a">Taking Seller Product Indexing Team to the Future: Re-platforming and Beyond at Trendyol</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>