<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ankur Ranjan on Medium]]></title>
        <description><![CDATA[Stories by Ankur Ranjan on Medium]]></description>
        <link>https://medium.com/@ranjanankur?source=rss-5ab3898edc74------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZeHPWfW-6ayDOlCv4T-aSA.png</url>
            <title>Stories by Ankur Ranjan on Medium</title>
            <link>https://medium.com/@ranjanankur?source=rss-5ab3898edc74------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:38:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ranjanankur/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Inside Razorpay’s RTO Model: The Engineering Behind its ML Lifecycle]]></title>
            <link>https://engineering.razorpay.com/inside-razorpays-rto-model-the-engineering-behind-its-ml-lifecycle-999fe75e66ec?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/999fe75e66ec</guid>
            <category><![CDATA[return-to-origin]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ml-engineering]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 10:00:31 GMT</pubDate>
            <atom:updated>2026-10-16T09:25:48.976Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RjCoENQ5T_6FZ84CW0_i_Q.png" /></figure><p>Contributor: <a href="https://medium.com/u/26d6601b9fcb">Abhik Debnath</a></p><p>In this article, we use <strong>Razorpay’s Return to Origin (RTO)</strong> risk model to explain the ML lifecycle: how we build a reproducible release, deploy it into checkout, monitor its behaviour, and use later outcomes to evaluate whether it needs retraining. We connect these stages through one decision: whether cash on delivery should be available to a customer.</p><p>A customer reaches checkout and looks for Cash on Delivery (COD). Offering it can help the merchant complete the sale. But if the <strong>order comes back undelivered</strong>, the merchant may pay for shipping in both directions and have inventory tied up along the way. Hiding COD has a cost too: a customer who would otherwise place the order might leave without buying.</p><p>That return journey is called Return to Origin (RTO). Razorpay supports <a href="https://razorpay.com/docs/payments/payment-methods/cod/">COD on its Checkout page</a> and uses an RTO risk model to help inform whether that option should appear. The model scores the checkout before the customer chooses a payment method; a separate product policy combines the score with other rules to decide whether to show COD.</p><p>The model produces a score at checkout, but delivery provides the outcome later. Connecting the two requires knowing which inputs were available, which release produced the score, and which policy acted on it. That connection lets us investigate a production decision and learn from its outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eUnX-Nrit4AB4gEmkvw-hQ.png" /><figcaption><em>The RTO decision sits between checkout and payment selection. COD availability is a product decision informed by the model score; an order is placed only if the customer completes checkout.</em></figcaption></figure><h3>A score is only half the decision.</h3><p>The model estimates RTO risk for an order if COD is offered and chosen. A policy turns that estimate into a checkout action. Keeping those responsibilities separate lets us change a threshold without retraining the model.</p><p>Suppose a checkout receives a score of 0.65. In a simplified policy, COD is hidden when the score meets or exceeds the threshold. At a threshold of 0.60, this checkout loses COD. Move the threshold to 0.70, and COD stays available. The model and its input have stayed exactly the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uiyexWSQMCYQDVwUdprlqw.png" /><figcaption>Moving the boundary changes the action. The blue marker is a fixed, synthetic risk score, not a calibrated 65% probability of return.</figcaption></figure><p>That boundary also changes what we can learn later. If a customer places a COD order, fulfilment can tell us whether it was delivered or returned. If COD is hidden, there is no COD order whose outcome we can observe. A lower observed return rate may reflect a different population of orders, so it cannot by itself establish that the model improved.</p><p>This is why the lifecycle needs to preserve the policy as well as the model. To explain a checkout decision later, we need to know which score was produced, what that score meant, and which rule acted on it.</p><h3>Build: Reconstruct what checkout could know.</h3><p>For RTO, building a training dataset starts with a historical question: what information was available when the customer reached checkout? Order history and aggregates can help estimate risk, but a dataset rebuilt today contains information that yesterday’s serving system may never have seen.</p><p>Consider a checkout at 2:00 p.m. An event occurred at 1:55 p.m., but its feature value became available to serving at 2:05 p.m. Looking only at the event timestamp would put it before checkout. The model could not use it yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LETNyUNLNHeasvFNhKgiuQ.png" /><figcaption><em>An earlier event is not necessarily an available input. Both its validity at the decision time and its availability to serve must be respected.</em></figcaption></figure><p>Putting that event into the training row gives the model knowledge it could not have had in production. Across historical joins, backfills, and rolling windows, this kind of leakage can make offline performance look better than the system can deliver. A point-in-time reconstruction therefore respects both when a fact is applied and when production could use it.</p><p>For example, a previous COD order may already exist in the customer’s history while its delivery outcome is still unknown. If it returns later, that return must not appear in an earlier checkout’s RTO count. The current customer’s eventual payment choice and delivery outcome also belong after the checkout boundary, outside the input vector.</p><p>The execution tools are secondary to that rule. Spark can perform large joins and backfills; Airflow can schedule the work. Orchestration decides when a job runs. The job still has to reconstruct each row as of the original decision.</p><h3>Keep preprocessing consistent with serving</h3><p>Correct timestamps do not guarantee matching inputs. Training might encode an unseen category as “unknown” while serving maps it to the first known category. Two implementations of a seven-day aggregate might disagree about whether to include an event at the window boundary. Both pipelines can return valid numbers, and the endpoint can stay healthy while scores shift.</p><p>These differences are training-serving skew. A feature contract makes the assumptions explicit: meaning, type, units, window boundaries, category mappings, transformation version, input order, and how old a value may be when it is served. Preprocessing is part of the release because changing any of those assumptions can change what the model receives.</p><p>A missing required column breaks the schema and should stop the build. A missing value in a permitted field can use the selected model’s defined representation. An unseen category follows the transformation’s versioned fallback. Treating all three as zero would erase useful distinctions, including the difference between “none observed” and “unknown.”</p><p>Timestamped replay examples provide a direct check. Run the same examples through training and serving, then compare values, types, order, mappings, and missing-value behaviour. Investigate any disagreement before exposing the candidate to customers.</p><h3>Evaluate on the population we can observe.</h3><p>The labelled RTO dataset consists of eligible orders that proceeded with COD and reached a known terminal outcome. A prepaid checkout, an abandoned session, an unresolved delivery, or a checkout where COD was hidden is not a negative RTO label. The results describe the observed COD cohort.</p><p>Time matters here too. Train on an earlier period and evaluate on a later holdout whose outcomes have had time to become final. Fit learned transformations on the training split. If a <a href="https://scikit-learn.org/stable/modules/calibration.html">calibrator converts scores into probabilities</a>, fit it on a separate calibration split or suitable out-of-fold predictions that preserve time, leaving the final holdout untouched.</p><p>AUROC and precision-recall measures help assess ranking. Calibration asks whether estimated probabilities agree with observed frequencies. Precision and recall at a selected threshold tell us about the errors the policy would accept. These measurements complement each other; none alone tells us whether checkout is making the intended trade-off.</p><p>We can compare thresholds offline within the observed cohort, but we cannot recover the conversion effect of offering COD to customers who previously could not choose it. That requires experimental evidence.</p><h3>Keep the release reproducible.</h3><p>Saving the trained model does not give us everything needed to reproduce a checkout score. We also need the training-data snapshot and code, the feature definitions and transformations, and the software that runs inference. Keeping these versions together lets us reconstruct the inputs and understand how the model produced its answer.</p><p>The release record also preserves the evidence behind approval: which orders formed the evaluation cohort, which holdout was used, and how the candidate performed. It states what the score means, including whether it is a raw model output or a calibrated probability. Alongside that evidence, we record the compatible checkout policy, monitoring baseline, fallback behaviour, and previous release to restore if a rollback is needed.</p><p>A model registry can catalogue these releases and their approval status. Its references should point to fixed data snapshots and artifacts that cannot be overwritten. Managed training jobs can provide isolated compute and save their outputs, but deployment still needs to use the exact model, transformations, and inference software that were reviewed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S0SVDu12HEWsrJ8EhgFMGg.png" /><figcaption><em>A simplified RTO build flow. Data preparation and orchestration support training; evaluation and approval determine whether the candidate can be deployed.</em></figcaption></figure><p>The policy keeps its own version because a threshold change need not trigger training. However, a new model or calibrator can change the score’s meaning. Applying the old threshold to those new scores may remove COD from more checkouts than intended. We therefore evaluate the model and policy together and keep a compatible combination available for rollback.</p><h3>Deploy: Fit the decision into checkout.</h3><p>A reproducible release now has to answer a live request. Checkout calls the RTO prediction service, which requests a score through Lambda. Lambda assembles the model inputs from the current payload and precomputed historical features, applies the serving transformations, and invokes the SageMaker RTO endpoint. The score returns to the prediction service, where COD policy and other product rules determine the checkout action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gQGHhrG1ppWz0c1uNpWiag.png" /><figcaption><em>A simplified view of the RTO request and response path. Feature retrieval is a side input to Lambda, not a hop between Lambda and the model. Deployment, authentication and telemetry details are omitted for readability.</em></figcaption></figure><p>Historical joins and aggregates belong outside this request path. Prepare them ahead of time and retrieve the results, rather than scanning order history or updating aggregates while the customer waits. How often they refresh depends on the feature: a batch update can be sufficient for a value allowed to be hours old, while another input may require much fresher data. Online inference does not make every feature a streaming feature.</p><p>Serving checks the same contract used during the build, including required fields, data types, categories, missing values, and freshness. If an input violates the contract, the failure needs a reason and a defined fallback. Silently repairing the vector into something the model accepts can conceal a change in its meaning.</p><h3>Measure the whole request</h3><p>A fast model can still sit inside a slow checkout decision. The deadline includes the product call, validation, feature retrieval, transformation, inference, policy evaluation, and response handling. Measure the complete request and its dependencies, especially tail latency, and allocate dependency timeouts within that deadline.</p><p>Autoscaling an endpoint adds scoring capacity. It does not remove a queue or network delay elsewhere in the path. The product needs an explicit response when a lookup returns stale data or inference times out. That fallback may use non-model rules; whether it offers or hides COD is a product decision that should be reviewed and recorded.</p><p>The decision record then connects serving to evaluation. It needs a privacy-preserving correlation key, timestamp, model and policy identities, score definition, action, feature as-of times, and fallback reason. Use structured, sanitised fields and redact or exclude sensitive values from logs and errors before writing them. Captured sensitive data needs encryption at rest, least-privilege access, and enforced expiry or deletion when its retention period ends.</p><h3>Give each rollout a specific purpose</h3><p>A release version identifies the artifacts in use. A deployment variant describes how a release participates in traffic. For example, the current release might continue answering checkout while a candidate receives copied requests in shadow mode. We can compare their inputs, scores, latency, and resource use without applying the candidate’s action to the customer.</p><p>A canary introduces the candidate to a limited share of eligible traffic. Errors, stale inputs, action rates, and fallback use give us immediate guardrails. Quality checks based on fulfilment must wait for enough final outcomes and sufficient coverage. A rollback restores the previous compatible model-policy release.</p><p>An A/B experiment serves another purpose: controlled assignment can estimate causal product impact. Shadowing tests behaviour without taking the new action; a canary limits deployment risk. Neither substitutes for an experiment when the question is whether a new policy improves conversion.</p><h3>Monitor: Delivery runs on a slower clock</h3><p>Once the release is live, we can immediately inspect errors, latency, missing inputs, feature age, score distributions, and COD actions. Delivery outcomes take longer. Waiting for those labels before checking the service would leave operational problems undetected; evaluating prediction quality before labels are ready would give us incomplete evidence.</p><p>The two paths meet through the decision record. If the customer places a COD order, its fulfilment outcome can later be joined to the score and policy used at checkout. The evaluation rule needs both an outcome window and a known final status; an order does not become a negative RTO label simply because enough days have passed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RegnPKn9T5JC6g0L8IEe7w.png" /><figcaption><em>Filled circles have final outcomes, which may be deliveries or returns; hollow circles are pending. Crossed markers represent COD-hidden checkouts, whose outcome under COD cannot be recovered by waiting.</em></figcaption></figure><p>At the evaluation cutoff in this synthetic example, eight of ten eligible placed COD orders have final outcomes. Label coverage is 80%, not a delivery success rate. Quality metrics use those eight labelled orders. Hidden, prepaid, and abandoned checkouts are outside this denominator. Label maturity describes whether an outcome is ready to evaluate; coverage describes how much of the eligible population is ready.</p><p>Report the cohort, outcome window, and coverage beside quality metrics. Newly final labels can change the result, and unresolved orders may differ from resolved ones. Even complete coverage of the eligible cohort would not remove the selection introduced by the COD policy.</p><h3>Follow the evidence when behaviour changes</h3><p>Monitoring spans service health, input health, predictions and policy actions, and eventual outcomes. Compare each release with its own reference distributions and expected missing-value rates. When an older order’s outcome arrives after a new release goes live, attribute it to the release that scored that order. Otherwise, a dashboard can credit the new model with decisions it never made. Versioned decision records also help separate a model change from a threshold change or a different traffic mix.</p><p>Suppose, in an illustrative incident, a scheduled feature job stops refreshing an aggregate. Requests still succeed, but the inputs grow stale and scores shift. Feature-age telemetry can expose the problem before delivery labels arrive. Restore the data path and use the agreed fallback where necessary. Retraining on the broken values would carry the data problem into a new model.</p><p>A higher COD-hidden rate can also follow a changed policy, riskier merchant mix, or model change. Investigating those explanations is more useful than treating every shift as a retraining trigger. If service and inputs are healthy, examine score distributions, traffic cohorts, policy versions, and label coverage. A decline that remains after those checks gives us a reason to build a replacement and a test it should pass.</p><h3>Retrain: Make the candidate earn its traffic</h3><p>Scheduled refreshes, new features, and degraded outcomes can all justify a training run. Each still needs representative data and final labels. More recent data may describe a different observed population if the COD policy changed along the way.</p><p>Freeze the feature-data snapshot and label-observation cutoff for the build. Our 2:00 p.m. checkout enters the labelled dataset only if it proceeded with COD, meets the eligibility rules, and has a final outcome available by that cutoff. Its features still reflect what checkout could know at 2:00 p.m., even though its label arrived later.</p><p>Compare the candidate with production on the same later-period holdout, against acceptance criteria fixed before training. Within that observed cohort, choose thresholds that allow the same share of orders and compare the RTO rate among the orders each policy would allow. Alternatively, hold the acceptable RTO rate fixed and compare how many orders each would allow. Inspect important cohorts and report coverage alongside those comparisons. These simulations help evaluate the model-policy pair. They do not establish what previously hidden checkouts would do if offered COD.</p><p>A candidate might rank orders better but require features that take too long to retrieve. Its scores might also shift enough that the existing threshold removes COD from far more sessions. Successful training is therefore only one acceptance condition. Score calibration where applicable, policy compatibility, freshness, latency, fallback behaviour, and business guardrails still need to hold.</p><p>An accepted candidate proceeds through the appropriate shadow, canary, or experimental path, with a rollback target. A rejected candidate keeps its artifacts and evaluation results so the team can understand what was tried. Workflow automation can repeat the work while preserving that distinction between producing a model and approving its use.</p><h3>Conclusion: Connecting checkout to the next release</h3><p>The RTO model makes its prediction in a moment, but understanding that decision takes the whole lifecycle. We need the inputs checkout could actually use, the model release that produced the score, and the policy that decided whether COD was available. Later, delivery outcomes tell us what happened to the eligible COD orders that were placed.</p><p>Each stage preserves part of that history. Build makes the inputs and evaluation reproducible. Serving carries those assumptions into a live checkout. Monitoring separates a slow request or stale feature from a change in model quality, while keeping pending outcomes and unobserved COD outcomes distinct. Retraining uses that evidence to evaluate a candidate before it replaces the current release.</p><p>For RTO, the product question remains the same throughout: how do we keep COD available to customers while managing the merchant’s risk of returned orders? Connecting checkout decisions to later outcomes lets us investigate that trade-off, understand what changed, and make the next release on evidence we can trace.</p><p><em>Editor: </em><a href="https://medium.com/u/0ff40057ddde"><em>Parth Sawhney</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=999fe75e66ec" width="1" height="1" alt=""><hr><p><a href="https://engineering.razorpay.com/inside-razorpays-rto-model-the-engineering-behind-its-ml-lifecycle-999fe75e66ec">Inside Razorpay’s RTO Model: The Engineering Behind its ML Lifecycle</a> was originally published in <a href="https://engineering.razorpay.com/">Razorpay Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why catalogs matter: The book-keeping of Apache Iceberg]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devgenius.io/why-catalogs-matter-the-book-keeping-of-apache-iceberg-2d16ba75c2b2?source=rss-5ab3898edc74------2"><img src="https://cdn-images-1.medium.com/max/1066/0*fc5vz6WWxaJN0eZx" width="1066"></a></p><p class="medium-feed-snippet">For me, in Apache Iceberg, the catalog is essentially bookkeeping for the latest metadata for the Iceberg table.</p><p class="medium-feed-link"><a href="https://blog.devgenius.io/why-catalogs-matter-the-book-keeping-of-apache-iceberg-2d16ba75c2b2?source=rss-5ab3898edc74------2">Continue reading on Dev Genius »</a></p></div>]]></description>
            <link>https://blog.devgenius.io/why-catalogs-matter-the-book-keeping-of-apache-iceberg-2d16ba75c2b2?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d16ba75c2b2</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[lakehouse-fundamentals]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Thu, 22 May 2025 11:41:15 GMT</pubDate>
            <atom:updated>2025-05-28T14:51:42.489Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Apache Hudi: Copy on Write(CoW) Table]]></title>
            <link>https://ranjanankur.medium.com/apache-hudi-copy-on-write-cow-table-77fb2b849733?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/77fb2b849733</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-hudi]]></category>
            <category><![CDATA[copy-on-write]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Fri, 06 Oct 2023 19:24:27 GMT</pubDate>
            <atom:updated>2023-10-06T19:24:27.194Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cr7znyMfFig3GC6t30jPlQ.png" /></figure><p>As Data Engineer, we frequently encounter the tedious task of performing multiple UPSERT(update + insert) and DELETE operations in batch and streaming pipelines on our Data Lake platform. This can be overwhelming and time-consuming in Data Lake architecture.</p><p>The primary reason why these operations are very tedious in DataLake is because HDFS/Cloud Storage(S3, GCS etc.) are immutable. Whenever there is any update or deletion in a single row, the whole data or partition folder has to be read in DataLake and then it has to be overwritten.</p><p>There is the classic way of handling UPSERT and DELETE in DataLake i.e. <strong><em>Read in all Data, Combine and then Overwrite. </em></strong>If you have created any ETL/ELT pipeline and used Data Lake architecture then most probably you might have used this approach. Do you know you can solve this problem very efficiently by moving toward Data LakeHouse architecture?</p><blockquote><strong>In this article, try to observe the image and illustration very carefully to understand the concepts clearly. I have mentioned steps there. Try to follow &amp; observe the steps carefully.</strong></blockquote><h4>1. Data Lake vs. Data LakeHouse</h4><p>Let’s try to understand the above problem by taking an example of a <strong>normal librarian</strong> and an <strong>intelligent librarian</strong>.</p><blockquote><em>Data Lake Classic approach:</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iWgoQlVqHYceVJPFhLmtbA.png" /></figure><p>If you have worked in a Data Lake environment, you might find the illustration above familiar. This is how we typically implement UPSERT in Data Lake: we read either some partitions or the entire data, combine the existing data with the new one, and overwrite the merged data to the Data Lake. To clarify, this method is known as the UPSERT process.</p><p>Now let’s see how a smart and organised librarian like Data LakeHouse architecture🤩 will try to handle new upcoming data.</p><blockquote><em>Data LakeHouse Approach:</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gmenbwoQshpAMmMCqY4cDQ.png" /></figure><p>If you have observed the above illustration then you might have observed the organised way of handling books or paper in a well-managed library. This is the main difference between Data Lake and LakeHouse architecture. LakeHouse provides a good abstraction of data by exposing tables to end users instead of just RAW files of Data Lake.</p><p>If you have worked with databases like <strong>Postgres or MySQL</strong>, you are familiar with the <strong>power of tables</strong>. These tables handle all the database operations such as UPDATES, DELETE, or ACID transactions behind the scenes. The LakeHouse architecture <strong>is trying</strong> to provide the same abstraction by providing a solution by making our system as robust and abstract as these tables.</p><p>To clarify, users don’t need to worry as much about UPDATE, INSERT, and DELETE operations. It’s table properties which will help users to implement these things efficiently.</p><h4>2. Enabling LakeHouse using an open table format</h4><p>Let’s try to understand first how to implement the LakeHouse architecture using an <strong>open table format on top of Data Lake</strong> and then we will try to understand the Copy On Write(CoW) table of Apache Hudi in depth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*sBbmHBXilSrUHnaEFwonMA.png" /></figure><p>If you observe the above illustration then you can see that this is mostly like a Data Lake except Open Table formats like Apache Hudi or Apache Iceberg.</p><p>I am keeping LakeHouse architecture very minimal here to focus on the <strong>Copy on Write(Cow)</strong> table of Apache Hudi. But do check out my old article to understand the need for LakeHouse architecture.</p><p>So, as you have seen in the above illustration I have used Apache Hudi as an open table format, Let’s try to understand the Table type of Apache Hudi in depth.</p><h4>3. Types of Apache Hudi:</h4><p>Hudi supports two types of tables</p><ol><li>Copy on Write(CoW)</li><li>Merge of Read(MoR) table</li></ol><p>But In this article, we will be focusing only on the CoW table in the article.</p><blockquote><em>Before we dive into CoW, let’s walk through some of the terminologies used in hudi for a better understanding of the following sections.</em></blockquote><h4>Data or Base File:</h4><p>Hudi stores data in columnar <strong>parquet format</strong> and is called <strong>data files</strong>/ <strong>base files</strong>. This is known to be very performant and is widely used across the industry. Data files and base files are interchangeably used in general, but both mean the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*l5ED5zilam7bLhLLHCOisQ.png" /></figure><p><strong>Here are some of the key features of base files in Apache Hudi:</strong></p><ul><li><strong>Base files are immutable:</strong> This means that once a base file is created, it cannot be changed.</li><li><strong>Base files are versioned:</strong> This means that each base file has a version number. The version number is used to track the changes made to the data in the base file.</li><li><strong>Base files are stored in a distributed file system</strong>: This allows Hudi to scale to large datasets.</li></ul><h3>File Version:</h3><p>Let me explain in the context of COW. Whenever an update happens to a data file, a newer version of the data file will be created which contains merged records from older data files and newer incoming records. <strong>This information will be passed to metadata and metadata will record the latest version of the file version</strong>. It helps to get the latest data and even time series data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*uOXEuZ3CWITN-sZNQ9piuQ.png" /></figure><p>Now let’s try to understand Copy-on-write (CoW) by illustration, example concepts and code examples.</p><h4>Copy on Write(CoW):</h4><ul><li>👉 Stores data using exclusively columnar file formats ( <strong>e.g. parquet</strong>).</li><li>👉 Updates simple version and rewrites the files by performing a synchronous merge during write.</li><li>🚀 COW tables are simpler to operate than MOR tables. All updates are written to base files in Apache Parquet format, eliminating the need for a <strong>separate compaction service</strong> to manage log files and improve efficiency. Updates are handled by entirely rewriting the file to generate a new version of the base file.</li><li>👉 Consequently, COW tables <strong>exhibit higher write amplification</strong> because there is a synchronous merge that occurs to create the new base file version.</li><li>🚀 However, a key advantage of COW tables is their <strong>zero read amplification</strong> because all data is available in the base file, ready to read. 🚀 The disk reads required for a query are minimal because they do not need to read multiple locations or merge data.</li></ul><p>Let’s say we have 3 file groups with data files as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0BSsl78UW8JcA1nnfMeXpA.png" /></figure><p>Now we know that in the case of CoW, every new batch of write to Hudi will result in creating a new copy of respective data files and a newer version will be created along with records from the incoming batch.</p><p>Let’s suppose we get a <strong>new batch of write</strong> and after<strong> indexing</strong>, we find that these have records matching <strong>file group1</strong> and <strong>file group2</strong> and there are new inserts too. So for the new INSERT, we will be creating a new file group ( <strong>file group4</strong>).</p><p>So, both data_file1 and data_file2 will have newer versions created. Data file 1 V2 is nothing but contents of data file 1 V1, merged with records from incoming batch matching records in data file 1.</p><p>If you have still not understood then let’s try to understand by taking some layman’s example.</p><p>Imagine you have a table that stores customer data. You want to be able to add new customers, update existing customer information, and delete customers. You also want to be able to run queries that return all of the customer data.</p><p>In the case of Copy-on-write (CoW), if there is an update in existing customer records, the CoW strategy doesn’t modify the original records. Instead, it creates a <strong>new copy</strong> of the entire dataset that incorporates the updates or changes. Here’s how it works:</p><ol><li><strong>Initial State</strong>: You have an existing dataset with 10 customer records, let’s call them C1 to C10.</li><li><strong>Update</strong>: Let’s say you want to update the information for customer C5.</li><li><strong>Copy and Update:</strong> Instead of modifying the existing data, CoW creates a new copy of the files containing the C5 dataset, including the updated information for customer C5. Now, there will be a synchronous merge during write: Updates a simple version and rewrites the files by performing a synchronous merge during write. So, you now have a new set of files or a new version of your dataset that contains all 10 customer records (C1 to C10), including the changes you made to C5.</li><li><strong>Result</strong>: After the update, you effectively have two versions of your dataset. The original version with the unmodified C5 record and the new version with the updated C5 record.</li><li><strong>Metadata:</strong> Hudi also maintain efficient metadata where you can efficiently extract what record changed after each commit.</li></ol><p>This approach ensures data immutability, as the original data remains unchanged. It also allows for time travel queries, where you can access data as it existed at different points in time by referring to the corresponding version of the dataset.</p><p>Let’s try to understand the above logic by writing some code and after that let’s try to understand the steps and pseudo code of UPSERTING data to the Apache Hudi Copy On Write(CoW) table.</p><p><strong>I will suggest going through the code snippet once which is added below and then reading all the steps mentioned below for more understanding. Even if you don’t understand the code don’t get overwhelmed, try reading the steps, and you will be able to get it.</strong></p><pre>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import col, to_date<br><br># First Load all the required library and also Start Spark Session<br># Load all the required library<br><br># STEP1: Start Spark Session<br><br>spark = SparkSession.builder.appName(&quot;lakeHouse&quot;)\<br>        .config(&#39;spark.jars.packages&#39;, &#39;org.apache.hudi:hudi-spark3.2-bundle_2.12:0.13.0&#39;)\<br>        .config(&#39;spark.sql.extensions&#39;, &#39;org.apache.spark.sql.hudi.HoodieSparkSessionExtension&#39;)\<br>        .config(&#39;spark.sql.catalog.spark_catalog&#39;, &#39;org.apache.spark.sql.hudi.catalog.HoodieCatalog&#39;)\<br>        .config(&#39;spark.serializer&#39;, &#39;org.apache.spark.serializer.KryoSerializer&#39;)\<br>        .config(&#39;spark.sql.catalog.local.warehouse&#39;,&#39;warehouse&#39;)\<br>        .config(&quot;spark.driver.host&quot;, &quot;localhost&quot;)\<br>        .getOrCreate()<br><br>sqlContext = SparkSession(spark)<br>#Dont Show warning only error<br>spark.sparkContext.setLogLevel(&quot;ERROR&quot;)<br><br>print(f&#39;The PySpark {spark.version} version is running...&#39;)<br><br><br># STEP 2: READ THE CSV FILE AND CREATE THE DATAFRAME<br># When ever new data wil come the CSV path will be replaced.<br>df = spark.read.option(&quot;header&quot;,&quot;true&quot;).\<br>    csv(&quot;dataset/Hudi_Customer_input - Sheet1.csv&quot;)<br><br>refactored_df = df.withColumn(<br>    &quot;created&quot;,<br>    to_date(col(&quot;created&quot;),&quot;yyyy-MM-dd&quot;)<br>).withColumn(<br>    &quot;updated&quot;,<br>    to_date(col(&quot;updated&quot;),&quot;yyyy-MM-dd&quot;)<br>)<br><br># STEP 3: SET the hudi_options, default writing strategy in Apache Hudi is Copy On Write(CoW)<br><br>tableName = &quot;customers_detail_hudi_cow_table&quot;<br><br>hudi_options = {<br>    &#39;hoodie.table.name&#39;: tableName,<br>    &#39;hoodie.datasource.write.recordkey.field&#39;: &#39;id&#39;,<br>    &#39;hoodie.datasource.write.partitionpath.field&#39;: &#39;created&#39;,<br>    &#39;hoodie.datasource.write.table.name&#39;: tableName,<br>    &#39;hoodie.datasource.write.operation&#39;: &#39;upsert&#39;,<br>    &#39;hoodie.datasource.write.precombine.field&#39;: &#39;updated&#39;,<br>    &#39;hoodie.upsert.shuffle.parallelism&#39;: 2,<br>    &#39;hoodie.insert.shuffle.parallelism&#39;: 2<br>}<br><br># STEP 4: Write to Hudi table in UPSERT mode.<br><br>path = &quot;/Users/ankur.ranjan/Desktop/customers_detail_hudi_cow_table&quot;<br><br>df.write.format(&quot;hudi&quot;). \<br>    options(**hudi_options). \<br>    mode(&quot;append&quot;). \<br>    save(path)<br><br># STEP 5:(Optional) Read the Hudi table<br><br>spark.read.format(&quot;hudi&quot;).load(path).select(<br>    &quot;id&quot;, &quot;name&quot;, &quot;address&quot;, &quot;created&quot;, &quot;updated&quot;<br>).show()</pre><p>Let’s try to understand the above code.</p><ol><li><strong>Import the required library.</strong></li><li><strong>Initiate the spark session:</strong><em> Don’t forget to add the required Apache Hudi jar. If you get stuck with some problem while running your code, try to check the Apache Hudi official documentation and always use the proper spark version and compatible Apache Hudi version otherwise it will be hard for you. But as you are a developer, I assume that you will be good with maintaining the dependency issue.</em></li><li><strong>Read your input files:</strong> For me, it is a simple CSV file, I have used Spark to create the <strong>Spark DataFrame </strong>using a CSV input file. Whenever new data comes the CSV path will be updated for reading new incoming data.</li><li><strong>Define the Hudi options:</strong> Here are a few things that you need to understand for applying to UPSERT. In <strong>hoodie.datasource.write.operation </strong>key we have defined UPSERT mode and by default, Hudi writes in the Copy on Write table. So we don’t need to specify the Copy-On-Write option in any key.</li><li>The two attributes which identify a record in Hudi are the <strong>record key</strong> (see: hoodie.datasource.write.recordkey.field) and <strong>partition path</strong> (see: hoodie.datasource.write.partitionpath.field). In our configuration, the id is defined as a <strong>record key</strong> and created plays the role of a <strong>partition path. </strong>The combination of the<strong> record key</strong> and <strong>partition path </strong>is called a <strong>hoodie key</strong>.</li><li>If the input batch contains two or more records with the same <strong>hoodie key</strong>, these are considered the same record. For example, let’s suppose in older data we have data something like this.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*73JQy0IhQ8k1iU_fW7Hv-Q.png" /></figure><p>&amp; after a few minutes, we received new sets of updated data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TbCk2UNMNeCClTU8nt-3Q.png" /></figure><p>Here if you observed clearly then there is an update made to ID number 314, 310 i.e. Sadashiv &amp; Matt and one brand new record with ID 324 and name Mariam has come in new input data.</p><ol><li>Sadashiv and Matt’s <strong>address</strong> column was changed to Spain &amp; and Germany from India &amp; USA respectively.</li><li>The <strong>updated column</strong> is also updated.</li><li>For us, our hoodie key is (record key + partition path).</li><li>Here we have taken the ID column as the record key &amp; and created a column as the partition path.</li><li>Now, let’s try to understand another term in Apache Hudi which will help us to make UPSERT &amp; i.e. <strong>pre-combining</strong>. Here in this example, I will consider the <strong>Updated column</strong> as my pre-combining key.</li><li>Here if you observed it clearly then our <strong>hoodie key(</strong>id + created<strong>) </strong>for all the records are same as older records. So for applying the UPSERT(update + insert), Hudi runs the deduplication step called <strong>pre-combining</strong>.</li><li>The <strong>pre-combining</strong> procedure picks the record with a greater value in the defined field. In our case, this field is the updated, column so updated=2023–09–11 i.e. 11th Sept picked over updated=2023–09–11 i.e. 10th Sept for changes/updates made for ID number 314(Sadashiv) &amp; 310(Matt) from older data</li><li>see<strong>:hoodie.datasource.write.precombine.field</strong> option in the code. Here we have defined the <strong>Updated column</strong> as a pre-combine key.</li></ol><p>So after the UPSERT is made you will get the final output as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkALPOJE3RA7EC-AyAxneg.png" /></figure><p>As we have understood applying UPSERT using Apache Hudi &amp; created a Copy-On-Write table using it. Try to see the CoW hudi table structure.</p><ul><li>You will see that after running the code, it will create one folder in a given path. I have given path = “/Users/ankur.ranjan/Desktop/customers_detail_hudi_cow_table”</li><li>Now let’s try to check the content of the hudi table.</li></ul><ol><li>You can observe that it has created a folder in the given path with the name customers_detail_hudi_cow_table. Here every information of our Hudi Copy-On-Write will reside.</li><li>If you observe the above image clearly then you will able to see that it has created two folders inside the main folder. These two folders are 2023–08–10 &amp; 2023–09–10.</li><li>If you have seen the data that we have passed to the code and our code then you might have observed that we have defined our partition-path as created column in the code. We had only two values of the created column in the input data and those were 2023–08–10 &amp; and 2023–09–10.</li><li>Now if you observe the given screenshot image again, then you will see that in the folder we have two parquet files and some metadata files. Observe the creation time of two parquet files. One is 23rd Sept 23:27 whereas the other is 23rd Sept 23:34.</li><li>It is because we received our first input data at 23:27 and the next set of data at 23:34 &amp; and as we have learnt in Copy-On-Write whenever new sets of data come then it will create one copy of the existing file and merge the latest change there. It will not change the existing Parquet file(Data/Base file).</li></ol><p>Let’s try to verify this by reading those Parquet files individually. Please observe the given input in the above screenshots before observing the below code.</p><blockquote><strong><em>Note down that we are only reading for the partition folder=2023–08–10.</em></strong></blockquote><p>Let’s read the first Parquet file which was created when the first set of input data came i.e. on <strong>23rd Sept 23:27</strong></p><pre>oldData =spark.read.format(&quot;parquet&quot;).load(&quot;/Users/ankur.ranjan/Desktop/customers_detail_hudi_cow_table/2023-08-10/28187041-60f6-4a77-af3c-f4fb43768938-0_0-560-579_20230923232714083.parquet&quot;)<br><br>oldData.select(&quot;id&quot;, &quot;name&quot;, &quot;address&quot;, &quot;created&quot;, &quot;updated&quot;).show()</pre><p>Output is</p><pre>+---+--------+-------+----------+----------+<br>| id|    name|address|   created|   updated|<br>+---+--------+-------+----------+----------+<br>|322|   Mulan|  Japan|2023-08-10|2023-09-10|<br>|314|Sadashiv|  India|2023-08-10|2023-09-10|<br>+---+--------+-------+----------+----------+</pre><p>If you have observed the image of our first input data above then you might have seen that the created date of Mulan and Sadashiv was 2023–08–10 whereas for Ankur and Matt, the created date was 2023–09–10. So that’s why you are only seeing the two records in the 2023–08–10 folder for this Parquet file.</p><p>Now let’s try to read when the next set of data has come i.e. on <strong>23rd Sept 23:34.</strong></p><pre>newData = spark.read.format(&quot;parquet&quot;).load(&quot;/Users/ankur.ranjan/Desktop/customers_detail_hudi_cow_table/2023-08-10/54a52289-bbe2-4e85-a7fa-89df4a62ecb8-0_0-506-527_20230923225057099.parquet&quot;)<br><br>newData.select(&quot;id&quot;, &quot;name&quot;, &quot;address&quot;, &quot;created&quot;, &quot;updated&quot;).show()</pre><p>Output is</p><pre>+---+--------+-------+----------+----------+<br>| id|    name|address|   created|   updated|<br>+---+--------+-------+----------+----------+<br>|322|   Mulan|  Japan|2023-08-10|2023-09-10|<br>|314|Sadashiv|  Spain|2023-08-10|2023-09-11|<br>|324|  Mariam|    UAE|2023-08-10|2023-09-11|<br>+---+--------+-------+----------+----------+</pre><p>If you have observed the output clearly then you might have realised the magic of LakeHouse architecture particularly the Apache Hudi Copy On Write table.</p><p>I hope that all these above illustrations, images, code, steps etc. have helped you to understand the Copy-On-Write table in Apache Hudi.</p><p>Let’s meet in the next article to learn some more wonderful concepts of Apache Hudi till then Thank you, Goodbye &amp; and keep learning.</p><p>We have discussed a lot of concepts in this article. It is becoming long. I hope some of my readers who do understand the LakeHouse architecture and worked on Apache Hudi will post their comment section about their reading.</p><p>Feel free to subscribe to my YouTube channel i.e. <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p>#apacheHudi #lakehouse #datalakehouse</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/apache-hudi-copy-writecow-table-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77fb2b849733" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supercharging Apps with Polyglot Persistence: A Simple Guide]]></title>
            <link>https://medium.com/analytics-vidhya/supercharging-apps-with-polyglot-persistence-a-simple-guide-aea974ccafc8?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/aea974ccafc8</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Mon, 04 Sep 2023 14:24:35 GMT</pubDate>
            <atom:updated>2023-09-04T14:24:35.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mj571s5WVGtu0UY6" /></figure><p>After working for more than 4 years on Data Intensive applications in a startup, consultancy and product-based companies. I think that the most important design and decision of any successful project is choosing the right Database and Data Storage at different layers of application.</p><p>Whenever I am discussing my design with my seniors or architect most of the time, I am most concerned about my databases. I have learned from my failures of choosing the wrong storage at different layers and It has impacted the design a lot.</p><p>Some days back I was reading my favourite book i.e. Designing Data-Intensive Applications and I found a wonderful concept i.e. Polyglot Persistence.</p><p>Let’s try to see what is polyglot persistence and why it hurt me a lot at the beginning of my career. In fact, I am still figuring out the right and learning daily.</p><p>Polyglot persistence advocates for selecting the most appropriate database or storage technology for each specific data type based on its characteristics and requirements.</p><p>It is about recognizing that there is no one-size-fits-all solution for data storage, and it encourages developers to choose the most suitable storage technology for each data type within an application’s architecture. This can lead to improved performance, scalability, and flexibility in managing different types of data effectively.</p><p>Imagine your data-intensive app as a puzzle with different pieces. Each piece of data has its own shape, size, and purpose. Now, think about having a magical toolbox that offers just the right tool for each puzzle piece. That’s what polyglot persistence is all about!</p><p>Here’s why it’s like having magic for your app:</p><p><strong>Custom Fit:</strong> Just like shoes — one size doesn’t fit all. Some data likes to be in tables, some in flexible spaces, and some just want to be found quickly.</p><p><strong>Speedy Spells:</strong> Different tools make things faster. Imagine searching through a big library — some tools help you find books faster, while others help you read them quickly.</p><p><strong>Growing Up Strong:</strong> Think of your app as a plant. Some data needs more water, some need more sun. Polyglot persistence lets your app grow strong by giving each piece of data what it loves.</p><p><strong>Saving Money, Saving Energy:</strong> Imagine a money jar. You don’t want to put the same amount of money in every day, right? Polyglot persistence helps you put more money in places that need it and less where it’s not necessary.</p><p><strong>Special Powers for Special Data:</strong> Just like superheroes have different powers, data has different needs. Some data needs extra security, and some data needs to be counted really quickly.</p><p><strong>Easy Puzzles, Quick Wins:</strong> Ever tried putting a circle puzzle piece into a square hole? It’s tough! With polyglot persistence, every puzzle piece fits perfectly, and you don’t waste time trying to make it fit.</p><p><strong>Less Frustration, More Fun:</strong> Have you ever spent hours fixing a toy? With polyglot persistence, you spend less time fixing and more time playing with your app.</p><p><strong>Mix and Match:</strong> Imagine mixing your favourite ice cream flavours. Polyglot persistence lets you combine different flavours (or tools) to make something super tasty (or super useful).</p><p><strong>Back-Up Buddy: </strong>You have a secret backup teddy bear, right? Polyglot persistence gives you backup friends for your app’s data, so you’re never alone when something goes wrong.</p><p><strong>Smart Thinking, Smart Choices:</strong> Remember when you chose the right colours for your painting? Polyglot persistence helps you choose the right tools for your app so it can do amazing things.</p><p>To ensure optimal performance, it is essential to select the most suitable technology for each application’s unique requirements. As such, it is highly probable that both relational databases and nonrelational data stores will persist side by side, a concept often referred to as polyglot persistence, for the foreseeable future.</p><p>Choosing the right tech 🤖 for a task depends on what you need 📈. So, old-school databases 💾 might still hang out with cool new data homes 📊. People call this mix-up “polyglot persistence” 🌎. It’s like tech diversity!</p><p>Feel free to subscribe to my YouTube channel i.e. <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/supercharging-apps-polyglot-persistence-simple-guide-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aea974ccafc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/supercharging-apps-with-polyglot-persistence-a-simple-guide-aea974ccafc8">Supercharging Apps with Polyglot Persistence: A Simple Guide</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solve Small File Problem using Apache Hudi]]></title>
            <link>https://ranjanankur.medium.com/solve-small-file-problem-using-apache-hudi-dbf76b7208fd?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbf76b7208fd</guid>
            <category><![CDATA[opentable]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-hudi]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Fri, 25 Aug 2023 06:04:27 GMT</pubDate>
            <atom:updated>2023-08-25T06:04:27.175Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_qdhB1VviAXlVSbFH2tO-g.png" /></figure><h3>One of the biggest pains of Data Engineers is small file problems.</h3><p>Let me tell you a short story and explain how one of the efficient tools solves this problem.</p><p>A few days ago, while UPSERTING data to my Apache Hudi table, I was observing the pipeline result and noticed that my small files were being compacted and converted into larger files. 🤔💻🔍💾</p><p>I experienced awe for a few minutes until I discovered the magical capabilities of Apache Hudi.</p><p>One of the best features that Apache Hudi provides is to overcome the dreaded small files problem.</p><p>For those unfamiliar with Apache Hudi, here’s a brief definition and usage.</p><p>Apache Hudi is an open table file format that can be used efficiently to enable Data Lakehouse. It provides multiple supports in building Data Lakehouse but according to me, the most impactful are the following three.</p><ol><li><strong>Efficient Ingestion: </strong>Support for mutability, Row level updates &amp; and deletes.</li><li><strong>Efficient reading/writing performance: </strong>Support for multiple types of index to make your write and read faster, support for MOR table, Improved file layout and timeline.</li><li><strong>Concurrency control and ACID guarantees.</strong></li></ol><p>To give better context, I feel Hudi is not only open table file formats, it has multiple other features that amaze me every day.</p><p>But this post is not regarding What is Apache Hudi and where to use it. It is more about one of its features that I fell in love with in recent times i.e. its ability to deal with small file problems.</p><p>One design decision in Hudi was to avoid creating small files by always writing properly sized files. There are two ways in which Hudi solves this issue.</p><h3>𝗔𝘂𝘁𝗼 𝗦𝗶𝘇𝗲 𝗱𝘂𝗿𝗶𝗻𝗴 𝗶𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻: For COW</h3><p>Automatically managing file sizes during ingestion may cause slight latency, but ensures efficient read queries immediately after a write is committed. Failing to manage file sizes during writing will result in slow queries until a resize cleanup is completed.</p><p>There are two important parameters that Apache Hudi uses during this process.</p><ol><li>𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐦𝐚𝐱.𝐟𝐢𝐥𝐞.𝐬𝐢𝐳𝐞:Target size in bytes for parquet files produced by Hudi write phases.</li><li>𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐬𝐦𝐚𝐥𝐥.𝐟𝐢𝐥𝐞.𝐥𝐢𝐦𝐢𝐭:During upsert operation, Hudi opportunistically expands existing small files on storage, instead of writing new files, to keep the number of files to an optimum.</li></ol><p>This config sets the file size limit below which a storage file becomes a candidate to be selected as such a small file. By default, treat any file &lt;= 100MB as a small file.</p><p>Let’s try to understand by taking one example for the COW(Copy on Write) hudi table.</p><p>So let’s suppose our configuration is set to the following.</p><p>𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐦𝐚𝐱.𝐟𝐢𝐥𝐞.𝐬𝐢𝐳𝐞: 120 MB</p><p>𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐬𝐦𝐚𝐥𝐥.𝐟𝐢𝐥𝐞.𝐥𝐢𝐦𝐢𝐭: 100 MB</p><p>Now let’s see how Apache Hudi solves small-size problems step by step for the COW table.</p><p>📊 <strong>Sizing Up Small Files with Hudi</strong></p><ol><li>📏 <strong>File Size Controls</strong>: Think of it like adjusting the size of a picture! Hudi lets you set the maximum size for base/Parquet files(𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐦𝐚𝐱.𝐟𝐢𝐥𝐞.𝐬𝐢𝐳𝐞). You can also decide when a file is “small” based on a soft limit(𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐬𝐦𝐚𝐥𝐥.𝐟𝐢𝐥𝐞.𝐥𝐢𝐦𝐢𝐭)</li><li>🚀 <strong>Smart Start</strong>: When you’re starting a Hudi table, estimating the size of each record is like fitting puzzle pieces. You’re aiming to neatly fit your data records into a Parquet file, so they’re arranged well and take up less storage space.</li><li>📝 <strong>Memory Lane</strong>: As you keep writing, Hudi remembers the average record size from previous times. This helps it write and organize data better.</li><li>📝 <strong>Writing Magic</strong>: Imagine Hudi as a clever organizer. It adds more records to small files as you write, kind of like filling up a box. The goal? To reach the maximum size you set. For example, if the file size is 40 MB and the <em>small.file.limit</em> is set to 100 and the <em>max.file.size</em> is set to 120 MB. During the next set of inserts, it will try to add 80 MB of file to make it to 120 MB.</li><li>🧩 <strong>The Perfect Fit</strong>: Let’s say you set a compactionSmallFileSize of 100MB and a limitFileSize of 120MB. Hudi looks at files smaller than 100MB and adds data until they’re a comfy 120MB.</li></ol><p>Hudi helps your data find the right fit, just like sorting puzzle pieces to complete a picture! 🧩📊🪄</p><p>Let’s understand by one illustration.</p><ul><li>Let’s suppose, For the first time you wrote to Hudi using Spark &amp; it created 5 files.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*3noPrXCfHB0fuDusQYBnfA.png" /></figure><ul><li>During the second insert, it will identify, File 1, File 2 &amp; and File 3 as a small file because we have set 𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐬𝐦𝐚𝐥𝐥.𝐟𝐢𝐥𝐞.𝐥𝐢𝐦𝐢𝐭: 100 MB. So incoming inserts are assigned to them first. For example, Hudi will try to assign more 90 MB to file 1, 100 MB to file 2, and 40 MB to file 3. So does that because we have set 𝐡𝐨𝐨𝐝𝐢𝐞.𝐩𝐚𝐫𝐪𝐮𝐞𝐭.𝐦𝐚𝐱.𝐟𝐢𝐥𝐞.𝐬𝐢𝐳𝐞: 120 MB &amp; we want to set this to full capacity. During the next run inserts again, it will try to add 60 MB of data to the last file first.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*KkL6Lyfc0Rma1IOftZxvsg.png" /></figure><ul><li>All of these file sizing algorithms, keep running to keep the optimal size of your file. This ensures that you don’t have small file problems in your cloud storage.</li></ul><p>NOTE: Small files will be auto-sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, if you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file.</p><p>This algorithm might seem a little robust :) but trust me Apache Hudi is magical when it comes to handling small file problems.</p><p>We have discussed a lot of concepts in this article. It is becoming long. I hope some of my readers who do understand the File Sizing algorithm for the Apache Hudi MOR(Merge On Read) table will post their explanation in the comment section or Feel free to subscribe to my YouTube channel i.e. <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/solve-small-file-problem-using-apache-hudi-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbf76b7208fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimize MERGE job in BigQuery]]></title>
            <link>https://blog.devgenius.io/optimize-merge-job-in-bigquery-5139fbd3ab43?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/5139fbd3ab43</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[merge]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Mon, 31 Jul 2023 15:43:14 GMT</pubDate>
            <atom:updated>2023-08-01T17:47:28.821Z</atom:updated>
            <content:encoded><![CDATA[<p>I love BigQuery and think It is one of the best products ever made by the Google Cloud Platform.</p><p>As someone who works in data engineering, I have found BigQuery to be one of the best products offered by the Google Cloud Platform. One feature I frequently use is the MERGE statement, which allows for UPSERT (Update + Insert) in BigQuery Native tables.</p><p>Having spent over a year using BigQuery at work, I’ve discovered several effective ways to optimize the merge statement. In this discussion, I’ll share each of these strategies and explain why they have proven helpful in my experience.</p><h3>1. What is MERGE in BigQuery?</h3><p>Let’s first understand what is MERGE in the context of Google BigQuery and what’s better than understanding it by one example.</p><blockquote><em>Sure! Below is an example of a BigQuery MERGE statement, also known as an </em><strong><em>upsert</em></strong><em> operation. In this example, we have two tables: </em><strong>source_table</strong><em> and </em><strong>target_table</strong><em>. We want to merge data from </em><strong>source_table</strong><em> into </em><strong>target_table</strong><em> based on a common key column (</em><strong><em>id</em></strong><em>).</em></blockquote><p>Assume the schema for both tables is as follows:</p><p><strong>source &amp; target table</strong>:</p><p>- id INTEGER</p><p>- name STRING</p><p>- age INTEGER</p><p>- programming STRING</p><p>- created DATE</p><p>- updated DATE</p><p>The goal is to update existing rows in target_table if the <strong>`id`</strong> already exists or insert new rows if the <strong>`id`</strong> doesn’t exist in target_table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2Wky96aQwp8VGyAGUF85Q.png" /></figure><p>Explanation:</p><ol><li>The `MERGE` keyword indicates the start of the merge operation.</li><li>We provide aliases (`target` and `source`) for the `target_table` and `source_table`, respectively, to use them in the `WHEN` clauses.</li><li>ON `target.id = source.id` specifies the condition for matching rows in both tables based on the `id` column.</li><li>WHEN MATCHED THEN indicates when a <strong>match is found</strong>. In this case, we update the existing row in the `target_table` with data from the `source_table`.</li><li>WHEN NOT MATCHED THEN specifies when<strong>there is no match</strong>. Here, we insert a new row into the `target_table` using data from the `source_table`.</li></ol><p>By running this MERGE statement, you can efficiently synchronize data between the `source_table` and `target_table`, <strong>ensuring that any updates or new records are correctly handled.</strong> It is very beneficial when you are running your pipeline daily or even streaming your data using Apache Spark or any other computing engine and want to keep only the latest updated data in your final BigQuery native table.</p><h3>2. Ways to optimize MERGE</h3><p>Now let’s try to understand different ways to optimize <strong>MERGE</strong> in BigQuery &amp; and before optimizing anything we should ask need of optimizing it questions to ourselves.</p><ul><li>Discovering ways to enhance the MERGE process can lead to better outcomes.</li><li>It will drastically reduce your execution time of the big data pipeline.</li><li>Every job running on BigQuery comes with cost and we should always work to reduce the cost.</li></ul><p>Enough of introduction, theory &amp; philosophy :) Let’s jump to the real thing i.e. our Engineering of optimization!!! Let’s note down the point and then we will try to understand each and every optimization in detail.</p><ol><li>The first thing you should do is apply <strong>Clustering</strong>: If possible, cluster the tables on the merge keys. Clustering reorders the data physically on disk, which can improve query performance.</li><li>Bigquery Merge Optimization through <strong>Partition Pruning</strong>. BigQuery prune partitions based on static predicates rather than dynamic predicates.</li><li>If you are sure that your incoming data can only have updates in a few columns then <strong>specify only those columns</strong> in your merge statement. Don’t mention all columns.</li><li>Avoid<strong> unnecessary clustering</strong> on multiple columns.</li></ol><p>Let’s start with the most interesting optimization from all these i.e. <strong>Partition Pruning.</strong></p><p>If you have observed the above example of MERGE then we have one column named <strong>`created`</strong>. Previously our table was not partitioned on any column. Now let’s suppose instead of a simple BigQuery native table if we create a partition table applying partitioning on the <strong>created</strong> column then Partitioning will significantly speed up merge operations as it reduces the amount of data scanned during the process.</p><p>Let’s see by example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJAfhM-SiQaFSNYvNgwz1Q.png" /></figure><p>But you will say, ohh Ankur 😄 I know about <strong>partitioning</strong> &amp; I know can it reduce the scanning but what is this <strong>Pruning?</strong></p><p>Let’s try to understand Pruning by applying it directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BErVI4Vb-CugQ0TlHZPLoQ.png" /></figure><p>If you observe the MERGE statement carefully and compare it with the previous MERGE statement then there are basically two differences.</p><ol><li>We have precomputed <strong>date_min</strong> and <strong>data_max</strong> in a separate query. Observe the <strong>DECLARE</strong> statement and <strong>SET</strong> statement carefully.</li><li>We have provided <strong>date_min</strong> and <strong>data_max </strong>as static values in the MERGE sql ON clause.</li></ol><p>This will be huge optimization &amp; due to the fact that the BigQuery prune partitions based on static predicates rather than dynamic predicates. You will easily observe that the <strong><em>bytes scanned</em></strong>, <strong><em>bytes shuffled, slot time consumed and elapsed time </em></strong>are a lot lower than the previous MERGE statement.</p><p>This concept applies not only to MERGE operations but also to other operations like join, filter etc.</p><p>I hope now you are clear on how Partition Pruning works in BigQuery MERGE statement and how it can optimize your query.</p><p>Now let’s try to understand how <strong>clustering</strong> helps during MERGE.</p><p>Clustering in BigQuery helps improve the efficiency and performance of the <strong>MERGE</strong> operation by physically organizing data in a more ordered manner on disk. Clustering works in conjunction with partitioning and involves the rearrangement of data within partitions based on the values of one or more chosen columns, known as clustering columns.</p><p>When data is clustered on specific columns, it benefits queries that involve filtering, joining, and sorting based on those columns. In the context of the <strong>MERGE</strong> statement, clustering can speed up the matching process, reducing the amount of data that needs to be scanned during the merge operation.</p><p>Let’s go through an example to illustrate the concept of clustering in the context of a <strong>MERGE</strong> operation:</p><p>Now we know that we have the following tables;</p><ol><li><strong>Target table </strong>— target_table_partitioned (partitioned on the <strong>created </strong>field)</li><li><strong>Source table</strong> — source_table</li></ol><p>Now, let’s create a clustered version of the <strong>`target_table`</strong>, clustering it on the <strong>`id`</strong> column:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSS9Why50c-2x5p9bbOuqw.png" /></figure><p>Now we will apply the same MERGE statement with the same <strong>Partitining Purning</strong> but this time target table will be <strong><em>target_table_partitioned_clustered </em></strong>instead of <strong><em>target_table_partitioned </em></strong>table. Now, let’s try to understand how this clustering on top of <strong>partitioning purning</strong> will do wonders.</p><p>With clustering applied, the <strong>`MERGE`</strong> operation can benefit from the improved data organization on disk. During the matching process, data with the same `id` values are more likely to be located together, reducing the amount of data scanned and improving the overall performance of the merge operation. This is especially advantageous when the `id` column is used as a frequent filter for the `MERGE` condition or when the `id` values have a non-uniform distribution across the partitions.</p><p>This article has gone long and I think points three and four are self-explanatory I would just love it if one of my readers actually tried to explain these two points in the comment sections.</p><p>Feel free to subscribe to my YouTube channel i.e. <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/optimize-google-bigquery-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5139fbd3ab43" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/optimize-merge-job-in-bigquery-5139fbd3ab43">Optimize MERGE job in BigQuery</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stateful transformations in Spark Streaming — Part 1 | Spark Streaming Session 3]]></title>
            <link>https://medium.com/analytics-vidhya/stateful-transformations-in-spark-streaming-part-1-spark-streaming-session-3-a18c8657a3?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/a18c8657a3</guid>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Sun, 26 Feb 2023 12:15:19 GMT</pubDate>
            <atom:updated>2023-02-26T12:15:19.837Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stateful transformations in Spark Streaming — Part 1 | Spark Streaming Session 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Umhh7YokcvnEDkgJ" /></figure><p>In the previous article of this series i.e. Spark Streaming in layman’s terms we have understood the following things.</p><ol><li>Different streaming sources</li><li>Stateful vs Stateless transformation</li></ol><p>For those who are reading this article without reading the previous article of this series, I recommend reading the last article or watching the video before starting this article. This will give you a better understanding of all spark streaming concepts.</p><p>I also created a YouTube video for the same concepts. Please follow these links.</p><p>Useful links:</p><ul><li><a href="https://www.linkedin.com/pulse/spark-streaming-session-2-ankur-ranjan/">LinkedIn Article Link</a></li><li><a href="https://www.youtube.com/watch?v=n_BR_g3VNhg&amp;list=PLiDUHlGx0KN8u1tqDkpG-CDoxf8uEMiBl">YouTube Video Link</a></li></ul><p>Here in this session, we will mainly focus on the following stateful transformations.</p><ul><li>updateStateByKey (This Article)</li><li>reduceByKeyAndWindow (Next Article)</li></ul><p>These functions will help you to understand the stateful transformation in an easy way.</p><p>So let’s start and discuss these concepts in very layman’s terms.</p><p>Before understanding these examples let’s quickly recall the meaning of stateful transformations.</p><h3>Stateful Transformation</h3><p>Stateful transformation is a particular property of s <em>park streaming, </em>it enables us to maintain the state between micro-batches. In other words, it maintains the state across a period of time, which can be as long as an entire session of streaming jobs.</p><p>We mostly achieved this by creating checkpoints on streaming applications.</p><p>If we understand it in simple terms then using stateful transformation we can use some data from the previous batch to generate the results for a new batch.</p><p>Let me try to make you understand the above concept using one simple example too.</p><ul><li>Let’s consider, you have a streaming application which runs for 12 hours continuously.</li><li>Suppose our batch interval is of 5 minutes. It means that a new RDD will be created every 5 minutes.</li><li>So during the entire stream, we will have 144 RDD.</li><li>It’s simple mathematics.</li><li>12 hours = 12 * 60 = 720 minutes.</li><li>720 minutes / 5 minutes = 144 RDD</li><li>Now let’s suppose we wanted to create a <strong>running sum </strong>then we need information about the previous states too. It is because it is a streaming application. At one point in time, you will only have some data. Some of the data might have already come or some might be coming in the next batch duration.</li></ul><p>There can be two scenarios here.</p><ul><li>We might be interested in calculating the running sum of the entire stream i.e. 12 hours.</li><li>Or, the last 30 minutes. It means that we are interested in the last 6 RDDs always. For this type of requirement, we use the concept of sliding interval and window size. Please check my previous article to understand the concept of sliding interval and window size.</li></ul><p>Let’s try to solve the first scenario i.e. finding the <strong>running sum</strong> of the entire stream. Here we will be calculating the frequency of each word across the entire stream.</p><p>We will be using the <strong>updateStateByKey</strong> transformation function for this.</p><h3>updateStateByKey</h3><p>updateStateByKey is a stateful transformation and it requires 2 steps.</p><ol><li>Define a state to start with — The state can be an arbitrary data type.</li><li>A function to update the state — In this function, you have to specify how to update the state with the previous state and the new value received from the stream.</li></ol><p>Do you remember, we have written a word count example in our previous article where we read a stream of data coming from the console?</p><p>Just follow <a href="https://www.linkedin.com/pulse/spark-streaming-part-1-ankur-ranjan/">this article</a> and you will find the word count code snippet.</p><p>Now let’s suppose we want to count the <strong>occurrence</strong> of some word in <strong>an entire stream</strong> of data. For example, we would like to check <strong>how many times</strong> the word <strong>“Data” </strong>is occurring in the entire stream.</p><p>I hope you have understood the above problem statement of finding the occurrence of one particular word in the entire stream.</p><p>In this example too we will be using a socket as the source or producer for producing data. For using a socket, we will simply use `<strong><em>nc -lk 9000`</em></strong>command. It will help us to produce data from port 9000.</p><p>Let’s suppose for the first time, we have entered the following line in the terminal.</p><blockquote><em>The Big Data Show is a great channel for learning Data Engineering concepts.</em></blockquote><p>So after applying the following code. Here all code snippet is written in scala</p><pre>val sc: SparkContext = spark.sparkContext <br><br>// Initiate the streaming context <br><br>val ssc: StreamingContext = new StreamingContext( sc, batchDuration = Seconds(15) ) <br><br>// Here we have used the check point directory. It will help us <br>// perform stateful operation.<br> <br>ssc.checkpoint(&quot;./checkpointing&quot;) <br><br>val lines: ReceiverInputDStream[String] = ssc.socketTextStream( &quot;localhost&quot;, 9000 ) <br>val words: DStream[String] = lines.flatMap(x =&gt; x.split(&quot; &quot;)) <br>val wordsFrequencyMap = words.map(w =&gt; { (w, 1) })</pre><p>We will get output something like this</p><pre>(The, 1) <br>(Big, 1) <br>(Data, 1) <br>(Show, 1) <br>(is, 1) <br>(a, 1) <br>(great, 1) <br>(channel, 1) <br>(for, 1) <br>(learning, 1) <br>(Data, 1) <br>(Engineering, 1) <br>(concepts, 1))</pre><p>Now we will use the <strong>updateStateByKey</strong> function to apply our first stateful operation.</p><p>We have just read above that updateStateByKey work on the entire stream of data &amp; it accepts one function as a parameter. Let’s look at an example</p><pre>def stateFullEntireStream(wordsFrequencyMap: DStream[(String, Int)]): Unit = {<br>  // Here we are filtering the data <br>  val filterMap: DStream[(String, Int)] = wordsFrequencyMap.filter(x=&gt; { <br>    x._1.startsWith(&quot;Data&quot;) <br>  }) <br>  <br>  val result = filterMap.updateStateByKey(updateFunc) result.print() <br>}</pre><p>Here you can see that the first thing that we are doing is filtering all messages of RDD which starts with <strong>“Data”. </strong>Then we are calling <strong>updateStateByKey</strong> on our Filtered Dstream and pass one <strong>updateFunc </strong>here. Let’s check the <strong>updateFunc </strong>code &amp; things will look more clear then.</p><pre>def updateFunc(newValues:Seq[Int], previousState:Option[Int]): Option[Int] = { <br>  val newCount = previousState.getOrElse(0) + newValues.sum Some(newCount) <br>}</pre><p>Here you can see that for the first time, there will be the following values of the parameter passed.</p><p>But as we have used <em>getOrElse(0) </em>with the previousState whose data type is Option[Int]. So the value will equal <strong>0</strong> in the code.</p><p>After doing the calculation, <strong>updateFunc</strong> will return 2 for two times occurrences of the word <strong>“Data”</strong></p><p>Now let’s suppose we enter the following lines in the terminal which is a producer for us.</p><blockquote><em>The Big Data Show is an interesting channel on Youtube but there are other channels too which are equally good for learning Data Engineering.</em></blockquote><p>Now once we enter this line and after our filter statement is applied.</p><pre>....filter(x=&gt; { x._1.startsWith(&quot;Data&quot;) })</pre><p>We will get the following parameter in <strong><em>updateFunc </em></strong>function.</p><p>(Data, <strong>{1, 1}</strong>) newValues = <strong>Seq(1, 1) </strong>previousState = <strong>2</strong></p><p>Here if you are observing and understanding it clearly then now <strong><em>previousState</em></strong> is not but .</p><p>After running this code, it will return the output as <strong>4</strong>.</p><p>So now you have understood how <strong><em>updateStateByKey </em></strong>can be used for calculating operations in the entire stream of data.</p><p>I hope, you do understand that we are using a checkpoint directory in our code where we are storing all our states.</p><p>But frankly speaking, there will be very few scenarios where you will be really interested in calculating the running aggregation for the entire stream of data. It is technically not flexible also because if we keep maintaining the state every time for the entire stream then the size of the checkpointing directory will keep on increasing and this creates a lot of problems.</p><p>I have not seen or implemented the aggregation of stream data for the entire duration of the entire stream.</p><p>One needs to understand that this information of checkpointing directory will actually be copied to every executor and if the size of checkpointing keeps on growing then there are greater chances of getting memory out of exception error.</p><p>Then how can we actually apply the aggregation or operation on streaming data?</p><p>We can actually do it in production but don’t apply an operation on the entire stream mostly but we can apply aggregation on the last 30 min or the last 15 min of data.</p><p>For this Window Size and Sliding Interval comes in very handy.</p><p>We will understand this concept in the next article and we will try to understand it with one function <strong>reduceByKeyAndWindow. </strong>Till then, keep learning and goodbye. Cheers to learning.</p><p>Please find the whole code snippet of <strong>updateStateByKey </strong>demo<strong>.</strong></p><pre>package sparkStreaming <br>import org.apache.spark.SparkContext <br>import org.apache.spark.sql.SparkSession <br>import org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream} <br>import org.apache.spark.streaming.{Seconds, StreamingContext} <br>import utils.{SparkIOUtil =&gt; SU} <br><br>object WordSearching { <br>  val WORDS_TO_SEARCH = &quot;Data&quot; <br><br>  def updateFunc(newValues:Seq[Int], previousState:Option[Int]): Option[Int] = { <br>    val newCount = previousState.getOrElse(0) + newValues.sum Some(newCount) <br>  } <br>  <br>  def stateFullEntireStream(wordsFrequencyMap: DStream[(String, Int)]): Unit = { <br>    <br>    val filterMap: DStream[(String, Int)] = wordsFrequencyMap.filter(x=&gt; {<br>       x._1.startsWith(WORDS_TO_SEARCH) <br>    }) <br>    <br>    val result = filterMap.updateStateByKey(updateFunc) <br>    <br>    result.print() <br>  } <br>  <br>  def main(args: Array[String]): Unit = { <br>    val spark: SparkSession = SU.getSparkSession <br>    spark.sparkContext.setLogLevel(&quot;ERROR&quot;) <br>    <br>    val sc: SparkContext = spark.sparkContext <br>    <br>    val ssc: StreamingContext = new StreamingContext( <br>      sc, batchDuration = Seconds(15) <br>    ) <br>    <br>    ssc.checkpoint(&quot;./checkpointing&quot;) <br>    <br>    val lines: ReceiverInputDStream[String] = ssc.socketTextStream(<br>      &quot;localhost&quot;, 9000<br>    ) <br>    <br>    val words: DStream[String] = lines.flatMap(w =&gt; w.split(&quot; &quot;)) <br>    <br>    val wordsFrequencyMap: DStream[(String, Int)] = words.map(w =&gt; (w, 1)) <br>    <br>    stateFullEntireStream(wordsFrequencyMap) <br>    <br>    ssc.start() <br>    <br>    ssc.awaitTermination() <br>  } <br>}</pre><p>Feel free to subscribe to my YouTube channel i.e <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/stateful-transformations-spark-streaming-part-1-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a18c8657a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/stateful-transformations-in-spark-streaming-part-1-spark-streaming-session-3-a18c8657a3">Stateful transformations in Spark Streaming — Part 1 | Spark Streaming Session 3</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Streaming: Session 2]]></title>
            <link>https://medium.com/analytics-vidhya/spark-streaming-session-2-8f07616ed364?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f07616ed364</guid>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Sun, 05 Feb 2023 10:06:55 GMT</pubDate>
            <atom:updated>2023-02-05T10:06:55.003Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*BPts1fZIOjvGOjhorOqeuQ.png" /></figure><p>In the first article about the spark streaming series, we have understood the following important concept.</p><p>We have also written word count code to understand these concepts. For those who are reading this article without reading the first article of this series, I recommend reading the previous article or watching the video before starting this article. I also created a YouTube video for the same concepts. Please follow these links.</p><p><strong>First session links:</strong></p><p>Here in this session, we will mainly focus following concepts.</p><ol><li>Different streaming sources</li><li>Stateful vs Stateless transformation</li></ol><p>So let’s start and discuss these concepts in very layman’s terms.</p><h3>Different Streaming Sources</h3><p>Spark Streaming mostly provides two types or categories of built-in streaming sources.</p><ul><li>Basic Source — Socket connection, File system</li><li>Advance Source — Kafka, Pub/Sub, Kinesis</li></ul><p>Basic source like socket connection &amp; File System comes in very handy when you are learning streaming concepts. These are very easy to set up for producing streaming data.</p><p>We have already seen one example of a socket in the word count example code where we have tried to produce data using <strong><em>nc -lk 9999</em></strong> command. Here we have just treated localhost with port 9999 as our source.</p><p>Producing stream data from a file-based system is also very easy. Here, one event just keeps adding new files to the source folder and the spark streaming application just keeps looking(pooling) for the new file after a certain interval of time.</p><p>Basic sources are directly available in the StreamingContext API.</p><p>Advanced sources like Kafka require some more effort in setting the Kafka cluster on your local system.</p><p>Advance Sources like Kafka, Kinesis, etc. are available through extra utility classes. These require linking against extra dependencies.</p><p>With time, we will explore all these sources by our code. So don’t worry if you are not getting some of the concepts related to these sources. For now, just understand that there are different sources that help to produce data in the streaming use cases.</p><p>Now, Let’s try to understand the concept of Window and sliding interval before starting the stateless vs stateful transformation discussion.</p><h3>Window Operations</h3><p>Spark Streaming also provides <em>windowed computations</em>, which allow you to apply transformations over a sliding window of data.</p><ul><li><strong>Window Length: </strong>The duration of the window in secs</li><li><strong>Sliding Interval:</strong> The interval at which the window operation is performed in seconds. These parameters must be <strong>multiples</strong> of the <strong>batch interval</strong>.</li></ul><p>Let’s check the following pics &amp; then we will understand it by one example.</p><p>Let’s try to make it more clear from one example :)</p><p>You remember in the last article we have written world count code.</p><ul><li>There we have used <strong>batch duration</strong> as 5 sec.</li><li>It means that after 5 seconds a new RDD will be created.</li><li>Let’s suppose, our <strong>window size</strong> is 10 seconds. It means that we are only interested in the last 2 RDD’.</li><li>Now suppose that <strong>Sliding Interval</strong> is 1 second. It means that after every 1 second one old RDD will go away and one new RDD will come in.</li></ul><p>I hope now you have some understanding of window time and sliding interval. Now let’s understand stateless and stateful transformation.</p><h3>Stateless Transformation</h3><ul><li>It forgets the previous state.</li><li>Here we perform operations on a single RDD always.</li><li>We are not worried about past records. Our calculations always involve the current records.</li><li>So every transformation in batch processing is stateless transformation.</li></ul><h3>Stateful Transformation</h3><ul><li>Here we do aggregation over more than one RDD.</li><li>When we talk about stateful transformation then we have two choices. We can either do aggregations or transformation on entire streams or we can consider some windows time.</li><li>Consider we have a streaming application that runs for 1 hour with a batch duration equal to 5 minutes. It means that a new RDD will be created every 5 minutes. For one hour it will create 12 RDDs.</li><li>If we want to do operations over the last 10 minutes then we will only consider 2 RDDs always.</li></ul><p>Let’s suppose we want to apply <strong>sum() </strong>on incoming streaming data then it is a stateless transformation but let’s suppose you want to calculate the <strong>running sum</strong> then it is a stateful operation.</p><p>For stateful operations, you will be two choices.</p><p>You can just calculate the running sum of entire streams. For example in the above example, we do have an entire streaming time equal to 1 hour i.e. 12 RDDs. We can apply one stateful transformation and find the running choice of 1-hour streaming data.</p><p>Another choice can be the use of windows time and sliding interval. Here with each sliding interval old records will be removed from the calculation and new records will be added. Here we will only consider the last results like the last 10 minutes running sum or the last 30 minutes running sum with 2 minutes of sliding interval time.</p><p>I hope now you have got some idea about StateFul vs StateLess transformation and you have now a basic understanding of windows time and sliding interval.</p><p>In the next article, we will see some of the stateful transformation examples like</p><ul><li>reduceByWindow</li><li>updateStateByKey</li><li>reduceByKeyAndWindow</li><li>countByWindow</li></ul><p>We will write code to understand the above transformation. Let’s meet in the next articles for discussing more Spark Streaming.</p><p>Feel free to subscribe to my YouTube channel i.e <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I have started the Spark Streaming playlist and I am explaining its core concepts by means of video illustration.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/spark-streaming-session-2-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f07616ed364" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/spark-streaming-session-2-8f07616ed364">Spark Streaming: Session 2</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Streaming — Part 1]]></title>
            <link>https://medium.com/analytics-vidhya/spark-streaming-part-1-48d55c694d8c?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/48d55c694d8c</guid>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Sun, 05 Feb 2023 10:05:20 GMT</pubDate>
            <atom:updated>2023-02-05T10:05:20.099Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spark Streaming — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*I46mpBBJPgYtf3_LHVQy6A.png" /></figure><p>A few months back, I was given a codebase that used Spark Streaming and it was written in scala. We were supposed to make major changes and go for the new version of the project. I had worked on Spark Structure Streaming before but not in Spark Streaming. I found that, as Spark Streaming is a legacy project, Apache foundation has stopped giving further updates on this. Hence there are very few good articles on Spark Streaming which talks about its functionalities, problems and use cases in layman’s terms in good depth.</p><p>I think that understanding Spark Streaming is a good way to enter into the Streaming world as it is very easy to learn. It is also written using a lower level of code, hence it’s very easier to visualize the concepts of streaming. So let’s start learning Spark streaming in very layman’s terms using this series of articles.</p><p><em>This article assumes that you know the basic terms of Spark and you have written some code for batch processing using spark.</em></p><h3>What is Spark Streaming?</h3><ul><li>Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams.</li><li>Data can be ingested from different sources like Socket, File Source, Kafka, Kinesis etc and can be processed with high-level functions like map, reduce, join etc of Spark Streaming.</li><li>Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.</li><li>Spark Streaming provides a high-level abstraction called <em>discretized stream</em> or <em>DStream, </em>which represents a continuous stream of data.</li><li>DStream is nothing but a continuous stream of RDDs (Spark abstraction). One can consider it as Seq[RDD]. Every RDD in DStream contains data from a certain interval.</li><li>Any operation on a DStream applies to all the underlying RDDs. It covers all the details. It provides the developer with a high-level API for convenience.</li></ul><p><strong><em>But Ankur</em></strong>, how do we understand this flow of live data and spark streaming concept in simpler terms?</p><p>There are mostly the following terms that one has to understand for understanding the flow of live data with Spark Streaming.</p><p>Let’s try to understand using one of my experiences. I started my career with one of the start-ups in Mumbai(India). I used to daily see people collecting water from the public water system.</p><p>Let’s suppose Bombay Municipal Corporation(BMC) allowed tap to run for the whole day then we can consider it as a producer producing data continuously. This is like a streaming use case where data is flowing continuously.</p><p>Now let’s suppose, we consider the water tab running time equal to 1 hour for our calculation &amp; consider a scenario where every person is given 2 minutes to fill their bucket.</p><p>Let’s visualize this above situation with a streaming use case. Here <strong>2 minutes</strong> will be treated as <strong>batch duration</strong> and <strong>every bucket</strong> as one <strong>RDD</strong>. <strong>Droplets of water</strong> will be considered <strong>messages or records</strong>.</p><p>As we know the flow of water might not be the same every time. So some buckets may contain more water and some might contain less water. Same way in the case of streaming some RDD might have <strong>more messages</strong> and some might have <strong>fewer messages</strong>. So one batch duration, we might have more messages whereas, in the next batch duration, we might receive fewer messages.</p><p>Now as the size of the stream is 1 hour and the batch duration is 2 minutes. It means that 30 people can fill their 30 buckets.</p><p>Here groups of buckets can be considered as DStream and each bucket as one RDDs.</p><p>So basically,</p><p>DStream -&gt; RDD -&gt; Messages (entities)</p><p>So here it has 30 RDD and each RDD can have lots of messages but when we use Spark Streaming then operations are performed as on DStream. Any operation on a DStream <strong>applies to all the underlying individual RDDs</strong>.</p><blockquote><em>So underneath spark still works in a batch style, usually batch size is so small that we get the feeling that it is real-time streaming.</em></blockquote><p>I hope my enthusiastic readers now understand the meaning of <strong>batch duration, messages, RDD and DStream</strong>.</p><h3>A Quick Example</h3><p>Before we start discussing the concepts of Spark Streaming. Let’s see one quick code. I think a word count example in any computing engine is a really good way to start.</p><ul><li>Step 1: Open your console or terminal and just use <strong><em>nc -lk 9000</em></strong> command to simulate producing data.</li><li>Step 2: Now our producer is ready, we have to start our consumer application for listening to our data. Here Spark Streaming will act as a consumer.</li><li>Step 3: In Spark Streaming one has to start the Streaming Context. StreamingContext is the main entry point for all streaming functionality. Here, we will create a local StreamingContext with two execution threads, and a batch interval of 5 seconds.</li><li>Step 4: Then we will create a DStream that will connect to hostname: port, like localhost:9000</li></ul><p>For our word count example, we will simulate the producer by our console.</p><p>Socket =&gt; IP Address + Port number =&gt; { localhost + 9000}</p><p>Let’s try to look into the code and comments in the code snippet for understanding the flow. The below code is written in Scala.</p><pre>val BATCH_INTERVAL_TIME = 5 def main(args: Array[String]): Unit = { <br><br>// Inititating spark session which will further help us to start <br>// spark streaming context <br><br>val spark: SparkSession = SparkSession.builder().master(&quot;local[*]&quot;).appName(&quot;theBigDataShow.com&quot;).getOrCreate() <br><br>// Setting spark error log level. This helps to debug in console. <br>spark.sparkContext.setLogLevel(&quot;ERROR&quot;)<br><br>// Start the streaming context. <br>// This it entry point for spark streaming.<br>// Here In our code base batch interval is 5 seconds. <br>// It means after every 5 seconds. <br>// New RDD will be created. <br><br>val sc: SparkContext = spark.sparkContext <br><br>val ssc = new StreamingContext(<br>  sc, <br>  batchDuration = Seconds(BATCH_INTERVAL_TIME) <br>) <br><br>// One can use checkpointing but this is very basic example so I <br>// have just commented it. <br><br>// ssc.checkpoint(&quot;./checkpointing&quot;) <br><br>// Using streaming context, we can create a DStream that <br>// represents streaming data from a TCP source, <br>// specified as hostname (e.g. localhost) and port (e.g. 900). <br><br>val lines: ReceiverInputDStream[String] = ssc.socketTextStream( &quot;localhost&quot;, 9000 ) <br><br>// This lines DStream represents the stream of data that <br>// will be received from the data server. <br>// Each record in this DStream is a line of text. <br>// Next, we want to split the lines by space characters into words. <br><br>val words: DStream[String] = lines.flatMap(x =&gt; x.split(&quot; &quot;)) <br><br>// flatMap is a one-to-many DStream operation that creates a <br>// new DStream by generating multiple new records from each <br>// record in the source DStream. In this case, each line will <br>// be split into multiple words and the stream of words is <br>// represented as the words DStream. Next, we want to count <br>// these words. <br><br>val wordsFrequencyMap: DStream[(String, Int)] = words.map(w =&gt; { (w, 1) }) <br><br>// Count each word in each batch <br><br>val wordsCounts: DStream[(String, Int)] = wordsFrequencyMap .reduceByKey(_ + _) <br><br>// Print the first ten elements of each RDD generated in this <br>// DStream to the console <br><br>wordsCounts.print() <br><br>ssc.start() <br>ssc.awaitTermination() <br><br>}</pre><ul><li>Stateless and Stateful Transformation</li><li>We will understand the difference b/w stateless and stateful transformation by using <strong>UpdateStateByKey</strong> and <strong>reduceByKey </strong>transformations.</li><li>Concept of Window Size and Sliding Interval.</li><li>We will use the <strong>reduceByKeyAndWindow</strong> and <strong>reduceByWindow</strong> for understanding window size and sliding interval concepts.</li><li>Joins in Streaming like stream-stream join, Stream-dataset join</li><li>Output Operations in Spark Streaming</li></ul><p>In this article, we have tried to build the foundation of Spark Streaming i.e. DStream, Rdd, Messages/Entity/Records.</p><p>In the next article of this series, we will try to understand some more concepts like.</p><p>I hope this small article sparks some interesting thoughts w.r.t to Spark Streaming. Let’s meet in the next articles for discussing the above topics.</p><p>Feel free to subscribe to my YouTube channel i.e <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/spark-streaming-part-1-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48d55c694d8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/spark-streaming-part-1-48d55c694d8c">Spark Streaming — Part 1</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Disaster Recovery in Kafka Servers]]></title>
            <link>https://towardsdev.com/disaster-recovery-in-kafka-servers-a2cfaa5ac3b?source=rss-5ab3898edc74------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2cfaa5ac3b</guid>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[mirrormaker]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ankur Ranjan]]></dc:creator>
            <pubDate>Sun, 25 Dec 2022 06:48:01 GMT</pubDate>
            <atom:updated>2023-02-14T17:57:23.540Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwTKSYCkS68C95fINF2plQ.png" /></figure><p>I recently tried onboarding disaster management for our streaming pipeline which involves Kafka, Spark Streaming and MongoDb in one of our use cases at <a href="https://medium.com/u/c884135151a4">Walmart Global Tech</a></p><p>The last few weeks were a good learning curve for me and I really enjoyed all these awesome implementations of the streaming pipeline. I am still trying to make it more scalable and cover all other edge cases. If you have also used the below concept to manage the disaster for your Kafka Cluster then feel free to correct my mistake and help me to cover more minute details. So let’s discuss our content in a brief way and then we will try to understand it deeply.</p><p>In this article, we will try to understand the following things.</p><ol><li>How Kafka provides <strong>fault toleranc</strong>e even if it is deployed in single clusters?</li><li>How can we use <strong>two different regions with Mirror Maker 2(MM2)</strong> capabilities to achieve disaster recovery in case of some failure at a region where the Kafka cluster is deployed? We will be discussing two implementations for managing Disasters. Those are <strong>Active Active</strong> and <strong>Active Passive</strong>.</li></ol><h3>Let’s try to understand how does Kafka provide fault tolerance, even with a single region server?</h3><p>Kafka Cluster is already reliable using the following things.</p><ul><li>Replication Factor</li><li>Rack Awareness</li></ul><p>In Kafka, messages are pointed to one Kafka topic which is further divided into multiple partitions. Each partition will be present in different brokers hence the records will be always distributed across different nodes.</p><p>Let’s try to understand the above concept using one simple example.</p><p>Let’s suppose we are having <strong>2 racks</strong> and <strong>6 brokers clusters</strong>. Now let’s suppose we have created some topic on this cluster with the name <strong>topic1.</strong></p><p>While creating the topic we decided that there will be 10 partitions &amp; replication factor will be three.</p><p>So We have <strong>10 * 3 = 30</strong> replica <a href="https://confluence.walmart.com/pages/createpage.action?spaceKey=GGDL&amp;title=%28p0-p9%2C+p1-p9%2C+p2-p9%29&amp;linkCreation=true&amp;fromPageId=1251384715">(p0-p9, p0`-p9`, p0``-p9``)</a></p><p>Let suppose in</p><h3>Now the question is how partitions of these topics are distributed across various brokers &amp; how fault tolerance is guaranteed using the replication factor.</h3><p>For distributing partitions of topics across various brokers, Kafka takes care of the following things</p><ol><li>Even Distributions (Achieving Workload balance)</li><li>Fault tolerance (Replicas should be stored on a different machine)</li></ol><p>For even distributions, Kafka will create an <strong>ordered list</strong> of available brokers.</p><p>R1 → B0</p><p>R2 → B3</p><p>R1 → B1</p><p>R2 → B4</p><p>R1 → B2</p><p>R2 → B5</p><p>There will be actually two types of partitions.</p><ul><li><strong>Leader Partitions </strong>— The leader performs the task of all reading and writing request</li><li><strong>Followers Partitions</strong> — while the followers passively replicate the role of a leader</li></ul><p>So let’s suppose that partitions(p0-p9) are the leader’s partitions and (p0`-p9`, p0``-p3``) are followers’ partitions for now.</p><p><strong>Leaders</strong> — p0,p1,p2,p3,p4,p5,p6,p7,p8,p9</p><p><strong>Followers</strong> — p0`,p1`,p2`,p3`,p4`,p5`,p6`,p7`,p8`,p9`</p><p>- p0``,p1``,p2``,p3``,p4``,p5``,p6``,p7``,p8``,p9``</p><p>Let’s see the even distributions of topic partitions and their replications on all brokers.</p><p>This round-robin fashion will determine that on every broker’s topics partitions are <strong>evenly distributed</strong>. So, this determines that our first point i.e. Distributions (Achieving Workload balance)</p><p>Now, Let’s try to understand, how fault tolerance will be achieved using the replication factor after the evenly distribution of partitions across different brokers and racks.</p><p>Let’s understand the following steps which Kafka uses for fault tolerance.</p><ul><li>The topic is divided into partitions and kept in different brokers.</li><li>If any broker fails, data should not be lost.</li><li>For fault-tolerance purposes, the partition is replicated and stored in different brokers.</li><li>If leader brokers fail, the controller will elect one of the replicas as the leader.</li><li>Even controller brokers can fail, in this case, the Zookeeper will help in selecting the broker as the controller.</li></ul><p><strong>Controller:</strong> It’s just a normal broker with extra responsibility. The controller Broker takes care of electing the leader broker for the partitions. The controller broker keeps track of brokers joining and leaving the cluster with the help of the Zookeeper.</p><p><strong>These above two concepts help Kafka to provide fault tolerance. But how it be fault tolerant if the whole server is down in one region? This creates a single point of failure and if your application uses Kafka intensively in your application then disaster in one region can create panic in your system and in our use case we could afford to lag that much. We are responsible for the Seller and Item onboarding smoothly after detecting any fraud and in case of failure on our side Seller and Item will be impacted.</strong></p><p>For managing the disaster, we have used the following things.</p><p>Let’s try to understand these concepts.</p><p>When I used to work as Software Developer, I set up Postgres Sever &amp; I implemented an <strong>active-passive</strong> setup. it was a standard active-passive cluster where the application reads and writes to a primary cluster. This primary cluster is asynchronously replicated to a secondary cluster. The application falls back to the secondary cluster in case the primary cluster fails.</p><p>The fallback can be automatic or manual. Database vendors such as Postgres, Oracle, MSSQL, etc have built products to automate the failover. Same way, newer distributed applications, such as NoSQL databases like Cassandra prefer an <strong>active-active</strong> approach. This is because these databases might not require very high consistency.</p><p>I found near to the same implementation with Kafka too. So <strong>active-passive</strong> and <strong>active-active</strong> setups are very standard implementations. One does implement this in their API management and database management. In the same way, we will be implementing these concepts in Kafka.</p><p>But Ankur, make this concept a little easier for me🫣. What do <strong>active-passive</strong> and <strong>active-active</strong> actually mean?</p><ul><li><strong>Active Active</strong> — It addresses a disaster recovery approach where consumers reconnect to a new cluster after the first one fail</li><li><strong>Active Passive </strong>— In this deployment, the participant’s cluster has its own producers and consumers, and topics are replicated. Here messages are processed by different consumers in the two data centres.</li></ul><p>Now, let’s try to understand the MirrorMaker in little brief way.</p><h3>What is Mirror Maker</h3><p>In simple terms, Mirror Maker allows you to mirror a Kafka cluster to another cluster. All topics, partitions, and messages are replicated. Any changes in the source, like the addition/deletion of topics, messages will be mirrored to the destination. Mirror Makes allows Kafka users to set up an active-passive cluster where an active cluster is continuously mirroring the data to a secondary cluster. We can set up replication unidirectional or bidirectional(using MM2).</p><h3>What is the problem?</h3><p>All is good until now <strong>when we realize that a huge amount of infrastructure and network bandwidth is being wasted to maintain this active-passive cluster</strong>, wherein the passive cluster will come into use only when the active servers go down. Not to mention that large Kafka deployments process millions of messages per sec and all that data needs to be replicated to the destination cluster.</p><h3>Solution — Mirror Maker 2.0</h3><p>The Kafka team has recently launched a new version of Mirror Maker 2.0, which allows you to set up a bi-directional mirror between two clusters. This solves the problem of the wasted network infrastructure of the second cluster which would have come into use only if the primary cluster had gone down.</p><p>So, how does this work? In simple terms, Mirror Maker 2.0 does bi-directional mirroring, i.e. data from the primary cluster will be mirrored to the secondary cluster, and data from the secondary cluster will be mirrored to the primary cluster.</p><p>If you logically think of this approach, this can result in an infinite loop, where in a message can be continuously mirrored between two clusters.</p><p>To solve, this problem, Mirror Maker allows topic renaming to eliminate the above problem. Let me illustrate how:</p><p>Let’s say, We have one server in South Central US( <strong>SCUS</strong>) and another in West US( <strong>WUS</strong>). Now if we have enabled MirrorMaker 2.0 (MM2) while creating any topic then instead of creating 2 topics, It will create 4.</p><p>Let’s suppose we have created one Kafka topic with the name <strong>topic1 </strong>then normally there will be the following topics created.</p><p>Here</p><p>Here Disaster recovery(DR) topics only have read-only permission whereas the main topic will have write and read-only permission.</p><p>Here let’s suppose some producer writes records to <strong>topic1(SCUS</strong>) then all records from this topic will be replicated to its corresponding DR topic i.e. <strong>kafka-v2-appname-scus.topic1(WUS)</strong>. The same applies to if some producer writes to <strong>topic1(WUS)</strong> then it will be replicated to its DR topic i.e. <strong>kafka-v2-appname-wus.topic1(SCUS)</strong></p><p><strong>Now we know that replication can be taken care of by MirrorMaker2 then how as an application team can use these replication benefits in implementing disaster recovery?</strong></p><p>We can actually do this using any one of the designs i.e.</p><h3>Active-Active implementation.</h3><p>In active-active, In one Producer will write messages in topic1(SCUS) and another producer will produce messages in topic1(WUS).</p><p>Note that these two are completely different clusters with zookeepers of their own and hence topics are local to their clusters.</p><p>MirrorMaker2(MM2) will keep replicating records to the DR topic bidirectionally.</p><p>One can see from the above pics that</p><ul><li>This model implies there are two clusters with <strong>bidirectional mirroring</strong> between them.</li><li>Data is asynchronously mirrored in both directions between the clusters.</li><li>Client applications are <strong>aware of several clusters</strong> and <em>can</em> be ready to switch to another cluster in case of a single cluster failure.</li><li>Client requests are processed by <strong>both clusters</strong>.</li></ul><p>In this case, the full network cluster bandwidth is utilised. The <strong>producers</strong> can load balance their traffic to two clusters either using <strong>round robin or local affinity</strong>.</p><p>The consumer in SCUS will have to subscribe to the data from both T1 and T1.WUS. This can be done easily using a wild card subscription which is supported in most frameworks. One pseudo-code can be</p><pre>@KafkaListener(id = &quot;xxx&quot;, topicPattern = &quot;kbgh.*&quot;) <br>public void listen(String in) { <br>    System.out.println(in); <br>}</pre><p>In the case of DR or Failure in a single cluster.</p><ul><li>The other one continues to operate properly with no downtime, providing high availability.</li><li>Data between clusters is eventually consistent, which means that the data written to a cluster won’t be immediately available for reading in the other one.</li><li>In case of a single cluster failure, some acknowledged ‘write messages’ in it may not be accessible in the other cluster due to the asynchronous nature of mirroring.</li></ul><p>But implementing Active Active means contract changes from the team who might be producing records in one region only. Now they have to make their code changes to set up producers that can send data to both regions. Awareness of multiple clusters for client applications is also required. <strong>Active-Active</strong> also involves the complexity of bidirectional mirroring between clusters</p><p>So let’s try to see the Active Passive setup in its pros and cons.</p><p><strong>Active Passive</strong></p><ul><li>The <em>active-passive</em> model suggests there are two clusters with unidirectional mirroring between them.</li><li>Data is asynchronously mirrored from an active to a passive cluster.</li><li>Client applications are aware of several clusters and <em>must</em> be ready to switch to a passive cluster once an active one fails.</li><li>Client requests are processed only by an active cluster.</li></ul><p>Here code changes are minimal but it needs manual intervention in case of downtime and the Lag may increase if developers are not able to set up the configurational changes like changing the brokers and topic information in a given amount of time.</p><p>Let’s see one approach which can be used here.</p><p>In the case of DR in SCUS region.</p><ul><li>The consumer will make the configuration information changes and start reading from the DR topic of WUS i.e.<strong> kafka-v2-appname-scus.topic1. </strong>It will read all the records which were left from the <strong>topic1(SCUS)</strong>. This can be set up using offset reset based on some timestamp.</li><li>The producer will start writing to topic1(WUS) instead of topic1(SCUS).</li><li>The consumer will switch back to topic1(WUS) once it does read all records from the DR topic of topic1(SCUS).</li></ul><p>So this ends our short discussion on disaster management strategy with Kafka and Mirror Maker 2.0.</p><p>Feel free to subscribe to my YouTube channel i.e <a href="https://www.youtube.com/@TheBigDataShow/">The Big Data Show</a>. I might upload a more detailed discussion of the above concepts in the coming days.</p><p>More so, thank you for that most precious gift to a me as writer i.e. your time.</p><p><em>Originally published at </em><a href="https://www.linkedin.com/pulse/disaster-recovery-kafka-servers-ankur-ranjan/"><em>https://www.linkedin.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2cfaa5ac3b" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/disaster-recovery-in-kafka-servers-a2cfaa5ac3b">Disaster Recovery in Kafka Servers</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>