<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amar_Kumar on Medium]]></title>
        <description><![CDATA[Stories by Amar_Kumar on Medium]]></description>
        <link>https://medium.com/@amar.kumar2013?source=rss-9002d419cfa4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pHFn-__KYtAE3kR6T9Ls1Q.jpeg</url>
            <title>Stories by Amar_Kumar on Medium</title>
            <link>https://medium.com/@amar.kumar2013?source=rss-9002d419cfa4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:34:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amar.kumar2013/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ How I Broke (and Then Fixed) My S3 Data Lake: A Hard Lesson in Partitioning]]></title>
            <link>https://medium.com/@amar.kumar2013/how-i-broke-and-then-fixed-my-s3-data-lake-a-hard-lesson-in-partitioning-b6c2b5388f07?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6c2b5388f07</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Wed, 15 Oct 2025 17:23:49 GMT</pubDate>
            <atom:updated>2025-10-15T17:23:49.138Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>I still remember my first major data lake deployment on Amazon S3.<br>We were managing <strong>IoT sensor readings</strong> from manufacturing lines — millions of events per day.</blockquote><blockquote>I confidently set up our folder structure like this:</blockquote><pre>s3://factory-data-lake/sensor_readings/year=2025/month=03/day=10/data.parquet</pre><p>Everything looked neat.<br>You could drill down to any date intuitively.<br>The team was impressed.</p><p>Until the first large query hit.</p><h3>⚠️ When Your “Perfect” Partitioning Turns Against You</h3><p>An analyst asked me:</p><p><em>“Can you give me temperature readings for the last 60 days?”</em></p><blockquote>A simple enough request — but our query engine (Athena) crawled through <strong>thousands of partitions</strong>, taking minutes to return a few MBs of data.</blockquote><p><strong>What I thought was “organized” turned out to be the root cause of our performance bottleneck.</strong></p><p>❌ Bad Partitioning: year/month/day</p><pre>📦 s3://factory-data-lake/<br> ┣ 📁 year=2025/<br> ┃  ┣ 📁 month=03/<br> ┃  ┃  ┣ 📁 day=10/<br> ┃  ┃  ┗ 📁 day=11/<br> ┗ 📁 year=2024/</pre><p>A simple query like:</p><pre>SELECT * FROM sensor_readings<br>WHERE event_date BETWEEN &#39;2025-02-01&#39; AND &#39;2025-03-10&#39;;</pre><p>…turns into an ugly WHERE clause behind the scenes with dozens (sometimes hundreds) of OR statements.</p><p><strong>Each “day” means a new partition scan, a new S3 API call, and a longer query time.</strong></p><h3>🧠 Why It Happens: S3 ≠ File System</h3><p>S3 is not HDFS or your local filesystem.<br> It’s an <em>object store</em>, and every folder you “see” is actually a <strong>prefix</strong> that must be resolved by an API call.</p><pre>| Concept        | HDFS               | S3                        |<br>| -------------- | ------------------ | ------------------------- |<br>| Metadata       | Stored in NameNode | Distributed via API calls |<br>| Folder listing | Local &amp; fast       | Remote &amp; rate-limited     |<br>| Hierarchy      | Physical           | Logical                   |<br>| Scaling        | Node-based         | Prefix-based              |</pre><p>In my “year/month/day” setup, every query touched <strong>tens of thousands of prefixes</strong>. Each prefix listing added latency and cost.</p><h3>✅ The Fix: Partition by Full Date</h3><p>After multiple painful debugging sessions, we restructured everything to partition by the full date (ISO 8601 format):</p><pre>s3://factory-data-lake/sensor_readings/dt=2025-03-10/data.parquet</pre><p>Now, our queries looked clean:</p><pre>SELECT * <br>FROM sensor_readings<br>WHERE dt BETWEEN &#39;2025-02-01&#39; AND &#39;2025-03-10&#39;;</pre><p>✅ <strong>Simple query</strong><br> ✅ <strong>Better partition pruning</strong><br> ✅ <strong>Massive performance boost</strong></p><p>Our query runtime dropped from <strong>7 minutes to 25 seconds</strong>, and partition counts fell by 80%.</p><p><strong><em>Comparison</em></strong></p><pre>| Aspect              | Old (`year/month/day`) | New (`dt=YYYY-MM-DD`) |<br>| ------------------- | ---------------------- | --------------------- |<br>| Query readability   | Complex OR chains      | Simple range filters  |<br>| Partition pruning   | Poor                   | Excellent             |<br>| S3 API calls        | Thousands              | Minimal               |<br>| Query length errors | Frequent               | None                  |<br>| Maintenance         | Painful                | Easy                  |</pre><h3>⚙️ Hidden Bottleneck: S3 Prefix Throttling</h3><p>Even after fixing partitioning, we hit another issue:<br> <strong>S3 started throwing “Slow Down” and “Internal Error” messages during heavy writes.</strong></p><p>Turns out, S3 has <em>prefix-level rate limits</em>:</p><ul><li>🧾 3,500 PUT/POST/DELETE requests per second <strong>per prefix</strong></li><li>🔍 5,500 GET/HEAD requests per second <strong>per prefix</strong></li></ul><p>When multiple Spark jobs wrote to the same prefix (e.g., dt=2025-03-10/), we hit the limit.</p><p>Result: retries, random failures, and massive frustration.</p><h3>🧊 Another Approach: Open Table Format</h3><h3>How Apache Iceberg Fixed It</h3><p>Iceberg allows you to switch from Hive-style storage to <strong>ObjectStoreLocationProvider</strong>, which automatically <strong>hashes file paths</strong> to distribute them evenly across prefixes.</p><pre>📦 s3://factory-data-lake/<br> ┣ 📁 sensor_readings/<br> ┃  ┣ 📁 data/<br> ┃  ┃  ┣ 📁 ab/<br> ┃  ┃  ┣ 📁 cd/<br> ┃  ┃  ┗ 📁 ef/<br> ┃  ┗ 📁 dt=2025-03-10/</pre><p>Each file gets stored under a <em>different subprefix</em>, avoiding throttling and spreading load evenly.</p><p>✅ Stable throughput<br> ✅ No prefix hot-spotting<br> ✅ Linear scalability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wwpTtjLn0bIUpvHgZmv4bg.png" /><figcaption>Types of Partitioning</figcaption></figure><p>📘 <strong>Learn more:</strong></p><ul><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html">AWS Docs: Using Prefixes</a></li><li><a href="https://iceberg.apache.org/docs/latest/aws/#internal-table-design">Apache Iceberg AWS Internal Design</a></li></ul><h3>🧱 Practical Checklist for Smarter Partitioning</h3><p>Here’s what I follow now on every new data lake project:</p><ol><li>🗓️ <strong>Partition by real-world query filters</strong> (usually full date).</li><li>📦 <strong>Avoid small files</strong> — use compaction jobs regularly.</li><li>🔍 <strong>Monitor partition counts</strong> →10k = red flag.</li><li>🧊 <strong>Adopt modern table formats</strong> — Iceberg, Delta, or Hudi.</li><li>⚗️ <strong>Benchmark early</strong> — test queries before finalizing the layout.</li></ol><h3>🎯 Wrapping It Up</h3><p>What I learned the hard way:</p><p><em>Good partitioning isn’t about organization — it’s about </em>optimization<em>.</em></p><ul><li>❌ Stop using year/month/day layouts blindly.</li><li>✅ Use ISO 8601 date-based partitions (dt=YYYY-MM-DD).</li><li>🧊 Leverage Iceberg’s hash distribution for heavy write pipelines.</li><li>☁️ Treat S3 as an <strong>object store</strong>, not a <strong>filesystem</strong>.</li></ul><p>Now, our queries fly, our jobs run stable, and our analysts don’t curse at Athena anymore.</p><p>So the next time someone suggests:</p><p><em>“Let’s partition by year, month, and day!”</em></p><p>Just smile and say:</p><p><em>“Let’s not build a 2010 data lake in 2025.” 😎</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6c2b5388f07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Catching Fake Invoices with Agentic AI on Google Vertex AI]]></title>
            <link>https://medium.com/@amar.kumar2013/catching-fake-invoices-with-agentic-ai-on-google-vertex-ai-8d8431befaba?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d8431befaba</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Fri, 11 Jul 2025 10:52:18 GMT</pubDate>
            <atom:updated>2025-07-11T10:52:18.970Z</atom:updated>
            <content:encoded><![CDATA[<h3>🤖 What If You Had an AI Auditor?</h3><p>Imagine an AI system that acts like a team of trained auditors, each expert in spotting a specific kind of fraud in digital invoices:</p><ul><li>One checks GSTIN numbers 🧾</li><li>Another inspects fonts and alignment 🔠</li><li>A third looks for evidence of editing ✂️</li><li>And one makes sure the math actually checks out 🧮</li></ul><p>This is <strong>Agentic AI</strong>: not just a single AI model, but a <strong>team of agents</strong> that work together to detect tampering — all powered by Google Cloud’s <strong>Vertex AI</strong>.</p><p>Thanks for reading! Subscribe for free to receive new posts and support my work.</p><h3>🧠 First, What is Agentic AI?</h3><blockquote><em>Think of it like a smart factory with specialized robots. Each robot does one job extremely well. Together, they build a reliable product.</em></blockquote><h3>Agentic AI is:</h3><ul><li>🎯 <strong>Goal-driven</strong> — Each agent has a job (e.g., validate GSTIN, check formatting).</li><li>🔍 <strong>Interpretable</strong> — Each one can explain <em>why</em> it made a decision.</li><li>🧩 <strong>Composable</strong> — You can add/remove agents as needed.</li><li>🤝 <strong>Collaborative</strong> — All results are aggregated into one final decision.</li></ul><h3>🚀 Use Case: Detecting Tampered Invoices</h3><p>Here’s the journey of an invoice through our Agentic AI system 👇</p><h3>🖼️ Step 1: Image is Retrieved from Cloud Storage</h3><p>Invoices are typically uploaded as images to a secure <strong>Google Cloud Storage</strong> bucket.</p><p>Our system starts by pulling each image from this repository. It knows the naming convention (like “InvoiceImages/123@1.jpg”) and fetches the correct files for processing.</p><h3>📸 Step 2: Image is Decoded &amp; Read Like a Document</h3><p>We convert the image into a base64 format, like turning it into a digital fingerprint, and pass it to <strong>Google Vision API</strong>.</p><p>This service acts like <strong>AI-powered eyes</strong>, extracting all the text from the image — even noting:</p><ul><li>What the text says</li><li>Where it’s located</li><li>How confident it is</li></ul><p>This lets us highlight tampered areas later 🔍</p><h3>🧠 Step 3: Gemini Understands the Content</h3><p>Next, the <strong>Gemini 2.0 model</strong> (running via <strong>Vertex AI</strong>) reads the invoice like a human would:</p><blockquote><em>“Oh, I see! This is the supplier name, here’s the total amount, and this must be the GSTIN.”</em></blockquote><p>Gemini parses the invoice and outputs a <strong>structured JSON</strong> with:</p><ul><li>Supplier name &amp; address</li><li>Receiver name</li><li>GSTIN number</li><li>Line items</li><li>Quantities, rates, totals</li><li>Tax values</li></ul><p>This gives us machine-readable data to validate.</p><h3>🧩 Step 4: Meet the AI Agents</h3><p>Now that we know what’s on the invoice, it’s time for each <strong>specialist AI agent</strong> to do their job.</p><pre>| Agent Name                 | Role                    <br>|----------------------------|------------------------------------------------<br>| SupplierValidationAgent  🏢| Checks supplier name, address, and GSTIN       |<br>| DigitalEditingAgent  ✂️    | Spots inconsistent text and formatting edits   | <br>| ManualEditingAgent  ✍️     | Detects handwriting or criss-cross marks       | <br>| FontConsistencyAgent  🔠   | Verifies fonts and alignments                  | <br>| AppearanceAgent   🎨       | Flags faded or overly bolded text              | <br>| AccuracyAgent 📊           | Validates totals, rates, quantities            | <br>| SpellingAgent 🔤           | Checks for typos and misspellings              | <br>| HandwrittenAgent 📝        | Flags handwritten invoices</pre><p>Each agent gives:</p><ul><li>✅ Pass or ❌ Fail</li><li>📄 Explanation</li><li>🧷 Field(s) responsible</li></ul><h3>🧠 Step 5: Orchestrator Makes the Final Decision</h3><p>The <strong>Orchestrator Agent</strong> acts like the manager:</p><blockquote><em>“Okay team, what did we find? Let’s combine your reports and decide if this invoice is legit or not.”</em></blockquote><blockquote>It reviews every agent’s result and builds a final report:</blockquote><ul><li>✅ Overall result (Pass/Fail)</li><li>🗂️ Detailed explanations</li><li>📍 Coordinates of suspect fields (so we can highlight them!)</li></ul><h3>📊 Step 6: Save the Report to BigQuery</h3><p>All the results are logged into <strong>Google BigQuery</strong> for further analysis:</p><ul><li>Build fraud dashboards 📊</li><li>Send alerts for failed invoices 🚨</li><li>Generate weekly audit trails 🧾</li></ul><p>You can also visualize errors by plotting bounding boxes on the actual image!</p><h3>🔄 Built for GCP: Ready for Vertex AI Agents</h3><p>As highlighted in <a href="https://cloud.google.com/blog/products/ai-machine-learning/build-and-manage-multi-system-agents-with-vertex-ai">Google’s official blog post</a>, <strong>Vertex AI Agents</strong> now support:</p><ul><li>Declarative multi-agent workflows</li><li>Tool integration (like search, DBs)</li><li>Autonomous decision-making</li></ul><p>This architecture is already aligned with GCP’s future of <strong>AI orchestration</strong>.</p><h3>✅ TL;DR — Why Use Agentic AI for Tampering Checks?</h3><p>✅ <strong>Modular</strong>: Add agents like building blocks<br>🔍 <strong>Transparent</strong>: Each result is explainable<br>📈 <strong>Scalable</strong>: Run on 1 or 1,000 invoices<br>🛡️ <strong>Fraud-proof</strong>: Detects even clever tampering<br>☁️ <strong>Built on GCP</strong>: Seamless with Vertex AI, Vision API, and BigQuery</p><h3>🚀 What’s Next?</h3><p>Ready to stop invoice fraud in its tracks? Whether you’re in fintech, procurement, or auditing — this system gives you:</p><ul><li>Human-level scrutiny</li><li>Lightning-fast automation</li><li>Full transparency</li></ul><p>Want a demo, codebase, or deployment guide? Drop a comment — let’s build together</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d8431befaba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Handling 100 GB IoT Data in AWS Glue with Advanced Spark Tuning]]></title>
            <link>https://medium.com/@amar.kumar2013/handling-100-gb-iot-data-in-aws-glue-with-advanced-spark-tuning-5ce8ed4090b0?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ce8ed4090b0</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[performance-tuning]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Fri, 11 Jul 2025 10:45:31 GMT</pubDate>
            <atom:updated>2025-07-11T10:47:02.298Z</atom:updated>
            <content:encoded><![CDATA[<h3>🔍 Introduction</h3><p>Processing massive IoT datasets is a growing challenge in cloud-native data engineering. In this guide, we’ll walk through how to ingest, transform, and optimize <strong>100 GB of nested JSON</strong> using <strong>AWS Glue (Spark 3.x)</strong> — complete with Spark tuning tips, real-world examples, and performance troubleshooting.</p><h3>📦 Use Case: Smart City IoT Data</h3><p>You work at a smart city analytics platform. Every hour, thousands of sensors upload JSON logs to <strong>Amazon S3</strong>. You’ve accumulated <strong>100 GB</strong> of historical JSON and need to:</p><ul><li>🧹 Parse and flatten deeply nested structures</li><li>🔍 Filter data for a time window</li><li>🧠 Partition output by event_date and device_type</li><li>📁 Write <strong>partitioned Parquet</strong> for Athena and Redshift</li></ul><h3>⚠️ Real-World Challenges</h3><p>Key bottlenecks encountered:</p><ul><li>📂 Excessive S3 file count → slow listing &amp; task launch</li><li>🧠 Memory pressure parsing large nested JSON</li><li>🔁 Shuffle skew → straggler tasks</li><li>🧾 Tiny Parquet files → slow downstream query</li></ul><h3>🏗️ Solution Architecture</h3><pre>S3 (raw JSON) → AWS Glue Job (Spark Engine) → S3 (Parquet Partitioned) → Athena / Redshift Spectrum</pre><ol><li><strong>Ingest</strong>: Load nested JSON from s3://iot/raw/</li><li><strong>Transform</strong>: Flatten, filter, repartition</li><li><strong>Write</strong>: Save as partitioned Parquet in s3://iot/processed/</li></ol><p>⚙️ Spark Tuning Strategy</p><p>💾 Memory &amp; Executor Configuration</p><pre>--conf spark.executor.memory=10g<br>--conf spark.sql.shuffle.partitions=400<br>--conf spark.serializer=org.apache.spark.serializer.KryoSerializer</pre><p>✅ Helps parse nested records and scale parallelism. Kryo reduces serialization overhead.</p><p>♻️ Adaptive Query Execution (AQE)</p><pre>--conf spark.sql.adaptive.enabled=true<br>--conf spark.sql.adaptive.coalescePartitions.enabled=true<br>--conf spark.sql.adaptive.skewJoin.enabled=true</pre><p>✅ Dynamically adjusts joins, partitions, and handles skew. Great for real-world data irregularities.</p><p>📁 File Size and Partition Control</p><pre>df.repartition(400).write.partitionBy(&quot;event_date&quot;, &quot;device_type&quot;)</pre><p><strong>✅ Prevents small file explosion. Ensures consistent 256MB file sizes.</strong></p><p>🧑‍💻 Sample ETL Code Explained</p><pre># Step 1: Read &amp; Filter JSON<br>df = glueContext.create_dynamic_frame_from_options(<br>    &quot;s3&quot;, {&quot;paths&quot;: [&quot;s3://iot/raw/&quot;], &quot;recurse&quot;: True}, format=&quot;json&quot;<br>).filter(f=lambda x: x[&quot;event&quot;][&quot;timestamp&quot;] &gt;= &quot;2025-01-01&quot;)</pre><pre># Step 2: Flatten nested JSON<br>df_flat = df.toDF().selectExpr(<br>    &quot;device.id as device_id&quot;,<br>    &quot;event.timestamp as timestamp&quot;,<br>    &quot;event.metrics.temperature as temperature&quot;,<br>    &quot;event.location.lat as lat&quot;,<br>    &quot;event.device_type as device_type&quot;,<br>    &quot;event.event_date as event_date&quot;<br>)</pre><pre># Step 3: Write to partitioned Parquet<br>df_flat.repartition(400).write.partitionBy(&quot;event_date&quot;, &quot;device_type&quot;).mode(&quot;overwrite&quot;).parquet(&quot;s3://iot/processed/&quot;)</pre><h3>🧠 Best Practices Used:</h3><ul><li>Filter early to reduce shuffle volume</li><li>Flatten for OLAP query friendliness</li><li>Repartition for parallel output and file size control</li></ul><p>🔧 Troubleshooting Playbook</p><p>🛠️ Common ETL Problems and Fixes</p><h3>|⚠️ What Went Wrong? |🔎 What It Looks Like |💥 Why It Happens |✅ What You Should Do</h3><p>|🚫 Job stuck during startup<br>|Long init time, no DAG built<br>|Too many small files; slow metadata ops<br>|✅ Consolidate files in S3<br>✅ Enable ignoreCorruptFiles=true<br>✅ Reduce nested folder recursion</p><p>|💣 Out of Memory (OOM)<br>|Heap or GC errors in logs<br>|Large JSON rows or massive shuffles<br>|✅ Use .filter() early<br>✅ Convert to DataFrame<br>✅ Tune executor memory and partitions</p><p>|🐢 Slow shuffle &amp; runtime<br>|Long tail tasks; skew in Spark UI<br>|Skewed keys or unnecessary wide joins<br>|✅ Enable AQE + skew join handling<br>✅ Push filters early<br>✅ Allow dynamic coalescing</p><p>|📦 Too many tiny files<br>|8K+ small Parquet files in output<br>|No control on partition count<br>|✅ Use .repartition(400) before write<br>✅ Partition by selective high-cardinality keys</p><h3>✅ Troubleshooting Matrix: Common Practical Issues</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/0*437BiszsKj7RcrYB.png" /><figcaption>Troubleshooting Matrix</figcaption></figure><p>✅ Measured Performance Gains</p><p>Metric Before After Total Runtime 40 minutes 8 minutes Output Files ~8,000 ~400 Join Skew High Eliminated Athena Query Speed Poor Instantaneous</p><p>🏁 Final Takeaways</p><p>This tuning pipeline transformed a complex workload into a scalable solution:</p><ul><li>📚 Master Spark concepts like partitioning, joins, and execution plans</li><li>🧪 Leverage AQE to handle runtime skew and join selection</li><li>🎯 Write optimized Parquet output for analytics consumption</li></ul><p>📊 End Result: Reliable, fast, cloud-native ETL at scale — with lower cost and better observability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ce8ed4090b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Data Lifecycle on Google Cloud: Choosing the Right Storage Solution]]></title>
            <link>https://medium.com/@amar.kumar2013/navigating-the-data-lifecycle-on-google-cloud-choosing-the-right-storage-solution-41ab045d585b?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/41ab045d585b</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Fri, 14 Feb 2025 11:50:17 GMT</pubDate>
            <atom:updated>2025-02-14T11:57:18.537Z</atom:updated>
            <content:encoded><![CDATA[<h3>Navigating the Data Lifecycle on Google Cloud: Choosing the Right Solution</h3><p>In today’s data-driven world, businesses are constantly generating and consuming information. Effectively managing this data is crucial for gaining valuable insights and driving informed decisions. A key element of data management is understanding the data lifecycle and choosing the right storage solution for each stage. This blog will explore the four stages of the data lifecycle — Ingest, Store, Process &amp; Analyze, and Explore &amp; Visualize — and guide you on selecting the optimal Google Cloud storage solutions for each.</p><h3><strong>The Four Stages of the Data Lifecycle:</strong></h3><h4>1. Ingest: Acquiring Data and Bringing it to the Cloud</h4><p>This is the first and arguably most crucial step. How you bring data into the cloud significantly impacts subsequent stages. The right ingestion method ensures data arrives efficiently, reliably, and in a format suitable for downstream processing.</p><p>We can broadly categorize data ingestion into three modes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwQB1GYHLxRa6NYFlM-GjA.jpeg" /><figcaption><strong>Data Transfer Services on Google Cloud</strong></figcaption></figure><p><strong>a. Application Data:</strong> This includes data generated by applications, mobile apps, and user interactions. Think of user profiles, product reviews, or in-app activity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*e-URoBDi0scQdanJprUGUw.png" /><figcaption><strong>Application Data Categorization</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*-bTt7OOVVnJaDoSDixygWQ.png" /><figcaption><strong>Ingestion Methods for Application Data on Google Cloud</strong></figcaption></figure><p><strong>b. Streaming Data: </strong>This refers to continuous streams of data, often sent in small messages from various sources. Examples include real-time sensor data, user activity feeds, or stock market tickers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*AAQsIWMzNGZi9P2SGd1onA.png" /><figcaption><strong>Streaming Data Categorization</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*FwTwsf2FiIjCzsIBh-qVGw.png" /><figcaption><strong>Ingestion Method for Streaming Data on Google Cloud</strong></figcaption></figure><p><strong>c. Batch Data:</strong> This involves ingesting data in bulk, typically in files. This is often the case with historical data, data exports, or large datasets from external sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*dLazgI0nzLuR3P6MWAtuNA.png" /><figcaption><strong>Batch Data Categorization</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*C1fC1GhxmtWDAY15m2ZPhg.png" /><figcaption><strong>Ingestion Method for Batch data on Google Cloud</strong></figcaption></figure><h4>2. Store: Persisting Data to a Storage System</h4><p>Once the data is ingested, it needs to be stored securely and efficiently. Choosing the right storage system is critical for performance, cost, and scalability.</p><p><strong>Several factors influence this decision:</strong></p><ul><li><strong>Data Access Pattern</strong>: How you access the data — by individual records (OLTP) or by aggregating columns across many records (OLAP).</li><li><strong>Access Control Requirements</strong>: How granularly access controls need to be implemented — at the schema, database, or finer-grained level.</li><li><strong>Data Retention Period:</strong> How long you need to store the data.</li></ul><p><strong>Four Key Metrics for Data Access:</strong></p><ul><li><strong>Read Volume</strong>: How much data is retrieved in each read operation? (e.g., a few rows vs. millions of rows).</li><li><strong>Write Volume</strong>: How much data is written in each insert operation? (e.g., single row vs. batch of records).</li><li><strong>Write Frequency</strong>: How often is data written? (e.g., every second vs. once a day).</li><li><strong>Read Frequency</strong>: How often is data read? (e.g., constantly vs. rarely).</li></ul><p><strong>Storage Solutions and Use Cases:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*QoEE_RzKWpvn39TI9tzzCA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IFbFHMV-MJRXkjU_trU6-g.png" /><figcaption><strong>Google Storage Services Comparison</strong></figcaption></figure><h4>3. Process and Analyze: Transforming Data</h4><p>This stage involves transforming raw data into a usable format for analysis. This often includes cleaning, filtering, aggregating, and enriching the data.</p><ul><li><strong>Google Cloud Dataflow</strong>: A fully managed, serverless data processing service for batch and stream data processing. Perfect for transforming and enriching data at scale.</li><li><strong>Google Cloud Dataproc</strong>: A fully managed Hadoop and Spark service for big data processing. Ideal for complex data transformations and analysis.</li><li><strong>BigQuery</strong>: A fully managed, serverless data warehouse for large-scale data analysis. Perfect for querying and analyzing large datasets.</li></ul><h4>4. Explore and Visualize: Deriving Insights</h4><p>The final stage is about exploring the processed data and visualizing the results to gain actionable insights.</p><ul><li><strong>Looker</strong>: A modern business intelligence and data visualization platform. It allows you to explore your data, create dashboards, and share insights with others.</li><li><strong>Google Data Studio:</strong> A free data visualization tool that allows you to create interactive dashboards and reports.</li><li><strong>Custom Applications:</strong> Build your own custom applications to explore and visualize data using programming languages like Python and tools like matplotlib and seaborn.</li></ul><h4>Conclusion:</h4><p>Choosing the right storage solution for each stage of the data lifecycle on Google Cloud is crucial for optimizing performance, cost, and scalability. By carefully considering the data characteristics, access patterns, and security requirements, you can build a robust and efficient data pipeline that drives valuable insights and empowers your business to make data-driven decisions. Remember to continuously evaluate your storage strategy as your data needs evolve to ensure you’re leveraging the optimal solutions for your specific requirement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41ab045d585b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Big Data Extraction Pipelines: Harnessing Paddle OCR & Multi-GPU for Text Extraction in…]]></title>
            <link>https://medium.com/@amar.kumar2013/mastering-big-data-extraction-pipelines-harnessing-paddle-ocr-multi-gpu-for-text-extraction-in-8d5f890e9a34?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d5f890e9a34</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 06:17:26 GMT</pubDate>
            <atom:updated>2025-01-27T11:15:42.525Z</atom:updated>
            <content:encoded><![CDATA[<h3>Big Data Extraction Pipelines: Harnessing Paddle OCR &amp; Multi-GPU for Text Extraction in Production</h3><h4>Supercharge Your Data Pipelines: High-Speed OCR with Paddle and Multi-GPU</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0eJw8U_YFFgNEBJ6Qdp5Q.png" /></figure><h3>Introduction</h3><p>In today’s data-driven world, a vast amount of valuable information resides within unstructured documents like PDFs. Extracting this data efficiently and accurately can be a game-changer for businesses and researchers alike. Enter PaddleOCR, a powerful open-source OCR (Optical Character Recognition) engine that’s been making waves in the field. In this blog post, we’ll delve into the technical aspects of using PaddleOCR for data extraction from PDFs, exploring its capabilities, how to integrate it into your workflow, and some important considerations</p><h3>Use -case</h3><p>“But the sheer volume and unstructured nature of our PDF data were making this incredibly difficult. Trying to manually extract data from <strong>10,000 PDFs</strong>? Forget about it! It’s not just the tediousness; it’s the inconsistencies, errors, and the sheer time commitment that made the task next to impossible.</p><p>We needed a solution, and not just any solution. We needed one that was:</p><ul><li><strong>Scalable</strong>: Able to handle a daily load of 10,000 PDFs and scale further as we grow.</li><li><strong>Accurate</strong>: Extract data reliably with minimal errors.</li><li><strong>Efficient</strong>: Process data quickly without breaking the bank.</li><li><strong>Maintainable</strong>: Easy to update and troubleshoot as things change.</li></ul><h4><strong>Solution</strong></h4><p>PDF Conversion &amp; Preprocessing: Preparing for the OCR Assault:</p><p>Challenge: Dealing with both digital and scanned PDFs, often with noise, skew, or poor resolution. Accurately extracting text from a wide variety of document types.</p><p>Solution:</p><p>1. Using PaddleOCR for the core text extraction, specifically the latest version which has improved detection and recognition accuracy.</p><p>2. Utilizing GPU resources to achieve the best possible performance and to reduce processing time. Also test and compare between various PaddleOCR models and select the optimal model that gives the best results in terms of speed and accuracy.</p><p>Post-Processing and Structured Extraction: Making Sense of the Text</p><ul><li>Challenge: Taking the raw OCR output and converting it to structured data (JSON).</li><li>Solution: Implementing a combination of layout analysis, regular expressions, and GenAI models, specifically custom fine-tuned Large Language Models (LLMs), depending on the complexity of the extraction tasks. Parse the OCR output, identify relevant entities (names, dates, addresses, figures, tables) and their relations, and then structure the data. If the PDF contains tables, we extract it using table detection and extraction techniques. GenAI plays a key role here to not only extract but also do reasoning.</li></ul><h3>Implementation(Production Setup)</h3><p><strong>Setting up the NVIDIA Driver to utilize GPU on RedHat Server</strong></p><p><strong>Driver Setup</strong></p><pre>-- 1.Checking the details of GPU -&gt; Follow the Commands Below to check for the configuration<br>a.sudo lshw -numeric -C display<br>b.lspci | grep -i nvidia<br>c.uname -m &amp;&amp; cat /etc/*release -- Check the OS Release<br>d.uname -r // Check the Kernel <br><br>-- 2. Check For Nvidia Details on the Server <br>   a. yum search nvidia <br>   b. yum search &#39;nvidia-driver-&#39; | grep &#39;^nvidia-driver-[[:digit:]]*&#39; | sort -k 3 -t &#39;-&#39;    <br>   c. yum search &#39;nvidia-driver-*&#39; | grep &#39;^nvidia-driver-[0-9]*&#39;yum search nvidia<br></pre><p>Install Cuda Toolkit : <a href="https://developer.nvidia.com/cuda-downloads?target_os=Linux&amp;target_arch=x86_64&amp;Distribution=RHEL&amp;target_version=9&amp;target_type=rpm_local">CUDA Toolkit 12.5 Update 1 Downloads | NVIDIA Developer</a></p><figure><img alt="Select the Categories based on your Server Configuration" src="https://cdn-images-1.medium.com/max/1024/1*bfbudS3THw9CLn-2ySz87g.png" /><figcaption>Select based on the Server Configuration</figcaption></figure><pre>a. sudo yum install wget<br>b. sudo rpm -i cuda-repo-rhel9-12-5-local-12.5.1_555.42.06-1.x86_64.rpm<br>c. sudo dnf clean all<br>d. sudo dnf -y install cuda-toolkit-12-5</pre><h4><strong>Install the latest Driver from</strong> : <a href="https://www.nvidia.com/download/index.aspx?lang=en-us">Official Drivers | NVIDIA</a></h4><ul><li>** <strong><em>Check for the compatibility of your cuda-toolkit and Nvidia Driver</em></strong>**</li><li><strong><em>Get the link</em></strong></li></ul><pre>a.wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.100/NVIDIA-Linux-x86_64-550.100.run<br>b.sudo dnf install kernel-devel-$(uname -r) -- Install Kernel <br>c.sudo sh NVIDIA-Linux-x86_64-550.100.run -- Run the File to Install the driver. <br>d.nvidia-smi -- Check the Driver is installed or not.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*dchnSM2nK9z8GsqDAWNopA.png" /><figcaption>Verification driver is Installed Properly</figcaption></figure><h4><strong>Installing NCCL — For Multi-GPU Use case</strong></h4><p><a href="https://developer.nvidia.com/nccl/nccl-download"><em>NVIDIA Collective Communications Library (NCCL) | NVIDIA Developer</em></a></p><p>Log in and get the link to download it</p><pre>--- Network Installer <br>a.sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo<br> <br>b.sudo yum install libnccl-2.23.4-1+cuda12.4 libnccl-devel-2.23.4-1+cuda12.4 libnccl-static-2.23.4-1+cuda12.4<br><br>-- For Local Installer (If above is not working)<br>a.sudo wget https://developer.download.nvidia.com/compute/machine-learning/nccl/secure/2.22.3/rhel9/aarch64sbsa/nccl-local-repo-rhel9-2.22.3-cuda12.5-1.0-1.aarch64.rpm?YVFzHn7ygiBdlS_aaqlelMa_G7d7hGsjS9vnlG0KOq4Hwq-Yd727yeJaVVBm0NkRyOSno9Be-KO_cJwy7PZJEnS-DQG8l0FAaT5lhiRaYoRaRJWKQLGNQy4cx8meAR9TSxUzx5dErIhrMYTPhRSy3N6VBpOv8hAxyfgH3QfPQgM8-oNkVMlmva9Yj2Wd-2b_a_-TyWB2Z6VWEabpDSwYzH5kUAR8Ap79oS5_-Oekr44q0cCU&amp;t=eyJscyI6ImJzZW8iLCJsc2QiOiJodHRwczovL3d3dy5iaW5nLmNvbS8ifQ==<br>b.sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/sbsa/cuda-rhel9.repo<br>c.sudo yum install libnccl-2.22.3-1+cuda12.5 libnccl-devel-2.22.3-1+cuda12.5 libnccl-static-2.22.3-1+cuda12.5</pre><h4><strong>Installing paddlepaddle-gpu for utilizing gpu with PaddleOcr</strong></h4><p><strong>Follow the link for more information</strong>: <a href="https://www.paddlepaddle.org.cn/"><em>PaddlePaddle</em></a></p><p><a href="https://www.paddlepaddle.org.cn/whl/linux/cudnnin/stable.html">“<strong>Download the CUDA-compatible PaddlePaddle </strong></a><a href="https://www.paddlepaddle.org.cn/whl/linux/cudnnin/stable.html"><strong>.whl file here</strong></a>”</p><pre>1.conda install -c conda-forge libstdcxx-ng  -- The GNU C++ Runtime Library required to run deeplearning libraries<br>-- Install Paddlepaddle-gpu <br>2.python3 -m pip install paddlepaddle-gpu==2.6.1.post120 -f https://www.paddlepaddle.org.cn/whl/linux/cudnnin/stable.html<br>3.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/miniconda3/lib   ### Resolving /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21&#39; not found error<br></pre><pre>#### To check the if paddlepaddle-gpu is installed<br>import paddle<br>paddle.utils.run_check()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aZ_RzTBC57Kts8l0YHVBkQ.png" /><figcaption>Verification both the GPU is working with paddleocr.</figcaption></figure><h4>*** Congratulation you are done with all the requirement setup ***</h4><h4>Extracting text using PaddleOCR from PDF</h4><pre>from paddleocr import PaddleOCR<br>import locale<br>import os<br>import json<br>import ast<br>import json<br>import base64<br>from pdf2image import convert_from_path<br><br>ocr = PaddleOCR(use_angle_cls=True, lang=&#39;en&#39;,use_space_char=True,show_log=False,enable_mkldnn=True,use_gpu=True)<br><br>def Information_Extraction(img_path):<br>    &quot;&quot;&quot; Extracting the Infotmation from the image by implementing OCR and then passing it to model<br>        to generate output tensor<br><br>    Args:<br>        img_path (string): directory where the image is present<br><br>    Returns:<br>        tensor: model ouput tensor<br>    &quot;&quot;&quot;<br>    boxes=[]<br><br>    txts = []<br>    scores = []<br>    Ocr_value=[]<br>    result = ocr.ocr(img_path, cls=False)<br>    if result !=[None]: <br>        <br>        for entry in result[0]:<br>            confidence_score=entry[1][1]<br>            bounding_box=entry[0]<br>            text=entry[1][0]<br><br>            entry_dict={<br>                &#39;confidence_score&#39; : confidence_score,<br>                &#39;bounding_box&#39;: bounding_box,<br>                &#39;text&#39;: text<br>            }<br>            Ocr_value.append(entry_dict)<br><br>    ocr_string = &quot;&quot;<br>    # Extract the text from the OCR result and concatenate it to ocr_string<br>    if result !=[None]:<br>        for entry in result[0]:<br>            text=str(entry[1][0])<br>            ocr_string = ocr_string + text + &quot; &quot;<br>    else:<br>        ocr_string=&#39;No Data&#39;<br>    return Ocr_value,ocr_string<br><br><br>def image_to_base64(image_path):<br>    with open(image_path, &quot;rb&quot;) as image_file:<br>        base64_encoded = base64.b64encode(image_file.read()).decode(&quot;utf-8&quot;)<br>    return base64_encoded<br><br><br>def Create_JSON(pdf_path,output_directory):<br>    ocr_dict={}<br>    <br>    pages = convert_from_path(pdf_path,first_page=1,last_page=10)<br><br>    # Save each page as a JPEG image<br>    for count, page in enumerate(pages):<br><br>        output_path = os.path.join(output_directory, &#39;TmpImages&#39;,os.path.basename(pdf_path).split(&#39;.&#39;)[0])<br>        os.makedirs(output_path, exist_ok=True)<br>        output_path_image=os.path.join(output_path,str(count))<br>        page.save(f&quot;{output_path_image}.jpg&quot;, &quot;JPEG&quot;)<br>        ocr_dict[count]={}<br>        ocr_dict_noimage[count]={}<br>        a,b=Information_Extraction(f&#39;{output_path_image}.jpg&#39;)<br>        ocr_dict[count][&#39;TextInfo&#39;]=a<br>        ocr_dict[count][&#39;SummarizedText&#39;]=b<br><br><br>        ocr_dict[count][&#39;InputImage&#39;]=image_to_base64(f&quot;{output_path_image}.jpg&quot;)<br><br>    return ocr_dict</pre><p>The returned output is a Nested dictionary of format(Sample Output)</p><pre>{<br>    0: { # Page 1<br>        &#39;TextInfo&#39;: [<br>            {<br>                &#39;confidence_score&#39;: 0.987,<br>                &#39;bounding_box&#39;: [[100, 50], [300, 50], [300, 80], [100, 80]],<br>                &#39;text&#39;: &#39;Invoice&#39;<br>            },<br>            {<br>               &#39;confidence_score&#39;: 0.956,<br>                &#39;bounding_box&#39;: [[50, 120], [150, 120], [150, 140], [50, 140]],<br>                &#39;text&#39;: &#39;Date:&#39;<br>            },<br>             {<br>               &#39;confidence_score&#39;: 0.898,<br>                &#39;bounding_box&#39;: [[200, 120], [350, 120], [350, 140], [200, 140]],<br>                &#39;text&#39;: &#39;2023-10-27&#39;<br>            }<br>        ],<br>        &#39;SummarizedText&#39;: &#39;Invoice Date: 2023-10-27&#39;,<br>         &#39;InputImage&#39;:&#39;base64encodedstring..&#39;<br>    },<br>    1: { #Page 2<br>        &#39;TextInfo&#39;: [<br>             {<br>                &#39;confidence_score&#39;: 0.977,<br>                &#39;bounding_box&#39;: [[50, 50], [300, 50], [300, 80], [50, 80]],<br>                &#39;text&#39;: &#39;Customer:&#39;<br>            },<br>              {<br>                &#39;confidence_score&#39;: 0.944,<br>                &#39;bounding_box&#39;: [[200, 50], [500, 50], [500, 80], [200, 80]],<br>                &#39;text&#39;: &#39;John Doe&#39;<br>            },<br>             {<br>               &#39;confidence_score&#39;: 0.967,<br>                &#39;bounding_box&#39;: [[50, 120], [150, 120], [150, 140], [50, 140]],<br>                &#39;text&#39;: &#39;Address:&#39;<br>            },<br>             {<br>               &#39;confidence_score&#39;: 0.988,<br>                &#39;bounding_box&#39;: [[200, 120], [400, 120], [400, 140], [200, 140]],<br>                &#39;text&#39;: &#39;123 Main St&#39;<br>            },<br>             {<br>               &#39;confidence_score&#39;: 0.899,<br>                &#39;bounding_box&#39;: [[200, 150], [400, 150], [400, 170], [200, 170]],<br>                &#39;text&#39;: &#39;New York, NY&#39;<br>            }<br>        ],<br>        &#39;SummarizedText&#39;: &#39;Customer: John Doe Address: 123 Main St New York, NY&#39;,<br>       &#39;InputImage&#39;:&#39;base64encodedstring...&#39;<br>    }</pre><h4><strong>Deployment</strong></h4><p><strong><em>To achieve the best performance, our text extraction pipeline is designed to run on multiple cores while fully utilizing all available GPUs. We use shell scripting to batch and parallelize the tasks, combined with </em></strong><strong><em>nohup for continuous execution. This approach allows us to extract text from your documents with maximum speed and efficiency</em></strong></p><pre>-- To run this Process on both the GPU <br>-- CUDA_VISIBLE_DEVICES=0 &gt; This enables to use gpu with the python file<br>CUDA_VISIBLE_DEVICES=0 paddleocr.py<br>CUDA_VISIBLE_DEVICES=1 paddleocr.py<br><br></pre><h4>Conclusion</h4><p>We turned a mountain of PDFs into actionable insights. By combining the power of PaddleOCR with multi-GPU acceleration and intelligent pre/post-processing, we built a text extraction pipeline that’s not just scalable and accurate, but <em>business-ready</em>. And, with the help of GenAI, we’re extracting the critical details to fuel smarter decisions. This is how big data becomes big value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d5f890e9a34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Big Data ETL Pipelines: Harnessing Apache Airflow for orchestration and AWS Glue Jobs]]></title>
            <link>https://medium.com/@amar.kumar2013/mastering-big-data-etl-pipelines-harnessing-apache-airflow-for-orchestration-and-aws-glue-jobs-c27140886e6a?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c27140886e6a</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Wed, 11 Oct 2023 09:08:57 GMT</pubDate>
            <atom:updated>2023-10-11T09:14:56.368Z</atom:updated>
            <content:encoded><![CDATA[<p>· <a href="#03f4">Introduction</a><br>· <a href="#1841">Use -case:</a><br>· <a href="#5fb8">Components of Airflow:</a><br> ∘ <a href="#7681">Directed Acyclic Graphs (DAGs): </a><br> ∘<a href="#1f8c"> Operators: </a><br> ∘<a href="#2c3e"> Sensors: </a><br> ∘ <a href="#92b2">Connections:</a><br> ∘ <a href="#d8a2">Monitoring and Alerting: </a><br>· <a href="#1981">Conclusion</a></p><h3>Introduction</h3><p>In today’s data-driven world, efficient ETL (Extract, Transform, Load) pipelines are the backbone of data processing and analytics. Orchestrating these pipelines can be complex, especially when dealing with a plethora of tasks, dependencies, and automation. This is where Apache Airflow comes to the rescue. In this blog, we’ll not only dive into the importance of Apache Airflow but also provide a step-by-step breakdown of a practical code example.</p><h3>Use -case:</h3><p>“Let’s say I work for an organization called DataWorks Inc. Our company is a hub of data-driven decision-making, managing a diverse range of data sources. We’re dealing with everything from customer interactions and sales data to monitoring online user behavior. However, as our data volumes continue to surge, orchestrating our ETL (Extract, Transform, Load) pipelines has become a rather daunting challenge. But that’s not all. Here’s where it gets interesting — our ETL jobs are not isolated; they have intricate dependencies.</p><p>In fact, some of our critical jobs, namely Job2, Job4, Job6, and Job8, aren’t just required by our team but are essential components for other teams too. They are derived from the previous Jobs. This intricate web of dependencies was giving us sleepless nights.</p><p>That’s when we decided to take control of our ETL pipeline, streamline the process, and ensure that all those dependencies were met. So, what was our solution, you might ask? We decided to implement Apache Airflow, a powerful workflow automation tool, in collaboration with AWS Glue Jobs, a managed ETL service. Here’s how it all came together. Let&#39;s implement the pipeline along with explaining the intricacies of airflow. Find the whole code for this blog in this <a href="https://github.com/amar5075kumar/ETL-Pipeline--Orchestration-with-Apache-Airflow"><strong>GitHub Repository</strong></a></p><h3>Components of Airflow:</h3><h4><strong>Directed Acyclic Graphs (DAGs)</strong>:</h4><p>Apache Airflow uses Directed Acyclic Graphs (DAGs) to represent workflows. These DAGs are a collection of tasks, and the dependencies between them are defined within the DAG. Tasks can be Python functions, scripts, or even external processes.</p><pre>default_args = {<br>    &#39;owner&#39;: &#39;Amar&#39;,<br>    &#39;start_date&#39;: datetime(2023, 9, 19),        <br>    &#39;depends_on_past&#39;: False,<br>    &#39;email&#39;: [&#39;*******************&#39;],<br>    &#39;email_on_failure&#39;: True,<br>    &#39;retries&#39;: 1,<br>}<br><br>dag = DAG(<br>    dag_id=&#39;glue_pipeline&#39;,<br>    description=&#39;Run Glue Pipeline&#39;,<br>    start_date=datetime(2023, 9, 15),<br>    schedule_interval=None,<br>    default_args=default_args,<br>    catchup=False,<br>)<br><br>start_task = EmptyOperator(task_id=&#39;start&#39;, dag=dag)<br># Use a EmptyOperator as the ending point<br>end_task = EmptyOperator(task_id=&#39;end&#39;, dag=dag)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ze-bvK2y0gVmSgn8Y6tfLg.png" /><figcaption><strong>DAG for the current Pipeline</strong></figcaption></figure><h4><strong>Operators</strong>:</h4><p>Airflow offers a wide range of operators to define and execute tasks. For instance, the PythonOperator allows you to execute Python functions, while AthenaOperator and GlueJobOperator enable integration with AWS services. Below mentioned is Glue Job task created using Glue Job operator</p><pre>glue_job_task = GlueJobOperator(<br>    task_id=f&quot;submit_glue_job_{job_name}&quot;,<br>    job_name=job_name,<br>    script_location=f&quot;s3://{glue_s3_folder}/{job_name}.py&quot;,<br>    s3_bucket=&#39;**********&#39;, ## Specify the s3_bucket name <br>    region_name=&#39;us-east-1&#39;,<br>    aws_conn_id=None,<br>    iam_role_name=role_name,<br>    wait_for_completion=True,<br>    create_job_kwargs={&quot;GlueVersion&quot;: &quot;3.0&quot;, &quot;NumberOfWorkers&quot;: 2, &quot;WorkerType&quot;: &quot;G.1X&quot;},<br>    dag=dag,<br>)</pre><h4><strong>Sensors</strong>:</h4><p>Sensors are special operators that wait for a specific condition to be met before proceeding. In our example, we’ll use a TimeDeltaSensor to introduce a time delay between tasks, which is valuable for managing task sequences. Below is creating a wait sensor using time delta sensor. The total number of sensors created for the following pipeline is half of the total jobs.</p><pre>wait_time = timedelta(minutes=1)<br>wait_sensor_d={}<br>for i in range(len(glue_job_name)//2):<br>    wait_sensor = TimeDeltaSensor(<br>    task_id=f&#39;wait_sensor_{i}&#39;,<br>    delta=wait_time,<br>    dag=dag,<br>    )<br>    wait_sensor_d[i]=wait_sensor</pre><h4><strong>Connections</strong>:</h4><p>Airflow lets you define external connections, such as AWS credentials, as named connections. These connections can be securely stored in Airflow’s metadata database, ensuring the security of your AWS credentials. In the following pipeline we create a AWS connection in Admin tab. Enter the following details of your AWS account where you are running your pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCo5osigB3A2JPGSzb7BfA.png" /><figcaption>AWS connection in Airflow</figcaption></figure><h4><strong>Monitoring and Alerting</strong>:</h4><p>Airflow provides a comprehensive monitoring and alerting system. You can keep a close eye on the progress of tasks, receive alerts in case of failures, and troubleshoot issues promptly. This proactive approach to data pipeline management saves time and resources. Here we have created custom generated mail which can be sent to particular mail when a particular job fails. <br>Note: You need to define SMTP connections for sending the mail in Airflow.cfg file.</p><p>Add your customized html_content ,subject and body. Creating different job tasks for each glue job and using the send_failure_email function to send the alert in case of failure.</p><pre>def send_failure_email(context):<br>    &quot;&quot;&quot;Sends an email with detailed log information when a task fails.&quot;&quot;&quot;<br>    error = str(context[&#39;exception&#39;])<br>    task_instance = context[&#39;task_instance&#39;]<br>    task_id = task_instance.task_id<br>    dag_id = task_instance.dag_id<br>    execution_date = task_instance.execution_date.strftime(&#39;%Y-%m-%d %H:%M:%S&#39;)<br>    exception_html = str(context[&#39;exception&#39;]).replace(&quot;\n&quot;, &quot;&lt;br&gt;&quot;)<br>    log_file = os.path.join(context[&#39;ti&#39;].log_directory, context[&#39;task_instance&#39;].log_filename)<br>    log_content = task_instance.log.export_log()<br>    <br>    html_content = f&quot;&quot;&quot; &quot;&quot;&quot;<br>    subject = &quot;&quot;&quot; &quot;&quot;&quot;<br>    body=&quot;&quot;&quot; &quot;&quot;&quot;<br>        <br>    <br>    send_email(<br>        to=default_args[&#39;email&#39;],<br>        subject=subject,<br>        html_content=html_content,<br>        body=body,<br>    )<br>### Creating Glue task for each job and on failure callback using send_failure_email function.<br>for job_name in glue_job_name:<br>    task_id=f&quot;submit_glue_job_{job_name}&quot;<br>    if task_id not in glue_job_tasks:<br>        glue_job_task = GlueJobOperator(<br>            task_id=f&quot;submit_glue_job_{job_name}&quot;,<br>            job_name=job_name,<br>            script_location=f&quot;s3://{glue_s3_folder}/{job_name}.py&quot;,<br>            s3_bucket=&#39;****************&#39;,<br>            region_name=&#39;us-east-1&#39;,<br>            aws_conn_id=None,<br>            iam_role_name=role_name,<br>            wait_for_completion=True,<br>            on_failure_callback=send_failure_email,<br>            create_job_kwargs={&quot;GlueVersion&quot;: &quot;3.0&quot;, &quot;NumberOfWorkers&quot;: 2, &quot;WorkerType&quot;: &quot;G.1X&quot;},<br>            dag=dag,<br>        )<br>        glue_job_tasks[job_name] = glue_job_task</pre><h3>Conclusion</h3><p>In the world of data engineering, orchestrating ETL pipelines efficiently is vital. Apache Airflow, with its automation capabilities, dependency management, and extensibility, is a game-changer. When combined with AWS Glue Jobs, it becomes a potent tool for managing data pipelines. This practical example showcases how these technologies can streamline your ETL processes, making them more reliable and less prone to errors.</p><p>By using Apache Airflow and AWS Glue Jobs, you can focus on deriving insights from your data rather than spending excessive time managing your data pipelines. It’s a win-win for both data engineers and businesses that rely on data-driven decision-making. Happy orchestrating!</p><p>Find the whole code for this blog in this <a href="https://github.com/amar5075kumar/ETL-Pipeline--Orchestration-with-Apache-Airflow"><strong>GitHub Repository</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c27140886e6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Delta Lake with AWS EMR : A CDC Use Case]]></title>
            <link>https://medium.com/@amar.kumar2013/demystifying-delta-lake-with-aws-emr-b842084cdb9b?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b842084cdb9b</guid>
            <category><![CDATA[data-warehousing]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Fri, 15 Sep 2023 10:27:56 GMT</pubDate>
            <atom:updated>2023-09-16T06:50:38.810Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving landscape of data storage and processing, three distinct solutions have emerged as game-changers: Data Lakes, Data Lakehouses, and Delta Lakes. While each of these technologies holds its own promise, the choice between them can significantly impact how you handle your data. To shed light on the matter, we will explore these data storage paradigms and dive into a specific use case related to Change Data Capture (CDC)</p><p>Table of contents :</p><ul><li><a href="https://medium.com/@amar.kumar2013/comparing-data-lake-data-lakehouse-and-delta-lake-which-one-suits-your-data-needs-fc0d96e004de">Part 0: Data Lake vs. Data Lakehouse vs. Delta Lake: A CDC Use Case</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lake-with-amazon-s3-and-emr-use-cases-c2b3983f6d55">Part1: Building a Data Lake with Amazon S3 and EMR</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-a73a273ec3df">Part2a: Building Data Lakehouse using HUDI — part 1/2</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-part2-2-8e159309ffc4">Part2b : Building Data Lakehouse using HUDI — part 2/2</a></li><li><a href="https://medium.com/@amar.kumar2013/demystifying-delta-lake-with-aws-emr-b842084cdb9b">Part3: Building Delta Lake with AWS EMR</a></li></ul><p><strong>Overview :</strong></p><p>· <a href="#451a">The Journey So Far</a><br>· <a href="#88c2">Delta Lake Architecture: A Bridge Between Data Lakes &amp; Data Warehouses</a><br> ∘ <a href="#e9d5">Delta Lake Architecture</a><br> ∘ <a href="#d231">Capabilities:</a><br>· <a href="#eec6">Implementing CDC using AWS EMR and Delta- Lake</a><br> ∘ <a href="#eac3">Prerequisite:</a><br> ∘ <a href="#a9cf">Generating a dummy dataset</a><br> ∘ <a href="#699c">Checking for memory usage</a><br> ∘ <a href="#57fb">Writing and reading data from target</a><br> ∘ <a href="#13c6">Creating Delta table Instance</a><br> ∘ <a href="#a46a">Lets perform batch update and insert operation on Delta table</a><br> ∘ <a href="#c8bb">Arbitrary selective overwrite with replaceWhere</a><br> ∘ <a href="#fc5c">Delta Lake time travel</a><br> ∘ <a href="#c0df">Delta Lake Delete Operation</a><br> ∘ <a href="#0676">Code</a><br>· <a href="#c3bf">Conclusion</a></p><h3>The Journey So Far</h3><p>Before we dive into our next topic, let’s take a moment to recap our journey. In our previous posts, we ventured into the realms of Data Lakes, Data Lakehouses, and Apache Hudi, each offering unique capabilities in the data management landscape.</p><p>We explored Data Lakes, understanding their flexibility and raw data storage potential. Then, we delved into Data Lakehouses, witnessing how they blend the best of Data Lakes and data warehouses for structured querying and governance.</p><p>Our journey also led us to Apache Hudi, where we harnessed its power for Change Data Capture (CDC) operations. We discussed crucial features like ACID compliance, incremental processing, and time travel that make Apache Hudi a robust data management framework.</p><p>Now, as we embark on the Delta Lake chapter, we bring with us a wealth of knowledge and insights from our previous adventures. Delta Lake will unveil a new dimension of data storage, integrity, and real-time processing, adding to our ever-expanding arsenal of data management tools. Stay with us as we explore the intricacies and capabilities of Delta Lake!</p><h3>Delta Lake Architecture: A Bridge Between Data Lakes &amp; Data Warehouses</h3><p>Delta Lake is an open-source storage layer built atop a data lake that confers reliability and ACID (Atomicity, Consistency, Isolation, and Durability) transactions. It enables a continuous and simplified data architecture for organizations.</p><h4>Delta Lake Architecture</h4><p>Delta Lake is an improvement from the lambda architecture whereby streaming and batch processing occur parallel, and results merge to provide a query response</p><p>The stored data file has three layers, with the data getting more refined as it progresses downstream in the dataflow;</p><ol><li><strong>Bronze tables:</strong> This table contains the raw data ingested from multiple sources like the Internet of Things (IoT) systems, CRM, RDBMS, and JSON files.</li><li><strong>Silver tables:</strong> This layer contains a more refined view of our data after undergoing transformation and feature engineering processes.</li><li><strong>Gold tables:</strong> This final layer is often made available for end users in BI reporting and analysis or use in machine learning processes.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_LO14gfXfxZZakHCpRA_w.png" /><figcaption>Delta-Lake Multi-hop architecture</figcaption></figure><h4>Capabilities:</h4><ol><li>Atomic, consistent, isolated, durable (ACID) transactions on Spark. Readers see a consistent view of the table during a Spark job.</li><li>Scalable metadata handling with distributed processing by Spark.</li><li>Combines streaming and batch uses cases with the same Delta table.</li><li>Automatic schema enforcement to avoid bad records during data ingestion.</li><li>Time travel with data versioning.</li><li>Supports merge, update, and delete operations for complex use cases like change data capture (CDC), streaming upserts.</li></ol><h3>Implementing CDC using AWS EMR and Delta- Lake</h3><h4>Prerequisite:</h4><p>Configure on the EMR notebook to use Delta- Lake</p><pre>%%configure -f<br>{<br>  &quot;conf&quot;: {<br>    &quot;spark.sql.extensions&quot;:  &quot;io.delta.sql.DeltaSparkSessionExtension&quot;,<br>     &quot;spark.sql.catalog.spark_catalog&quot;:  &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;,<br>      &quot;spark.jars&quot;:&quot;hdfs:///apps/delta/lib/delta-core_2.12-2.0.0.jar&quot;,<br>      &quot;spark.jars.packages&quot;: &quot;io.delta:delta-core_2.12:2.0.0&quot;<br>  }<br>}<br><br>from delta import *<br> <br>builder = spark.sql.SparkSession.builder.appName(&quot;Delta_App&quot;) \<br>    .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;) \<br>    .config(&quot;spark.sql.catalog.spark_catalog&quot;, &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;)<br><br>spark = configure_spark_with_delta_pip(builder).getOrCreate()<br><br>delta_version =spark.conf.get(&quot;spark.jars.packages&quot;)<br>print(&quot;Delta Lake version:&quot;, delta_version)<br><br>## OUTUPUT:<br>Delta Lake version: io.delta:delta-core_2.12:2.4.0</pre><h4>Generating a dummy dataset</h4><pre>from delta.tables import *<br>from pyspark.sql.functions import *<br>from datetime import datetime<br>from delta import DeltaTable<br><br>config = {<br>    &quot;table_name&quot;: &quot;delta_trips_table&quot;,<br>    &quot;target&quot;: &quot;/tmp/delta-table&quot;,<br>    &quot;primary_key&quot;: &quot;trip_id&quot;<br>}<br><br># Generate Dummy Dataset <br><br>from datetime import datetime<br>dest = [&quot;Seattle&quot;, &quot;New York&quot;, &quot;New Jersey&quot;, &quot;Los Angeles&quot;, &quot;Las Vegas&quot;, &quot;Tucson&quot;,&quot;Washington DC&quot;,&quot;Philadelphia&quot;,&quot;Miami&quot;,&quot;San Francisco&quot;]<br>def get_json_data(start, count, dest):<br>    time_stamp = datetime.now().strftime(&#39;%Y-%m-%d %H:%M:%S.%f&#39;)[:-3]<br>    data = [{&quot;trip_id&quot;: i, &quot;ts&quot;: time_stamp, &quot;route_id&quot;: chr(65 + (i % 10)), &quot;destination&quot;: dest[i%10]} for i in range(start, start + count)]<br>    return data<br><br>#Create a spark dataframe<br>def create_json_df(spark, data):<br>    sc = spark.sparkContext<br>    return spark.read.json(sc.parallelize(data))<br><br>#Inserting dataset of large size<br>df = create_json_df(spark, get_json_data(0, 2000000, dest))</pre><h4>Checking for memory usage</h4><pre># Checking the memory <br>from pyspark.serializers import PickleSerializer, AutoBatchedSerializer<br>def _to_java_object_rdd(rdd):  <br>    &quot;&quot;&quot; Return a JavaRDD of Object by unpickling<br>    It will convert each Python object into Java object by Pyrolite, whenever the<br>    RDD is serialized in batch or not.<br>    &quot;&quot;&quot;<br>    rdd = rdd._reserialize(AutoBatchedSerializer(PickleSerializer()))<br>    return rdd.ctx._jvm.org.apache.spark.mllib.api.python.SerDe.pythonToJava(rdd._jrdd, True)<br><br>JavaObj = _to_java_object_rdd(df.rdd)<br>nbytes = spark._jvm.org.apache.spark.util.SizeEstimator.estimate(JavaObj)<br>print(&quot;Memory usage of DataFrame: {:.2f} MB&quot;.format(nbytes/1000000))<br><br>## OUTPUT<br>Memory usage of DataFrame: 80.26 MB</pre><h4>Writing and reading data from target</h4><pre><br>import time <br>df.write.format(&#39;delta&#39;).mode(&quot;overwrite&quot;).save(config[&#39;target&#39;])<br><br>#  set the table location in Spark config, which allows you to read the data using SQL format<br>spark.conf.set(&#39;table.location&#39;, config[&#39;target&#39;])<br><br>df_delta = spark.read.format(&quot;delta&quot;).load(config[&#39;target&#39;])<br><br>+-------------+--------+-------+--------------------+<br>|  destination|route_id|trip_id|                  ts|<br>+-------------+--------+-------+--------------------+<br>|      Seattle|       A|      0|2023-09-15 06:00:...|<br>|     New York|       B|      1|2023-09-15 06:00:...|<br>|   New Jersey|       C|      2|2023-09-15 06:00:...|<br>|  Los Angeles|       D|      3|2023-09-15 06:00:...|<br>|    Las Vegas|       E|      4|2023-09-15 06:00:...|<br>|       Tucson|       F|      5|2023-09-15 06:00:...|<br>|Washington DC|       G|      6|2023-09-15 06:00:...|<br>| Philadelphia|       H|      7|2023-09-15 06:00:...|<br>|        Miami|       I|      8|2023-09-15 06:00:...|<br>|San Francisco|       J|      9|2023-09-15 06:00:...|<br>|      Seattle|       A|     10|2023-09-15 06:00:...|<br>|     New York|       B|     11|2023-09-15 06:00:...|<br>|   New Jersey|       C|     12|2023-09-15 06:00:...|<br>|  Los Angeles|       D|     13|2023-09-15 06:00:...|<br>|    Las Vegas|       E|     14|2023-09-15 06:00:...|<br>|       Tucson|       F|     15|2023-09-15 06:00:...|<br>|Washington DC|       G|     16|2023-09-15 06:00:...|<br>| Philadelphia|       H|     17|2023-09-15 06:00:...|<br>|        Miami|       I|     18|2023-09-15 06:00:...|<br>|San Francisco|       J|     19|2023-09-15 06:00:...|<br>+-------------+--------+-------+--------------------+</pre><h4>Creating Delta table Instance</h4><pre># create DeltaTable instances using the path of the Delta table <br># This will help us to use different feature created for CDC operation on delta table<br>deltaTable = DeltaTable.forPath(spark, config[&#39;target&#39;])</pre><h4>Lets perform batch update and insert operation on Delta table</h4><p>There is an excellent documentation available for delta lake and how to implement its several statement. Refer the link below</p><p>Link: <a href="https://docs.delta.io/latest/delta-update.html">https://docs.delta.io/latest/delta-update.html</a></p><pre># Changing data for upsert<br>upsert_dest_delta= [&quot;mexico&quot; for i in range (0,100)]<br>df3_delta=create_json_df(spark, get_json_data(1000000,100, upsert_dest_delta))<br>df3_delta.show(20,False)<br><br>+-----------+--------+-------+-----------------------+<br>|destination|route_id|trip_id|ts                     |<br>+-----------+--------+-------+-----------------------+<br>|mexico     |A       |1000000|2023-09-15 06:04:14.833|<br>|mexico     |B       |1000001|2023-09-15 06:04:14.833|<br>|mexico     |C       |1000002|2023-09-15 06:04:14.833|<br>|mexico     |D       |1000003|2023-09-15 06:04:14.833|<br>|mexico     |E       |1000004|2023-09-15 06:04:14.833|<br>|mexico     |F       |1000005|2023-09-15 06:04:14.833|<br>|mexico     |G       |1000006|2023-09-15 06:04:14.833|<br>|mexico     |H       |1000007|2023-09-15 06:04:14.833|<br>|mexico     |I       |1000008|2023-09-15 06:04:14.833|<br>|mexico     |J       |1000009|2023-09-15 06:04:14.833|<br>|mexico     |A       |1000010|2023-09-15 06:04:14.833|<br>|mexico     |B       |1000011|2023-09-15 06:04:14.833|<br>|mexico     |C       |1000012|2023-09-15 06:04:14.833|<br>|mexico     |D       |1000013|2023-09-15 06:04:14.833|<br>|mexico     |E       |1000014|2023-09-15 06:04:14.833|<br>|mexico     |F       |1000015|2023-09-15 06:04:14.833|<br>|mexico     |G       |1000016|2023-09-15 06:04:14.833|<br>|mexico     |H       |1000017|2023-09-15 06:04:14.833|<br>|mexico     |I       |1000018|2023-09-15 06:04:14.833|<br>|mexico     |J       |1000019|2023-09-15 06:04:14.833|<br>+-----------+--------+-------+-----------------------+<br><br># PERFORM THE UPSERT OPERATION<br>(deltaTable<br>.alias(&#39;t&#39;)<br>.merge(df3_delta.alias(&#39;u&#39;), &#39;t.trip_id = u.trip_id&#39;)<br>.whenMatchedUpdateAll()<br>.whenNotMatchedInsertAll()<br>.execute())<br><br># Inserting new data<br>insert_dest=[&quot;Bengaluru&quot; for i in range(0,100)]<br>df5_mor=create_json_df(spark,get_json_data(2000000,100,insert_dest))<br><br>## Perform the Insert Operation<br>(deltaTable<br>.alias(&#39;t&#39;)<br>.merge(df5_mor.alias(&#39;u&#39;), &#39;t.trip_id = u.trip_id&#39;)<br>.whenMatchedUpdateAll()<br>.whenNotMatchedInsertAll()<br>.execute())</pre><h4>Arbitrary selective overwrite with replaceWhere</h4><p>This sample code writes out the data in replace_data, validates that all rows match the predicate, and performs an atomic replacement using overwrite semantics. If any values in the operation fall outside the constraint, this operation fails with an error by default.</p><pre>(replace_data.write<br>  .mode(&quot;overwrite&quot;)<br>  .option(&quot;replaceWhere&quot;, &quot;start_date &gt;= &#39;2017-01-01&#39; AND end_date &lt;= &#39;2017-01-31&#39;&quot;)<br>  .save(config[&#39;target&#39;])<br>)<br></pre><h4>Delta Lake time travel</h4><p>Delta Lake stores data in Parquet files and information about transactions in the _delta_log metadata folder.</p><p>The _delta_log metadata folder tracks the Parquet data files that are added and removed from the Delta table for each transaction.</p><ul><li>For Version 0, Delta Lake just needs to read File A</li><li>Delta Lake will see both File A and File B should be read for Version 1</li><li>For Version 2, Delta Lake will see that File A, File B, and File C were added, but File A and File B were removed, so only File C should be read. Delta Lake will only read File C and skip the other files when reading Version 2.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*Ym3t5fQCSQT4qInD1OJnQQ.png" /><figcaption>Delta table Time travel</figcaption></figure><pre># Time travel <br>deltaTable.history(100).show()<br><br>+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+------------+--------------------+<br>|version|          timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend|    operationMetrics|userMetadata|          engineInfo|<br>+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+------------+--------------------+<br>|      2|2023-05-18 11:32:33|  null|    null|    MERGE|{predicate -&gt; (t....|null|    null|     null|          1|  Serializable|        false|{numTargetRowsCop...|        null|Apache-Spark/3.3....|<br>|      1|2023-05-18 11:29:55|  null|    null|    MERGE|{predicate -&gt; (t....|null|    null|     null|          0|  Serializable|        false|{numTargetRowsCop...|        null|Apache-Spark/3.3....|<br>|      0|2023-05-18 11:27:31|  null|    null|    WRITE|{mode -&gt; Overwrit...|null|    null|     null|       null|  Serializable|        false|{numFiles -&gt; 2, n...|        null|Apache-Spark/3.3....|<br>+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+------------+--------------------+<br><br>## Have a more clear understanding about the time travel by selecting important feature<br>deltaTable.history(100).select(&quot;version&quot;, &quot;timestamp&quot;, &quot;operation&quot;, &quot;operationParameters&quot;).show(truncate=False)<br><br><br>+-------+-------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------+<br>|version|timestamp          |operation|operationParameters                                                                                                                      |<br>+-------+-------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------+<br>|2      |2023-05-18 11:32:33|MERGE    |{predicate -&gt; (t.trip_id = u.trip_id), matchedPredicates -&gt; [{&quot;actionType&quot;:&quot;update&quot;}], notMatchedPredicates -&gt; [{&quot;actionType&quot;:&quot;insert&quot;}]}|<br>|1      |2023-05-18 11:29:55|MERGE    |{predicate -&gt; (t.trip_id = u.trip_id), matchedPredicates -&gt; [{&quot;actionType&quot;:&quot;update&quot;}], notMatchedPredicates -&gt; [{&quot;actionType&quot;:&quot;insert&quot;}]}|<br>|0      |2023-05-18 11:27:31|WRITE    |{mode -&gt; Overwrite, partitionBy -&gt; []}                                                                                                   |<br>+-------+-------------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------+<br><br>## Read the selected data using version - Time Travel<br>## Selecting the version -1 <br>df_time_travel = spark.read.format(&quot;delta&quot;).option(&quot;versionAsOf&quot;, 1).load(config[&#39;target&#39;])<br>df_time_travel.show()<br><br>+-------------+--------+-------+--------------------+<br>|  destination|route_id|trip_id|                  ts|<br>+-------------+--------+-------+--------------------+<br>|        Miami|       I|     28|2023-05-18 11:23:...|<br>| Philadelphia|       H|    107|2023-05-18 11:23:...|<br>|San Francisco|       J|    139|2023-05-18 11:23:...|<br>| Philadelphia|       H|    157|2023-05-18 11:23:...|<br>|    Las Vegas|       E|    174|2023-05-18 11:23:...|<br>|        Miami|       I|    178|2023-05-18 11:23:...|<br>|        Miami|       I|    208|2023-05-18 11:23:...|<br>| Philadelphia|       H|    227|2023-05-18 11:23:...|<br>|   New Jersey|       C|    242|2023-05-18 11:23:...|<br>|  Los Angeles|       D|    253|2023-05-18 11:23:...|<br>|       Tucson|       F|    255|2023-05-18 11:23:...|<br>|Washington DC|       G|    266|2023-05-18 11:23:...|<br>|        Miami|       I|    268|2023-05-18 11:23:...|<br>| Philadelphia|       H|    307|2023-05-18 11:23:...|<br>|Washington DC|       G|    316|2023-05-18 11:23:...|<br>|      Seattle|       A|    320|2023-05-18 11:23:...|<br>|Washington DC|       G|    326|2023-05-18 11:23:...|<br>|  Los Angeles|       D|    353|2023-05-18 11:23:...|<br>|       Tucson|       F|    365|2023-05-18 11:23:...|<br>|      Seattle|       A|    370|2023-05-18 11:23:...|<br>+-------------+--------+-------+--------------------+</pre><p>Note : Similarly, we can also perform time travel based on timestamp</p><h4>Delta Lake Delete Operation</h4><p>Delta Lake makes it easy to efficiently delete rows from a Delta table. Delta Lake delete operations add a new transaction that tombstones any files with data that should be deleted and adds new files with the deleted rows filtered out</p><p><strong><em>Delta Lake Delete Limitations because of Parquet</em></strong></p><p>Delta Lakes consist of a transaction log and data stored in Parquet files.Parquet files are immutable, which means that they don’t allow you to add or delete rows. If you want to “delete” rows from a Parquet file, you need to read the data into memory, filter out the rows you don’t want, and create a new Parquet file.</p><p>Delta Lake has to write new files when deleting rows because of the immutable nature of Parquet files</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*KhmGJsRK1Tp9EgThrIQB7w.png" /><figcaption>Delete Operation</figcaption></figure><pre>## Delete Operation<br>condition=&quot;trip_id &gt; 2000050&quot;<br>deltaTable.delete(condition)</pre><h4>Code: <a href="https://github.com/amar5075kumar/CDC-use-case">You can find the full code on my GitHub repository</a></h4><h3>Conclusion</h3><p>As we conclude our exploration of Delta Lake, it’s clear that Delta Lake isn’t just another data storage solution; it’s an indispensable tool for organizations striving for excellence in data handling. Incorporating Delta Lake into your data architecture empowers your organization to unlock the full potential of your data, enabling data-driven decision-making, advanced analytics, and real-time insights.</p><p>So, as you embark on your own data management journey, consider the transformative power of Delta Lake. It’s more than a technology; it’s a partner on your path to data excellence, and it’s ready to elevate your data management practices to new heights.</p><p>Thank you for joining us in unraveling the capabilities of Delta Lake, and we look forward to continuing this data-driven adventure together in our future blog posts. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b842084cdb9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Lakehouse with AWS EMR and Apache Hudi: A CDC Use Case — part2/2]]></title>
            <link>https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-part2-2-8e159309ffc4?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e159309ffc4</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 16:40:07 GMT</pubDate>
            <atom:updated>2023-09-16T06:49:46.480Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Data Lakehouse with AWS EMR and Apache Hudi: A CDC Use Case — part2/2</h3><p>In the ever-evolving landscape of data storage and processing, three distinct solutions have emerged as game-changers: Data Lakes, Data Lakehouses, and Delta Lakes. While each of these technologies holds its own promise, the choice between them can significantly impact how you handle your data. To shed light on the matter, we will explore these data storage paradigms and dive into a specific use case related to Change Data Capture (CDC)</p><p>Table of contents :</p><ul><li><a href="https://medium.com/@amar.kumar2013/comparing-data-lake-data-lakehouse-and-delta-lake-which-one-suits-your-data-needs-fc0d96e004de">Part 0: Data Lake vs. Data Lakehouse vs. Delta Lake: A CDC Use Case</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lake-with-amazon-s3-and-emr-use-cases-c2b3983f6d55">Part1: Building a Data Lake with Amazon S3 and EMR</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-a73a273ec3df">Part2a: Building Data Lakehouse using HUDI — part 1/2</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-part2-2-8e159309ffc4">Part2b : Building Data Lakehouse using HUDI — part 2/2</a></li><li><a href="https://medium.com/@amar.kumar2013/demystifying-delta-lake-with-aws-emr-b842084cdb9b">Part3: Building Delta Lake with AWS EMR</a></li></ul><p>Overview:</p><p>· <a href="#e506">Today’s Focus: Apache Hudi — MOR Table Type</a><br>· <a href="#5b8a">Merge On Read(MOR) Table type In Hudi:</a><br>· <a href="#f328">Implementation of MOR table</a><br> ∘ <a href="#b129">1. Generating a dummy dataset</a><br> ∘ <a href="#f5cc">2. Use Spark SQL to read the MOR table</a><br> ∘ <a href="#442f">3. Lets perform batch update operation</a><br> ∘ <a href="#41a4">4. Lets perform batch insert operation</a><br> ∘ <a href="#6ba2">5. Read different read-optimized and real time data in dataframe</a><br> ∘ <a href="#9cb5">6. Time Travel</a><br> ∘ <a href="#85a4">Code</a><br>· <a href="#673b">Conclusion</a></p><p><strong><em>…Contiuation to the last part 2a . Do revert back to the previous post for building the Apache Hudi foundation for Change Data Capture (CDC)</em></strong></p><p>In our previous post, we laid the groundwork for implementing Change Data Capture (CDC) using Apache Hudi. Today, we’re diving deeper into the world of Apache Hudi with a specific focus on the MOR (Merge on Read) table type, which is particularly well-suited for CDC use cases.</p><p><strong>The Journey So Far</strong></p><p>Before we proceed, let’s recap our journey. In our last post, we set the stage for harnessing Apache Hudi’s power for CDC operations. We introduced the concept of Data Lakehouses, explored the role of Apache Hudi as a data management framework, and discussed its key features like ACID compliance, incremental processing, and time travel.</p><h3>Today’s Focus: Apache Hudi — MOR Table Type</h3><p>Today, our focus shifts to Apache Hudi’s MOR (Merge on Read) table type. The MOR approach is pivotal when dealing with real-time data changes and CDC. It ensures that data is efficiently merged and maintained as new changes arrive.</p><p><strong>What to Expect</strong></p><p>In this post, we’ll walk you through the practical implementation of an Apache Hudi MOR table. We’ll guide you step by step on how to set up and use this table type for CDC operations. By the end of this journey, you’ll have a clear understanding of how MOR tables play a vital role in capturing real-time changes.</p><p>So, buckle up and get ready to delve into the world of MOR tables with Apache Hudi!</p><p>Stay with us for the continuation of this exciting data-driven journey. In our next post, we’ll take you through the practical steps of setting up an Apache Hudi MOR table for your CDC needs.</p><h3>Merge On Read(MOR) Table type In Hudi:</h3><p>The default table type is Copy-On-Write which is best suited for read-heavy workloads with modest writes. Copy-On-Write creates commit files with original data + the new changes during writing itself. While this increases latency on writes, this set up makes it more manageable for faster read.</p><p>For near real-time applications that mandate quick upserts, MERGE_ON_READ table type would be better suited. MOR table stores incoming upserts for each file group, onto a row based delta log (In Avro file format). This log is then merged with the existing Parquet file using a a compactor during reads.</p><h3>Implementation of MOR table</h3><p>In the previous post we have configured all how to configure HUDI on EMR notebook and set the HUDI parameter. So lets directly jump towards the implementation part</p><pre>config = {<br>    &quot;table_name&quot;: &quot;hudi_mor_trips_table&quot;,<br>    &quot;table_name_rt&quot;: &quot;hudi_mor_trips_table_rt&quot;,<br>    &quot;target&quot;: &quot;s3:******************/hudi_mor/&quot;,<br>    &quot;primary_key&quot;: &quot;trip_id&quot;,<br>    &quot;sort_key&quot;: &quot;ts&quot;,<br>    &quot;commits_to_retain&quot;: &quot;2&quot;,<br>}<br><br>### Other than this we also need to add extra parameter for  using HUDI MOR<br>STORAGE_TYPE_OPT_KEY=&quot;hoodie.datasource.write.storage.type&quot;<br>COMPACTION_INLINE_OPT_KEY=&quot;hoodie.compact.inline&quot;<br>COMPACTION_MAX_DELTA_COMMITS_OPT_KEY=&quot;hoodie.compact.inline.max.delta.commits&quot;<br><br></pre><h4>1. Generating a dummy dataset</h4><p>Timestamp will be required as HUDI uses it for its internal computation based on Commit time.</p><pre>from datetime import datetime<br>dest = [&quot;Seattle&quot;, &quot;New York&quot;, &quot;New Jersey&quot;, &quot;Los Angeles&quot;, &quot;Las Vegas&quot;, &quot;Tucson&quot;,&quot;Washington DC&quot;,&quot;Philadelphia&quot;,&quot;Miami&quot;,&quot;San Francisco&quot;]<br>def get_json_data(start, count, dest):<br>    time_stamp = datetime.now().strftime(&#39;%Y-%m-%d %H:%M:%S.%f&#39;)[:-3]<br>    data = [{&quot;trip_id&quot;: i, &quot;ts&quot;: time_stamp, &quot;route_id&quot;: chr(65 + (i % 10)), &quot;destination&quot;: dest[i%10]} for i in range(start, start + count)]<br>    return data<br><br>#Create a spark dataframe<br>def create_json_df(spark, data):<br>    sc = spark.sparkContext<br>    return spark.read.json(sc.parallelize(data))<br><br>#Inserting dataset of large size<br>df = create_json_df(spark, get_json_data(0, 20000000, dest))<br><br>+-------------+--------+-------+-----------------------+<br>|destination  |route_id|trip_id|ts                     |<br>+-------------+--------+-------+-----------------------+<br>|Seattle      |A       |0      |2023-05-16 04:58:18.557|<br>|New York     |B       |1      |2023-05-16 04:58:18.557|<br>|New Jersey   |C       |2      |2023-05-16 04:58:18.557|<br>|Los Angeles  |D       |3      |2023-05-16 04:58:18.557|<br>|Las Vegas    |E       |4      |2023-05-16 04:58:18.557|<br>|Tucson       |F       |5      |2023-05-16 04:58:18.557|<br>|Washington DC|G       |6      |2023-05-16 04:58:18.557|<br>|Philadelphia |H       |7      |2023-05-16 04:58:18.557|<br>|Miami        |I       |8      |2023-05-16 04:58:18.557|<br>|San Francisco|J       |9      |2023-05-16 04:58:18.557|<br>|Seattle      |A       |10     |2023-05-16 04:58:18.557|<br>|New York     |B       |11     |2023-05-16 04:58:18.557|<br>|New Jersey   |C       |12     |2023-05-16 04:58:18.557|<br>|Los Angeles  |D       |13     |2023-05-16 04:58:18.557|<br>|Las Vegas    |E       |14     |2023-05-16 04:58:18.557|<br>|Tucson       |F       |15     |2023-05-16 04:58:18.557|<br>|Washington DC|G       |16     |2023-05-16 04:58:18.557|<br>|Philadelphia |H       |17     |2023-05-16 04:58:18.557|<br>|Miami        |I       |18     |2023-05-16 04:58:18.557|<br>|San Francisco|J       |19     |2023-05-16 04:58:18.557|<br><br>### Saving the data in target location<br>(df_mor.write.format(HUDI_FORMAT)<br>      .option(PRECOMBINE_FIELD_OPT_KEY,config[&quot;sort_key&quot;])<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, BULK_INSERT_OPERATION_OPT_VAL)<br>      .option(BULK_INSERT_PARALLELISM, 3)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)<br>      .option(STORAGE_TYPE_OPT_KEY,&quot;MERGE_ON_READ&quot;)<br>      .option(COMPACTION_INLINE_OPT_KEY,&quot;false&quot;)<br>      .option(COMPACTION_MAX_DELTA_COMMITS_OPT_KEY,&quot;20&quot;)<br>      .mode(&quot;Overwrite&quot;)<br>      .save(config[&#39;target&#39;])<br>)<br></pre><h4>2. Use Spark SQL to read the MOR table</h4><pre>Here 2 tables will be created for MOR that is read- optimized(_ro) and real-time(_rt)<br>RO tables are optimized for read-heavy workloads where you need fast query performance.<br>RT tables are designed for real-time data ingestion and querying, ideal for scenarios requiring low-latency access to data.<br></pre><pre><br>spark.sql(&quot;show tables&quot;).show(10,False)<br><br>+---------+-----------------------+-----------+<br>|namespace|tableName              |isTemporary|<br>+---------+-----------------------+-----------+<br>|default  |hudi_mor_trips_table_ro|false      |<br>|default  |hudi_mor_trips_table_rt|false      |<br>|default  |hudi_trips_table       |false      |<br>+---------+-----------------------+-----------+<br><br></pre><p>Hudi Creates extra features for its internal consumption and make process streamline . Do refer the above documentation for it.</p><p>_hoodie_commit_time, _hoodie_commit_seqno, _hoodie_record_key,_hoodie_partition_path,_hoodie_file_name</p><pre>df2_mor=spark.read.format(HUDI_FORMAT).load(config[&#39;target&#39;] + &quot;/*&quot;)<br>df2_mor.show(20,False)<br><br>+-------------------+----------------------+------------------+----------------------+-------------------------------------------------------------------------+-------------+--------+-------+-----------------------+<br>|_hoodie_commit_time|_hoodie_commit_seqno  |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name                                                        |destination  |route_id|trip_id|ts                     |<br>+-------------------+----------------------+------------------+----------------------+-------------------------------------------------------------------------+-------------+--------+-------+-----------------------+<br>|20230516095126885  |20230516095126885_0_2 |0                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|Seattle      |A       |0      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_5 |1                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|New York     |B       |1      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_6 |2                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|New Jersey   |C       |2      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_7 |3                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|Los Angeles  |D       |3      |2023-05-16 09:51:14.378|<br><br>spark.sql(&quot;select * from &quot; + config[&#39;table_name&#39;] + &quot;_ro&quot;).show(20,False)<br></pre><h4>3. Lets perform batch update operation</h4><pre># Changing data for upsert<br>upsert_dest_mor= [&quot;noida&quot; for i in range (0,100)]<br>df3_mor=create_json_df(spark, get_json_data(1000000,100, upsert_dest_mor))<br>df3_mor.show(20,False)<br><br>(df3_mor.write.format(HUDI_FORMAT)<br>      .option(PRECOMBINE_FIELD_OPT_KEY, config[&quot;sort_key&quot;])<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, UPSERT_OPERATION_OPT_VAL)<br>      .option(UPSERT_PARALLELISM, 3)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HUDI_CLEANER_POLICY, KEEP_LATEST_COMMITS)<br>      .option(HUDI_COMMITS_RETAINED,config[&quot;commits_to_retain&quot;])<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)  <br>      .option(STORAGE_TYPE_OPT_KEY, &quot;MERGE_ON_READ&quot;)<br>      .option(COMPACTION_INLINE_OPT_KEY, &quot;false&quot;)<br>      .option(COMPACTION_MAX_DELTA_COMMITS_OPT_KEY, &quot;20&quot;)<br>      .mode(&quot;Append&quot;)<br>      .save(config[&#39;target&#39;]))<br><br>df4_mor=spark.read.format(HUDI_FORMAT).load(config[&#39;target&#39;] + &quot;/*&quot;)<br><br></pre><h4>4. Lets perform batch insert operation</h4><pre>insert_dest=[&quot;ghaziabad&quot; for i in range(0,100)]<br>df5_mor=create_json_df(spark,get_json_data(2000000,100,insert_dest))<br>df5_mor.show()<br><br>(df5_mor.write.format(HUDI_FORMAT)<br>      .option(PRECOMBINE_FIELD_OPT_KEY, config[&quot;sort_key&quot;])<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, UPSERT_OPERATION_OPT_VAL)<br>      .option(UPSERT_PARALLELISM, 3)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HUDI_CLEANER_POLICY, KEEP_LATEST_COMMITS)<br>      .option(HUDI_COMMITS_RETAINED,config[&quot;commits_to_retain&quot;])<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)  <br>      .option(STORAGE_TYPE_OPT_KEY, &quot;MERGE_ON_READ&quot;)<br>      .option(COMPACTION_INLINE_OPT_KEY, &quot;false&quot;)<br>      .option(COMPACTION_MAX_DELTA_COMMITS_OPT_KEY, &quot;20&quot;)<br>      .mode(&quot;Append&quot;)<br>      .save(config[&#39;target&#39;]))<br><br>df6_mor=spark.read.format(HUDI_FORMAT).load(config[&#39;target&#39;] + &quot;/*&quot;)<br><br></pre><h4>5. Read different read-optimized and real time data in dataframe</h4><pre># read RO file as dataframe<br># Read Optimized (RO) files: These files contain immutable data that is optimized for read-heavy workloads. <br># RO files are created as a result of compaction and contain a full snapshot of the data at the time of compaction. <br># RO files can be queried using standard SQL engines like Presto, Athena, and Spark SQL<br># When a read query is executed against a Hudi MOR table, both RO and RT files are scanned to retrieve the latest data.<br># RO files are used for historical data, while RT files are used for the latest changes</pre><pre><br>ro_df = spark.read.format(&quot;org.apache.hudi&quot;) \<br>    .option(&quot;hoodie.table.name&quot;, &quot;hudi_mor_trips_table&quot;) \<br>    .option(&quot;hoodie.datasource.read.storage.type&quot;, &quot;RO&quot;) \<br>    .load(config[&#39;target&#39;] + &quot;/*&quot;)<br><br>ro_df.show(5,False)<br><br>+-------------------+----------------------+------------------+----------------------+-------------------------------------------------------------------------+-------------+--------+-------+-----------------------+<br>|_hoodie_commit_time|_hoodie_commit_seqno  |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name                                                        |destination  |route_id|trip_id|ts                     |<br>+-------------------+----------------------+------------------+----------------------+-------------------------------------------------------------------------+-------------+--------+-------+-----------------------+<br>|20230516095126885  |20230516095126885_0_2 |0                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|Seattle      |A       |0      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_5 |1                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|New York     |B       |1      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_6 |2                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|New Jersey   |C       |2      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_7 |3                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|Los Angeles  |D       |3      |2023-05-16 09:51:14.378|<br></pre><pre># Realtime (RT) files: These files contain mutable data that is optimized for write-heavy workloads.<br># RT files are written as new data is ingested into the Hudi dataset. <br># RT files are small and efficient, allowing for fast writes.</pre><pre>rt_df = spark.read.format(&quot;org.apache.hudi&quot;) \<br>    .option(&quot;hoodie.table.name&quot;, &quot;hudi_mor_trips_table&quot;) \<br>    .option(&quot;hoodie.datasource.read.storage.type&quot;, &quot;RT&quot;) \<br>    .load(config[&#39;target&#39;] + &quot;/*&quot;)<br>rt_df.show(5,False)<br><br>+-------------------+----------------------+------------------+----------------------+-------------------------------------------------------------------------+-------------+--------+-------+-----------------------+<br>|_hoodie_commit_time|_hoodie_commit_seqno  |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name                                                        |destination  |route_id|trip_id|ts                     |<br>+-------------------+----------------------+------------------+----------------------+-------------------------------------------------------------------------+-------------+--------+-------+-----------------------+<br>|20230516095126885  |20230516095126885_0_2 |0                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|Seattle      |A       |0      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_5 |1                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|New York     |B       |1      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_6 |2                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|New Jersey   |C       |2      |2023-05-16 09:51:14.378|<br>|20230516095126885  |20230516095126885_0_7 |3                 |                      |997f83b5-e22b-4bfb-9df5-11227aa3b185-0_0-2126-0_20230516095126885.parquet|Los Angeles  |D       |3      |2023-05-16 09:51:14.378|<br><br></pre><h4>6. Time Travel</h4><p>Used to query previous commit with incremental and point in time queries. This previous commit can be used for incremental queries</p><p>Incrementally query data — We can perform the incremental query which can speed up the retreival process based on the commit time.</p><pre>commits=spark.sql(&quot;select distinct(_hoodie_commit_time)  from hudi_trips_table order by _hoodie_commit_time&quot;).collect()<br>print(&quot;commits: &quot;)<br>for i in commits:<br>    print(i)<br><br>commits: <br>Row(_hoodie_commit_time=&#39;20230516073344246&#39;)<br>Row(_hoodie_commit_time=&#39;20230516073449459&#39;)<br>Row(_hoodie_commit_time=&#39;20230516073620516&#39;)<br><br><br><br># incrementally query data <br>beginTime=commits[-3]<br>incViewDF = spark.read.format(HUDI_FORMAT) \<br>                   .option(VIEW_TYPE_OPT_KEY, VIEW_TYPE_INCREMENTAL_OPT_VAL) \<br>                   .option(BEGIN_INSTANTTIME_OPT_KEY, beginTime) \<br>                   .load(config[&quot;target&quot;]+) </pre><h4>Code: <a href="https://github.com/amar5075kumar/CDC-use-case">You can find the full code on my GitHub repository</a></h4><h3>Conclusion</h3><p>Building a Data Lakehouse with AWS EMR, Apache Hudi, and S3 empowers you to harness the advantages of a modern data architecture while efficiently managing CDC use cases. You can capture and process real-time data changes with ease, ensuring data quality, consistency, and accessibility.</p><p>In this post we have implemented the HUDI- MOR implementation . This post completes the HUDI implementation using AWS EMR</p><p>In our upcoming blog post, we’ll redirect our attention to Delta Lake, delving into the intricacies of this powerful data storage solution. You’ll explore how Delta Lake bridges the gap between Data Lakes and data warehouses, offering a comprehensive platform for organizing, managing, and analyzing your data with unmatched reliability and security. Stay tuned for an enlightening dive into the world of Delta Lake — it’s an opportunity you won’t want to miss!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e159309ffc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Lakehouse with AWS EMR and Apache Hudi: A CDC Use Case — part1/2]]></title>
            <link>https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-a73a273ec3df?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a73a273ec3df</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 12:26:46 GMT</pubDate>
            <atom:updated>2023-09-16T06:48:20.652Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving landscape of data storage and processing, three distinct solutions have emerged as game-changers: Data Lakes, Data Lakehouses, and Delta Lakes. While each of these technologies holds its own promise, the choice between them can significantly impact how you handle your data. To shed light on the matter, we will explore these data storage paradigms and dive into a specific use case related to Change Data Capture (CDC)</p><p>Table of contents :</p><ul><li><a href="https://medium.com/@amar.kumar2013/comparing-data-lake-data-lakehouse-and-delta-lake-which-one-suits-your-data-needs-fc0d96e004de">Part 0: Data Lake vs. Data Lakehouse vs. Delta Lake: A CDC Use Case</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lake-with-amazon-s3-and-emr-use-cases-c2b3983f6d55">Part1: Building a Data Lake with Amazon S3 and EMR</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-a73a273ec3df">Part2a: Building Data Lakehouse using HUDI — part 1/2</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-part2-2-8e159309ffc4">Part2b : Building Data Lakehouse using HUDI — part 2/2</a></li><li><a href="https://medium.com/@amar.kumar2013/demystifying-delta-lake-with-aws-emr-b842084cdb9b">Part3: Building Delta Lake with AWS EMR</a></li></ul><p>Overview:</p><p>· <a href="#fd3a">Understanding Data Lakehouses(Brief Recap)</a><br>· <a href="#8d38">Meet Apache Hudi</a><br>· <a href="#323d">Specify the table type In Hudi: Copy on Write Vs. Merge on Read</a><br>· <a href="#b38f">HUDI Copy on Write (COW)</a><br>· <a href="#c5b8">HUDI MERGE ON READ(MOR)</a><br>· <a href="#7e1b">Implementing CDC using AWS EMR and Apache HUDI</a><br> ∘ <a href="#8de9">Prerequisites</a><br>· <a href="#eaa0">Lets get started with the implementation</a><br> ∘ <a href="#4128">1. Setting up the HUDI parameters for the use case.</a><br> ∘ <a href="#39d9">2. Generating a dummy dataset</a><br> ∘ <a href="#c0ed">3. Copy on Read (COW) implementation</a><br> ∘ <a href="#7dbf">4. Use Spark SQL to read the COW table</a><br> ∘ <a href="#83ec">5. Lets perform batch update operation</a><br> ∘ <a href="#497c">6. Lets perform batch insert operation</a><br> ∘ <a href="#b4cd">7. Lets perform delete operation</a><br> ∘ <a href="#78ab">Code</a><br>· <a href="#708f">Conclusion</a></p><h3>Understanding Data Lakehouses(Brief Recap)</h3><p>A Data Lakehouse is a modern data architecture that combines the best of both Data Lakes and data warehouses. It offers the ability to store vast amounts of raw data while providing structure and organization for efficient querying and analytics. Data Lakehouses address the challenges of data quality, consistency, and performance that Data Lakes sometimes encounter.</p><p>Key features of Data Lakehouses:</p><ul><li>Schema on Read: Like Data Lakes, Data Lakehouses allow you to store data without a predefined schema, enabling flexibility in data ingestion.</li><li>Structured Querying: Data Lakehouses introduce structured tables and indexes, facilitating SQL-based querying and making data more accessible for analysts and data scientists.</li><li>CDC Integration: Data Lakehouses are well-suited for Change Data Capture (CDC) use cases, where real-time data changes must be tracked and processed.</li></ul><p><strong>Simply put: Data Lake + Data Warehouse = Data Lakehouse.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*VZJqAIfl_vt4nANrHcP81Q.png" /><figcaption>Data Lakehouse Overview</figcaption></figure><h3>Meet Apache Hudi</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*ZgqeElsETdhzxRZALStU0w.png" /></figure><p>Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data management framework that simplifies the process of managing large-scale, incremental data. It provides support for both real-time and batch data processing and is particularly valuable for CDC operations.</p><p>Key features of Apache Hudi:</p><ul><li><strong>ACID Compliance</strong>: Hudi ensures Atomicity, Consistency, Isolation, and Durability (ACID) compliance for data operations, critical for maintaining data integrity during CDC.</li><li><strong>Incremental Processing</strong>: Hudi allows for efficient incremental data ingestion and processing, making it suitable for real-time data changes.</li><li><strong>Time Travel</strong>: The framework offers time-travel capabilities, enabling you to access data at different points in time, essential for historical analysis and auditing.</li></ul><p>There Is An excellent post explaining the which table type to be used in Apache Hudi. Do refer to this AWS blog to understand the intricacies of both the table and understand the key difference</p><p><a href="https://aws.amazon.com/blogs/big-data/part-1-build-your-apache-hudi-data-lake-on-aws-using-amazon-emr/">https://aws.amazon.com/blogs/big-data/part-1-build-your-apache-hudi-data-lake-on-aws-using-amazon-emr/</a></p><h3>Specify the table type In Hudi: Copy on Write Vs. Merge on Read</h3><h3><strong>HUDI Copy on Write (COW)</strong></h3><p>This is table type where commit table are fully merged into main table during write operation.</p><ul><li>For updates, the latest version of the that file id, is rewritten once, with new values for all records that have changed.</li><li>For inserts, the records are first packed onto the smallest file in each partition path, until it reaches the configured maximum size.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*1z7peTZU0kbbGIypKUP7kQ.png" /><figcaption>Hudi COW Table type</figcaption></figure><p><strong>2 Types of Queries Possible</strong>:</p><ol><li>Real Time/Snapshot queries — As soon as merge happens</li><li>Incremental Queries — Based on Commit_time</li></ol><p><strong>When to Use?</strong></p><p>1. COW is the default storage type and is preferred for simple read-heavy use cases</p><p>2. Tables with a lower ingestion rate and use cases without real-time ingestion</p><p>3. Use cases requiring the freshest data with minimal read latency because merging cost is taken care of at the write phase</p><p>4. Append-only workloads where existing data is immutable</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*OPE18IVB_9zGAiUeE0H7Bg.png" /><figcaption><strong>Process FLOW: HUDI DATA LAKE — COPY on Write Approach</strong></figcaption></figure><h3><strong>HUDI MERGE ON READ(MOR)</strong></h3><p>· In this table type, records written to the source file are quickly first written to log-files which are later merged with the source file using compaction action on the timeline.</p><p>· DELTA_COMMIT: Each new write creates incremental log files for updates, which are associated with the base Parquet files. For inserts, it creates a new version of the base file like CoW. Each write adds a delta commit action to the timeline.</p><p>· At a high level, MOR writer goes through same stages as COW writer in ingesting data. The updates are appended to latest log (delta) file belonging to the latest file slice without merging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*FL7p4rxj-GJoOfPNLK7DpA.png" /><figcaption><strong>HUDI MERGE ON READ(MOR)</strong></figcaption></figure><p><strong>3 types of queries are possible:</strong></p><ol><li>Read Optimized — Read-optimized queries show the latest compacted data, which doesn’t include the freshest updates in the not yet compacted log files.</li><li>Real Time/Snapshot queries — As soon as merge happens</li><li>Incremental Queries — Based on Commit_time</li></ol><p><strong>When to Use?</strong></p><ol><li>MoR is recommended for tables with write-heavy and update-heavy use cases.</li><li>Faster ingestion requirements and real-time ingestion use cases.</li><li>Varying or bursty write patterns (for example, ingesting bulk random deletes in an upstream database) due to the zero-merge cost for updates during write time.</li><li>Streaming use cases.</li><li>Mix of downstream consumers, where some are looking for fresher data by paying some additional read cost, and others need faster reads with some trade-off in data freshness.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*DO5zN4SB4PlwEPzKCikjRA.png" /><figcaption><strong>Process FLOW: HUDI DATA LAKE — MERGE on READ Approach</strong></figcaption></figure><h3>Implementing CDC using AWS EMR and Apache HUDI</h3><h4>Prerequisites</h4><p>Configure AWS EMR for Apache Hudi</p><ol><li><a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-work-with-dataset.html">So when you install spark on EMR cluster it comes with hudi already installed but it is local filesystem we need to copy it hdfs file system using the following step after that only we can run it in notebooks while you configure EMR</a> :</li></ol><p>In EC2 master instances connect it using SSH and implement the following commands to copy Hudi Jar file from its local filesystem to hdfs file system.</p><pre>## Make a lib folder inside the app folder on HDFS<br>hdfs dfs -mkdir -p /apps/hudi/lib<br><br>## Copy the Hudi Jar from local to HDFS<br>hdfs dfs -copyFromLocal /usr/lib/hudi/hudi-spark-bundle.jar /apps/hudi/lib/hudi-spark-bundle.jar<br><br>## Copy the spark-avro jar from local to HDFS<br>hdfs dfs -copyFromLocal /usr/lib/spark/external/lib/spark-avro.jar  /apps/hudi/lib/spark-avro.jar</pre><p>Configure on the EMR notebook to use HUDI</p><pre>%%configure -f<br>{ &quot;conf&quot;: {<br>            &quot;spark.jars&quot;:&quot;hdfs:///apps/hudi/lib/hudi-spark-bundle.jar&quot;,<br>            &quot;spark.serializer&quot;:&quot;org.apache.spark.serializer.KryoSerializer&quot;,<br>            &quot;spark.sql.catalog.spark_catalog&quot;: &quot;org.apache.spark.sql.hudi.catalog.HoodieCatalog&quot;,<br>            &quot;spark.sql.extensions&quot;:&quot;org.apache.spark.sql.hudi.HoodieSparkSessionExtension&quot;<br>          }}</pre><h3>Lets get started with the implementation</h3><h4>1. Setting up the HUDI parameters for the use case.</h4><p>Configuring the parameters can be tricky and need to have the understanding so it can be changed accordingly . Please refer to below documentation once to have a quick overview of parameter&#39;s and to take maximum advantage HUDI</p><p>Link: <a href="https://hudi.apache.org/docs/configurations">https://hudi.apache.org/docs/configurations</a></p><pre># General Constants - Basic Configuration<br>HUDI_FORMAT = &quot;org.apache.hudi&quot;<br>TABLE_NAME = &quot;hoodie.table.name&quot;<br>RECORDKEY_FIELD_OPT_KEY = &quot;hoodie.datasource.write.recordkey.field&quot;<br>PRECOMBINE_FIELD_OPT_KEY = &quot;hoodie.datasource.write.precombine.field&quot;<br>OPERATION_OPT_KEY = &quot;hoodie.datasource.write.operation&quot;<br>BULK_INSERT_OPERATION_OPT_VAL = &quot;bulk_insert&quot;<br>UPSERT_OPERATION_OPT_VAL = &quot;upsert&quot;<br>BULK_INSERT_PARALLELISM = &quot;hoodie.bulkinsert.shuffle.parallelism&quot;<br>UPSERT_PARALLELISM = &quot;hoodie.upsert.shuffle.parallelism&quot;<br>S3_CONSISTENCY_CHECK = &quot;hoodie.consistency.check.enabled&quot;<br>HUDI_CLEANER_POLICY = &quot;hoodie.cleaner.policy&quot;<br>KEEP_LATEST_COMMITS = &quot;KEEP_LATEST_COMMITS&quot;<br>HUDI_COMMITS_RETAINED = &quot;hoodie.cleaner.commits.retained&quot;<br>PAYLOAD_CLASS_OPT_KEY = &quot;hoodie.datasource.write.payload.class&quot;<br>EMPTY_PAYLOAD_CLASS_OPT_VAL = &quot;org.apache.hudi.common.model.EmptyHoodieRecordPayload&quot;<br><br># Hive Constants - Needed<br>HIVE_SYNC_ENABLED_OPT_KEY=&quot;hoodie.datasource.hive_sync.enable&quot;<br>HIVE_PARTITION_FIELDS_OPT_KEY=&quot;hoodie.datasource.hive_sync.partition_fields&quot;<br>HIVE_ASSUME_DATE_PARTITION_OPT_KEY=&quot;hoodie.datasource.hive_sync.assume_date_partitioning&quot;<br>HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY=&quot;hoodie.datasource.hive_sync.partition_extractor_class&quot;<br>HIVE_TABLE_OPT_KEY=&quot;hoodie.datasource.hive_sync.table&quot;<br><br># Partition Constants - For big dataset <br>NONPARTITION_EXTRACTOR_CLASS_OPT_VAL=&quot;org.apache.hudi.hive.NonPartitionedExtractor&quot;<br>MULTIPART_KEYS_EXTRACTOR_CLASS_OPT_VAL=&quot;org.apache.hudi.hive.MultiPartKeysValueExtractor&quot;<br>KEYGENERATOR_CLASS_OPT_KEY=&quot;hoodie.datasource.write.keygenerator.class&quot;<br>NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL=&quot;org.apache.hudi.keygen.NonpartitionedKeyGenerator&quot;<br>COMPLEX_KEYGENERATOR_CLASS_OPT_VAL=&quot;org.apache.hudi.ComplexKeyGenerator&quot;<br>PARTITIONPATH_FIELD_OPT_KEY=&quot;hoodie.datasource.write.partitionpath.field&quot;<br><br>#Incremental Constants<br>VIEW_TYPE_OPT_KEY=&quot;hoodie.datasource.view.type&quot;<br>BEGIN_INSTANTTIME_OPT_KEY=&quot;hoodie.datasource.read.begin.instanttime&quot;<br>VIEW_TYPE_INCREMENTAL_OPT_VAL=&quot;incremental&quot;<br>END_INSTANTTIME_OPT_KEY=&quot;hoodie.datasource.read.end.instanttime&quot;</pre><h4>2. Generating a dummy dataset</h4><p>Timestamp will be required as HUDI uses it for its internal computation based on Commit time.</p><pre>from datetime import datetime<br>dest = [&quot;Seattle&quot;, &quot;New York&quot;, &quot;New Jersey&quot;, &quot;Los Angeles&quot;, &quot;Las Vegas&quot;, &quot;Tucson&quot;,&quot;Washington DC&quot;,&quot;Philadelphia&quot;,&quot;Miami&quot;,&quot;San Francisco&quot;]<br>def get_json_data(start, count, dest):<br>    time_stamp = datetime.now().strftime(&#39;%Y-%m-%d %H:%M:%S.%f&#39;)[:-3]<br>    data = [{&quot;trip_id&quot;: i, &quot;ts&quot;: time_stamp, &quot;route_id&quot;: chr(65 + (i % 10)), &quot;destination&quot;: dest[i%10]} for i in range(start, start + count)]<br>    return data<br><br>#Create a spark dataframe<br>def create_json_df(spark, data):<br>    sc = spark.sparkContext<br>    return spark.read.json(sc.parallelize(data))<br><br>#Inserting dataset of large size<br>df = create_json_df(spark, get_json_data(0, 20000000, dest))<br><br>+-------------+--------+-------+-----------------------+<br>|destination  |route_id|trip_id|ts                     |<br>+-------------+--------+-------+-----------------------+<br>|Seattle      |A       |0      |2023-05-16 04:58:18.557|<br>|New York     |B       |1      |2023-05-16 04:58:18.557|<br>|New Jersey   |C       |2      |2023-05-16 04:58:18.557|<br>|Los Angeles  |D       |3      |2023-05-16 04:58:18.557|<br>|Las Vegas    |E       |4      |2023-05-16 04:58:18.557|<br>|Tucson       |F       |5      |2023-05-16 04:58:18.557|<br>|Washington DC|G       |6      |2023-05-16 04:58:18.557|<br>|Philadelphia |H       |7      |2023-05-16 04:58:18.557|<br>|Miami        |I       |8      |2023-05-16 04:58:18.557|<br>|San Francisco|J       |9      |2023-05-16 04:58:18.557|<br>|Seattle      |A       |10     |2023-05-16 04:58:18.557|<br>|New York     |B       |11     |2023-05-16 04:58:18.557|<br>|New Jersey   |C       |12     |2023-05-16 04:58:18.557|<br>|Los Angeles  |D       |13     |2023-05-16 04:58:18.557|<br>|Las Vegas    |E       |14     |2023-05-16 04:58:18.557|<br>|Tucson       |F       |15     |2023-05-16 04:58:18.557|<br>|Washington DC|G       |16     |2023-05-16 04:58:18.557|<br>|Philadelphia |H       |17     |2023-05-16 04:58:18.557|<br>|Miami        |I       |18     |2023-05-16 04:58:18.557|<br>|San Francisco|J       |19     |2023-05-16 04:58:18.557|<br>+-------------+--------+-------+-----------------------+</pre><h4>3. <strong>Copy on Read (COW)</strong> implementation</h4><pre>## Configuring the table and other parameter <br>config = {<br>    &quot;table_name&quot;: &quot;hudi_trips_table&quot;,<br>    &quot;target&quot;: &quot;s3://********************/***************/*****&quot;,<br>    &quot;primary_key&quot;: &quot;trip_id&quot;,<br>    &quot;sort_key&quot;: &quot;ts&quot;,<br>    &quot;commits_to_retain&quot;: &quot;2&quot;<br>}<br><br>### Writing the data to the S3 bucket in the target<br><br>(df.write.format(HUDI_FORMAT)<br>      .option(PRECOMBINE_FIELD_OPT_KEY,config[&quot;sort_key&quot;])<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, BULK_INSERT_OPERATION_OPT_VAL)<br>      .option(BULK_INSERT_PARALLELISM, 3)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)<br>      .mode(&quot;Overwrite&quot;)<br>      .save(config[&#39;target&#39;]))<br></pre><h4>4. Use Spark SQL to read the COW table</h4><pre>## Check the table created <br>spark.sql(&quot;show tables&quot;).show()<br><br>+---------+----------------+-----------+<br>|namespace|       tableName|isTemporary|<br>+---------+----------------+-----------+<br>|  default|hudi_trips_table|      false|<br>+---------+----------------+-----------+<br><br>## Reading the Metadata <br>spark.sql(&quot;show create table &quot;+config[&#39;table_name&#39;]).show(10,truncate=False)<br>+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>|createtab_stmt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |<br>+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>|CREATE TABLE default.hudi_trips_table (\n  _hoodie_commit_time STRING,\n  _hoodie_commit_seqno STRING,\n  _hoodie_record_key STRING,\n  _hoodie_partition_path STRING,\n  _hoodie_file_name STRING,\n  destination STRING,\n  route_id STRING,\n  trip_id BIGINT,\n  ts STRING)\nUSING hudi\nOPTIONS (\n  &#39;hoodie.query.as.ro.table&#39; = &#39;false&#39;)\nLOCATION &#39;s3://emrpocgigaforce/Output/hudi_data/hudi&#39;\nTBLPROPERTIES (\n  &#39;bucketing_version&#39; = &#39;2&#39;,\n  &#39;last_commit_time_sync&#39; = &#39;20230516073344246&#39;,\n  &#39;last_modified_by&#39; = &#39;hive&#39;,\n  &#39;last_modified_time&#39; = &#39;1684213183&#39;)\n|<br>+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br><br></pre><p>4. Hudi Creates extra features for its internal consumption and make process streamline . Do refer the above documentation for it.</p><p>_hoodie_commit_time, _hoodie_commit_seqno, _hoodie_record_key,_hoodie_partition_path,_hoodie_file_name</p><pre>## Reading the data from S3 in dataframe<br># Read the table from s3 <br>df2=spark.read.format(HUDI_FORMAT).load(config[&quot;target&quot;]+&quot;/*&quot;)<br>df2.show()<br>+-------------------+--------------------+------------------+----------------------+--------------------+-------------+--------+-------+--------------------+<br>|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|   _hoodie_file_name|  destination|route_id|trip_id|                  ts|<br>+-------------------+--------------------+------------------+----------------------+--------------------+-------------+--------+-------+--------------------+<br>|  20230516073344246|20230516073344246...|                 0|                      |b43666a0-e433-42d...|      Seattle|       A|      0|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 1|                      |b43666a0-e433-42d...|     New York|       B|      1|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 2|                      |b43666a0-e433-42d...|   New Jersey|       C|      2|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 3|                      |b43666a0-e433-42d...|  Los Angeles|       D|      3|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 4|                      |b43666a0-e433-42d...|    Las Vegas|       E|      4|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 5|                      |b43666a0-e433-42d...|       Tucson|       F|      5|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 6|                      |b43666a0-e433-42d...|Washington DC|       G|      6|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 7|                      |b43666a0-e433-42d...| Philadelphia|       H|      7|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 8|                      |b43666a0-e433-42d...|        Miami|       I|      8|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                 9|                      |b43666a0-e433-42d...|San Francisco|       J|      9|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                10|                      |b43666a0-e433-42d...|      Seattle|       A|     10|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                11|                      |b43666a0-e433-42d...|     New York|       B|     11|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                12|                      |b43666a0-e433-42d...|   New Jersey|       C|     12|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                13|                      |b43666a0-e433-42d...|  Los Angeles|       D|     13|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                14|                      |b43666a0-e433-42d...|    Las Vegas|       E|     14|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                15|                      |b43666a0-e433-42d...|       Tucson|       F|     15|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                16|                      |b43666a0-e433-42d...|Washington DC|       G|     16|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                17|                      |b43666a0-e433-42d...| Philadelphia|       H|     17|2023-05-16 04:58:...|<br>|  20230516073344246|20230516073344246...|                18|                      |b43666a0-e433-42d...|        Miami|       I|     18|2023-05-16 04:58:...|<br><br>df2.printSchema()<br><br>root<br> |-- _hoodie_commit_time: string (nullable = true)<br> |-- _hoodie_commit_seqno: string (nullable = true)<br> |-- _hoodie_record_key: string (nullable = true)<br> |-- _hoodie_partition_path: string (nullable = true)<br> |-- _hoodie_file_name: string (nullable = true)<br> |-- destination: string (nullable = true)<br> |-- route_id: string (nullable = true)<br> |-- trip_id: long (nullable = true)<br> |-- ts: string (nullable = true)</pre><h4>5. Lets perform batch update operation</h4><pre>upsert_dest= [&quot;boston&quot; for i in range (0,100)]<br>df3=create_json_df(spark, get_json_data(1000000,100, upsert_dest))<br><br>#  Update the changes in s3 now<br><br>(df3.write.format(HUDI_FORMAT)<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, UPSERT_OPERATION_OPT_VAL)<br>      .option(UPSERT_PARALLELISM, 4)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HUDI_CLEANER_POLICY, KEEP_LATEST_COMMITS)<br>      .option(HUDI_COMMITS_RETAINED,config[&quot;commits_to_retain&quot;])<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)  <br>      .mode(&quot;Append&quot;)   ## We will use append here to append it<br>      .save(config[&#39;target&#39;]))<br><br># Checking the unique commit time - This shows the data has been changed<br>spark.sql(&quot;select distinct(_hoodie_commit_time) as commitTime from hudi_trips_table order by commitTime&quot;).show(20, False)<br>+-----------------+<br>|commitTime       |<br>+-----------------+<br>|20230516073344246|<br>|20230516073449459|<br>+-----------------+</pre><h4>6. Lets perform batch insert operation</h4><pre># Inserting new data<br><br>insert_dest=[&quot;chennai&quot; for i in range(0,100)]<br>df4=create_json_df(spark,get_json_data(2000000,100,insert_dest))<br>df4.show()<br><br>+-----------+--------+-------+--------------------+<br>|destination|route_id|trip_id|                  ts|<br>+-----------+--------+-------+--------------------+<br>|    chennai|       A|2000000|2023-05-16 07:36:...|<br>|    chennai|       B|2000001|2023-05-16 07:36:...|<br>|    chennai|       C|2000002|2023-05-16 07:36:...|<br>|    chennai|       D|2000003|2023-05-16 07:36:...|<br>|    chennai|       E|2000004|2023-05-16 07:36:...|<br>|    chennai|       F|2000005|2023-05-16 07:36:...|<br>|    chennai|       G|2000006|2023-05-16 07:36:...|<br>|    chennai|       H|2000007|2023-05-16 07:36:...|<br>|    chennai|       I|2000008|2023-05-16 07:36:...|<br>|    chennai|       J|2000009|2023-05-16 07:36:...|<br>|    chennai|       A|2000010|2023-05-16 07:36:...|<br>|    chennai|       B|2000011|2023-05-16 07:36:...|<br>|    chennai|       C|2000012|2023-05-16 07:36:...|<br>|    chennai|       D|2000013|2023-05-16 07:36:...|<br>|    chennai|       E|2000014|2023-05-16 07:36:...|<br>|    chennai|       F|2000015|2023-05-16 07:36:...|<br>|    chennai|       G|2000016|2023-05-16 07:36:...|<br><br><br>#  Insert the changes in s3 now - same config of hudi options as upsert. Since it is new data it will be inserted directly <br>(df4.write.format(HUDI_FORMAT)<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, UPSERT_OPERATION_OPT_VAL)<br>      .option(UPSERT_PARALLELISM, 4)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HUDI_CLEANER_POLICY, KEEP_LATEST_COMMITS)<br>      .option(HUDI_COMMITS_RETAINED,config[&quot;commits_to_retain&quot;])<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)  <br>      .mode(&quot;Append&quot;)<br>      .save(config[&#39;target&#39;]))<br><br># Check the data <br>spark.sql(&quot;select * from &quot; + config[&#39;table_name&#39;] +&quot; where trip_id  between 2000000 and 2000100&quot;).show(50,False)</pre><h4>7. Lets perform delete operation</h4><p>Hudi supports implementing two types of deletes on data stored in Hudi tables, by enabling the user to specify a different record payload implementation.</p><p>For delete neither extra file not commit is added</p><p>It can’t be rollback since data has been deleted from hudi table</p><p>Soft Deletes : Retain the record key and just null out the values for all the other fields. This can be achieved by ensuring the appropriate fields are nullable in the table schema and simply upserting the table after setting these fields to null.</p><p>Hard Deletes : A stronger form of deletion is to physically remove any trace of the record from the table. This can be achieved in 3 different ways.</p><p>Using DataSource, set OPERATION_OPT_KEY to DELETE_OPERATION_OPT_VAL. This will remove all the records in the DataSet being submitted.</p><p>Using DataSource, set PAYLOAD_CLASS_OPT_KEY to “org.apache.hudi.EmptyHoodieRecordPayload”. This will remove all the records in the DataSet being submitted.</p><p>Using DataSource or DeltaStreamer, add a column named _hoodie_is_deleted to DataSet. The value of this column must be set to true for all the records to be deleted and either false or left null for any records which are to be upserted.</p><pre># Soft Delete<br>df6=df5.filter(col(&quot;trip_id&quot;).between(2000050,2000100))<br><br>(df6.write.format(HUDI_FORMAT)<br>      .option(RECORDKEY_FIELD_OPT_KEY, config[&quot;primary_key&quot;])<br>      .option(TABLE_NAME, config[&#39;table_name&#39;])<br>      .option(OPERATION_OPT_KEY, UPSERT_OPERATION_OPT_VAL)<br>      .option(UPSERT_PARALLELISM, 4)<br>      .option(S3_CONSISTENCY_CHECK, &quot;true&quot;)<br>      .option(HUDI_CLEANER_POLICY, KEEP_LATEST_COMMITS)<br>      .option(HUDI_COMMITS_RETAINED,config[&quot;commits_to_retain&quot;])<br>      .option(HIVE_TABLE_OPT_KEY,config[&#39;table_name&#39;])<br>      .option(HIVE_SYNC_ENABLED_OPT_KEY,&quot;true&quot;)<br>      .option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,NONPARTITION_EXTRACTOR_CLASS_OPT_VAL)<br>      .option(KEYGENERATOR_CLASS_OPT_KEY,NONPARTITIONED_KEYGENERATOR_CLASS_OPT_VAL)<br>      .option(PAYLOAD_CLASS_OPT_KEY,EMPTY_PAYLOAD_CLASS_OPT_VAL)    # Deleting the record so this option needs to be added <br>      .mode(&quot;Append&quot;)<br>      .save(config[&#39;target&#39;]))<br><br>## Check the hooide commit time to see the operations performed<br>spark.sql(&quot;select distinct(_hoodie_commit_time) as commitTime from hudi_trips_table order by commitTime&quot;).show(20, False)<br><br></pre><h4>Code: <a href="https://github.com/amar5075kumar/CDC-use-case">You can find the full code on my GitHub repository</a></h4><h3>Conclusion</h3><p>Building a Data Lakehouse with AWS EMR, Apache Hudi, and S3 empowers you to harness the advantages of a modern data architecture while efficiently managing CDC use cases. You can capture and process real-time data changes with ease, ensuring data quality, consistency, and accessibility.</p><p>In this post we have implemented the HUDI- COW implementation .</p><p>Next post will the implementaion of HUDI- MOR.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a73a273ec3df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Lake with Amazon S3 and EMR]]></title>
            <link>https://medium.com/@amar.kumar2013/building-a-data-lake-with-amazon-s3-and-emr-use-cases-c2b3983f6d55?source=rss-9002d419cfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2b3983f6d55</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Amar_Kumar]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 09:35:50 GMT</pubDate>
            <atom:updated>2023-09-16T06:47:14.347Z</atom:updated>
            <content:encoded><![CDATA[<h3>“Real-Time Data Capture in Data Lakes: Demystifying CDC with Practical Examples ”</h3><p>In our modern, data-centric world, we find ourselves inundated by a constant flow of information. For businesses, regardless of their scale or scope, the struggle isn’t just in accumulating and housing this data; it’s in uncovering valuable insights hidden within the digital torrent. This is precisely where the concepts of Data Lakes, Data Lakehouses, and Delta Lakes enter the picture, offering a lifeline to help organizations navigate this deluge of data</p><p>Table of contents :</p><ul><li><a href="https://medium.com/@amar.kumar2013/comparing-data-lake-data-lakehouse-and-delta-lake-which-one-suits-your-data-needs-fc0d96e004de">Part 0: Data Lake vs. Data Lakehouse vs. Delta Lake: A CDC Use Case</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lake-with-amazon-s3-and-emr-use-cases-c2b3983f6d55">Part1: Building a Data Lake with Amazon S3 and EMR</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-a73a273ec3df">Part2a: Building Data Lakehouse using HUDI — part 1/2</a></li><li><a href="https://medium.com/@amar.kumar2013/building-a-data-lakehouse-with-aws-emr-and-apache-hudi-a-cdc-use-case-part2-2-8e159309ffc4">Part2b : Building Data Lakehouse using HUDI — part 2/2</a></li><li><a href="https://medium.com/@amar.kumar2013/demystifying-delta-lake-with-aws-emr-b842084cdb9b">Part3: Building Delta Lake with AWS EMR</a></li></ul><p><strong>OVERVIEW</strong></p><p>· <a href="#7967">Understanding Data Lakes (Brief Recap)</a><br>· <a href="#8379">Building a Data Lake</a><br>· <a href="#cb6f">Setting Up Your Data Lake on Amazon S3</a><br>· <a href="#d287">Processing Data with EMR Notebooks</a><br>· <a href="#aee3">Change Data Capture (CDC) Upsert Operations: Code Example</a><br> ∘ <a href="#0d7e">1. Generating a dummy dataset</a><br> ∘ <a href="#2ea2">2. Checking for Memory Usage of Dataframe</a><br> ∘ <a href="#f028">3. Configuring the S3 Bucket and saving it in the bucket and reading back from it .</a><br> ∘ <a href="#15e4">4. Creating New target data for Insert and update:</a><br> ∘ <a href="#1618">5. Performing the Upsert operation</a><br> ∘ <a href="#93b7">6. Saving the data in S3 Bucket</a><br> ∘ <a href="#8ff6">Code</a><br>· <a href="#40be">Conclusion</a></p><h3>Understanding Data Lakes (Brief Recap)</h3><p>A data lake is a centralized, large-scale, and secure repository that allows the storage and management of structured and unstructured data at any scale. Unlike traditional data storage systems that organize data in a predefined structure, data lakes provide a flexible and scalable solution for storing raw data that can be accessed and analyzed by a wide range of applications and users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*CYOvH-tqZlwQsOyYt3DPwA.png" /><figcaption>Data Lake Overview</figcaption></figure><h3>Building a Data Lake</h3><h3>Setting Up Your Data Lake on Amazon S3</h3><ol><li>Create an S3 Bucket: Begin by creating an Amazon S3 bucket, where you will store your raw data. S3 buckets are highly scalable, durable, and accessible from EMR clusters.</li><li>Organize Data: Organize your data within the S3 bucket, creating directories or folders to structure it logically. For example, you might have separate folders for different data sources or data types.</li><li>Ingest Data: Upload or copy data into the S3 bucket. This can include log files, parquet, CSVs, JSON files, or any other data you need to analyze. S3 allows you to ingest data in a straightforward manner.</li></ol><h3>Processing Data with EMR Notebooks</h3><p>It is a fully managed big data processing service that makes it easy to run and scale Apache Hadoop, Apache Spark, and Presto clusters in the AWS Cloud.</p><p>With EMR, users can easily launch clusters, scale clusters up or down, and use a variety of big data processing frameworks and tools to process data.</p><p>EMR also integrates with a variety of AWS services, such as S3, EC2, and Kinesis, making it easy to move data in and out of EMR clusters.</p><p>Since this post is not about setting up the infrastructure. Configure the EMR before moving to next part.</p><h3><strong>Change Data Capture (CDC) </strong>Upsert Operations: Code Example</h3><p>Now, let’s dive into a code example that demonstrates how to pull data from your S3 Data Lake, perform <strong>Change Data Capture (CDC)</strong> upsert operations using EMR notebooks, and store the results back in the Data Lake.</p><h4>1. Generating a dummy dataset</h4><pre>## Generate Dummy Dataset<br>## Data Size is for 20 Million Rows <br><br>from datetime import datetime<br>dest = [&quot;Seattle&quot;, &quot;New York&quot;, &quot;New Jersey&quot;, &quot;Los Angeles&quot;, &quot;Las Vegas&quot;, &quot;Tucson&quot;,&quot;Washington DC&quot;,&quot;Philadelphia&quot;,&quot;Miami&quot;,&quot;San Francisco&quot;]<br>def get_json_data(start, count, dest):<br>    time_stamp = datetime.now().strftime(&#39;%Y-%m-%d %H:%M:%S.%f&#39;)[:-3]<br>    data = [{&quot;trip_id&quot;: i, &quot;ts&quot;: time_stamp, &quot;route_id&quot;: chr(65 + (i % 10)), &quot;destination&quot;: dest[i%10]} for i in range(start, start + count)]<br>    return data<br><br>#Create a spark dataframe<br>def create_json_df(spark, data):<br>    sc = spark.sparkContext<br>    return spark.read.json(sc.parallelize(data))<br><br>df = create_json_df(spark, get_json_data(0, 20000000, dest))</pre><h4>2. Checking for Memory Usage of Dataframe</h4><pre>## Check for the memory usage of the dataframe <br><br>from pyspark.serializers import PickleSerializer, AutoBatchedSerializer<br>def _to_java_object_rdd(rdd):  <br>    &quot;&quot;&quot; Return a JavaRDD of Object by unpickling<br>    It will convert each Python object into Java object by Pyrolite, whenever the<br>    RDD is serialized in batch or not.<br>    &quot;&quot;&quot;<br>    rdd = rdd._reserialize(AutoBatchedSerializer(PickleSerializer()))<br>    return rdd.ctx._jvm.org.apache.spark.mllib.api.python.SerDe.pythonToJava(rdd._jrdd, True)<br><br>JavaObj = _to_java_object_rdd(df.rdd)<br>nbytes = spark._jvm.org.apache.spark.util.SizeEstimator.estimate(JavaObj)<br>print(&quot;Memory usage of DataFrame: {:.2f} MB&quot;.format(nbytes/1000000))</pre><h4>3. Configuring the S3 Bucket and saving it in the bucket and reading back from it .</h4><pre>config = {<br>   <br>    &quot;target&quot;: &quot;s3:*********************&quot;<br>}<br>## Saving the data in  S3 Bucket <br>df.write.format(&#39;parquet&#39;).mode(&#39;overwrite&#39;).save(config[&#39;target&#39;])<br><br># Reading the data from S3 bucket<br>df=spark.read.format(&#39;parquet&#39;).load(config[&#39;target&#39;]+&quot;/*&quot;)<br><br></pre><h4>4. Creating New target data for Insert and update:</h4><pre># For update operation<br>update_dest= [&quot;boston&quot; for i in range (0,100)]<br>df_tgt_update=create_json_df(spark, get_json_data(1000000,100, update_dest))<br><br># For Insert Operation<br>insert_dest= [&quot;chennai&quot; for i in range (0,100)]<br>df_tgt_insert=create_json_df(spark, get_json_data(2000000,100, insert_dest))<br></pre><h4>5. Performing the Upsert operation</h4><pre>from pyspark.sql.functions import col, when<br><br>def upsert_function(df_src, df_tgt):<br>    # define suffix for df1 columns<br>    df_src_suffix = &#39;_src&#39;<br>    # define suffix for df2 columns<br>    df_tgt_suffix = &#39;_tgt&#39;<br><br>    # Adding Suffix to source &amp; target<br>    df_src = df_src.select([col(column_name).alias(column_name + df_src_suffix) for column_name in df_src.columns])<br>    df_tgt = df_tgt.select([col(column_name).alias(column_name + df_tgt_suffix) for column_name in df_tgt.columns])<br><br>    # Joining the data with a left join to only include matching trip IDs<br>    df_join = df_src.join(df_tgt, df_src.trip_id_src == df_tgt.trip_id_tgt, how=&#39;left&#39;)<br><br>    # Flag the Records for Update and Insert<br>    df_upsert = df_join.withColumn(&#39;Update_Flag&#39;,<br>                                   when((df_join.trip_id_src == df_join.trip_id_tgt) &amp;<br>                                        (df_join.ts_tgt &gt; df_join.ts_src), &quot;Y&quot;).otherwise(&quot;NA&quot;))<br><br>    # Create a DataFrame for records to be inserted (where trip_id_tgt is null)<br>    df_insert = df_upsert.filter(df_upsert.trip_id_tgt.isNull()).withColumn(&#39;Insert_Flag&#39;, lit(&#39;Y&#39;))<br><br>    src_cols = [col for col in df_upsert.columns if col.endswith(&#39;_src&#39;)]<br><br>    # Change the column names based on the list for updates<br>    for old_col, new_col in zip(df_upsert.columns, src_cols):<br>        df_upsert = df_upsert.withColumnRenamed(old_col, new_col)<br><br>    # Change the column names based on the list for inserts<br>    for old_col, new_col in zip(df_insert.columns, src_cols):<br>        df_insert = df_insert.withColumnRenamed(old_col, new_col)<br><br>    # Select only the source columns for updates<br>    df_updates = df_upsert.filter(df_upsert.Update_Flag == &#39;Y&#39;).select(src_cols)<br><br>    # Select only the source columns for inserts<br>    df_inserts = df_insert.filter(df_insert.Insert_Flag == &#39;Y&#39;).select(src_cols)<br><br>    # Union the records that don&#39;t need to be overridden (updates) and the records to be inserted<br>    df_final = df_updates.unionAll(df_inserts)<br><br>    return df_final<br><br>result=upsert_function(df,df_tgt_insert)<br>result.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*izaifviOhYGORGjmKhESLQ.png" /><figcaption><strong>Upserted</strong> <strong>Dataframe</strong></figcaption></figure><h4>6. Saving the data in S3 Bucket</h4><pre>result.write.format(&#39;parquet&#39;).save(config[&#39;target&#39;]+&quot;/output/&quot;)</pre><p>In the code example above, we initialize a Spark session, connect to S3, and read data from S3 into a Spark DataFrame. Within the DataFrame, you can implement CDC upsert operations to identify and process changes in your data. For instance, you can detect and apply updates to existing records, insert new records, and handle deletions based on specific criteria. Finally, the processed data is written back to S3.</p><h4>Code: <a href="https://github.com/amar5075kumar/CDC-use-case"><strong>You can find the full code on my GitHub repository</strong></a></h4><h3>Conclusion</h3><p>Building a Data Lake with Amazon S3 and utilizing Amazon EMR notebooks for data processing, including CDC upsert operations, empowers you to manage and analyze your data effectively. You can efficiently ingest, process, and capture changes in data from various sources, enabling real-time insights and informed decision-making.</p><p>Stay tuned for the next installment in our series, where we’ll dive into advanced data processing techniques within Amazon EMR, showcasing how to harness the full potential of your Data Lakehouse.</p><p>In our upcoming blog post, we’ll shift our focus to Data Lakehouses and explore how this innovative approach can further enhance your data processing capabilities. Discover how Data Lakehouses combine the best of Data Lakes and data warehouses, enabling you to efficiently organize, manage, and analyze your data for actionable insights. Don’t miss it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2b3983f6d55" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>