<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tushar Hatwar on Medium]]></title>
        <description><![CDATA[Stories by Tushar Hatwar on Medium]]></description>
        <link>https://medium.com/@tusharhatwar785?source=rss-3db603ff49fa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KgdPo88-opm1EF025U3B1A.jpeg</url>
            <title>Stories by Tushar Hatwar on Medium</title>
            <link>https://medium.com/@tusharhatwar785?source=rss-3db603ff49fa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:55:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tusharhatwar785/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Airflow Trigger Rules: A Comprehensive Guide]]></title>
            <link>https://medium.com/@tusharhatwar785/airflow-trigger-rules-a-comprehensive-guide-c34ee3ea5e3e?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/c34ee3ea5e3e</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Thu, 15 Feb 2024 17:16:02 GMT</pubDate>
            <atom:updated>2024-02-15T17:16:02.191Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XYoPIsPE2J5lI10Uzb6HFQ.jpeg" /></figure><p>Orchestrating complex workflows efficiently is key to successful data pipeline management.</p><p>One critical aspect of this orchestration is defining how tasks within a Directed Acyclic Graph (DAG) interact with each other.</p><p>This is where trigger rules come into play.</p><p>Trigger rules determine when a task should execute within the context of its dependencies.</p><p>They define the conditions under which a task can proceed with its execution. Airflow offers various trigger rules, each serving different purposes and offering unique control over workflow behavior.</p><pre># Import necessary libraries<br>import airflow<br>from airflow.models import DAG<br>from airflow.operators.python import PythonOperator<br><br># Define default arguments<br>default_args = {<br>    &#39;owner&#39;: &#39;Airflow&#39;,<br>    &#39;start_date&#39;: airflow.utils.dates.days_ago(1),<br>}<br><br># Define Python functions representing tasks<br>def download_website_a():<br>    print(&quot;Downloading website A&quot;)<br>    raise ValueError(&quot;error&quot;)<br><br>def download_website_b():<br>    print(&quot;Downloading website B&quot;)<br>    raise ValueError(&quot;error&quot;)<br><br>def download_failed():<br>    print(&quot;Download failed&quot;)<br>    # raise ValueError(&quot;error&quot;)<br><br>def download_succeed():<br>    print(&quot;Download succeeded&quot;)<br>    # raise ValueError(&quot;error&quot;)<br><br>def process():<br>    print(&quot;Processing data&quot;)<br>    # raise ValueError(&quot;error&quot;)<br><br>def notif_a():<br>    print(&quot;Notification A&quot;)<br>    # raise ValueError(&quot;error&quot;)<br><br>def notif_b():<br>    print(&quot;Notification B&quot;)<br>    # raise ValueError(&quot;error&quot;)<br><br># Define the DAG<br>with DAG(dag_id=&#39;trigger_rule_dag&#39;, <br>    default_args=default_args, <br>    schedule_interval=&quot;@daily&quot;) as dag:<br>    <br>    # Define tasks with corresponding trigger rules<br>    download_website_a_task = PythonOperator(<br>        task_id=&#39;download_website_a&#39;,<br>        python_callable=download_website_a,<br>        trigger_rule=&quot;all_success&quot;<br>    )<br><br>    download_website_b_task = PythonOperator(<br>        task_id=&#39;download_website_b&#39;,<br>        python_callable=download_website_b,<br>        trigger_rule=&quot;all_success&quot;    <br>    )<br><br>    download_failed_task = PythonOperator(<br>        task_id=&#39;download_failed&#39;,<br>        python_callable=download_failed,<br>        trigger_rule=&quot;all_failed&quot;<br>    )<br><br>    download_succeed_task = PythonOperator(<br>        task_id=&#39;download_succeed&#39;,<br>        python_callable=download_succeed,<br>        trigger_rule=&quot;all_success&quot;<br>    )<br><br>    process_task = PythonOperator(<br>        task_id=&#39;process&#39;,<br>        python_callable=process,<br>        trigger_rule=&quot;one_success&quot;<br>    )<br><br>    notif_a_task = PythonOperator(<br>        task_id=&#39;notif_a&#39;,<br>        python_callable=notif_a,<br>        trigger_rule=&quot;none_failed&quot;<br>    )<br><br>    notif_b_task = PythonOperator(<br>        task_id=&#39;notif_b&#39;,<br>        python_callable=notif_b,<br>        trigger_rule=&quot;one_failed&quot;<br>    )<br>    <br>    # Define task dependencies<br>    [download_website_a_task, download_website_b_task] &gt;&gt; download_succeed_task <br>    [download_website_a_task, download_website_b_task] &gt;&gt; download_failed_task <br>    [download_failed_task, download_succeed_task] &gt;&gt; process_task &gt;&gt; [notif_a_task, notif_b_task]</pre><figure><img alt="Graph View for the Dag" src="https://cdn-images-1.medium.com/max/1024/1*3_uLks2zEUS5Gqz_F6vTHQ.png" /></figure><h3>Trigger Rules</h3><p>Now, let’s dissect the trigger rules applied to each task and understand their implications:</p><ol><li>all_success: This rule dictates that the task should execute only if all of its directly upstream tasks have succeeded. In our example, both download_website_a_task and download_website_b_task must succeed for download_succeed_task to execute.</li><li>all_failed: This rule specifies that the task should execute only if all of its directly upstream tasks have failed. In our scenario, download_failed_task will run only if both download_website_a_task and download_website_b_task fail.</li><li>one_success: This rule indicates that the task should execute as soon as at least one of its directly upstream tasks succeeds. Here, process_task will run as soon as either download_failed_task or download_succeed_task completes successfully.</li><li>none_failed: This rule ensures that the task executes only if none of its directly upstream tasks have failed. In our example, notif_a_task will execute if neither download_website_a_task nor download_website_b_task fails.</li><li>one_failed: This rule mandates that the task should execute as soon as at least one of its directly upstream tasks fails. Thus, notif_b_task will run if either download_website_a_task or download_website_b_task fails.</li><li>none_skipped: This rule stipulates that the task should execute only if none of its directly upstream tasks are skipped. However, this rule is not explicitly demonstrated in our example DAG.</li></ol><p>Trigger rules play a crucial role in governing task execution within Airflow DAGs, providing flexibility and control over workflow behavior.</p><p>By strategically applying trigger rules, you can design robust and efficient data pipelines tailored to your specific requirements.</p><p>Feel free to connect with on <a href="https://www.linkedin.com/in/hatwar-tushar/">Linkedin</a> and <a href="https://github.com/tushar-hatwar">Github</a> :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c34ee3ea5e3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Task Groups in Apache Airflow]]></title>
            <link>https://medium.com/@tusharhatwar785/task-groups-in-apache-airflow-dcdb1e2be1a8?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcdb1e2be1a8</guid>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Mon, 12 Feb 2024 17:12:34 GMT</pubDate>
            <atom:updated>2024-02-12T17:12:34.472Z</atom:updated>
            <content:encoded><![CDATA[<p>Task Groups were introduced in Apache Airflow 2.0 as a way to group related tasks within a DAG.</p><p>They provide a logical structure for organizing tasks, making DAG definitions more modular and easier to understand.</p><p>With Task Groups, developers can encapsulate sets of tasks that belong together, improving code readability and maintainability.</p><p>Example DAG with Task Groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTVPcDRTzhm8QHCTsBnqAA.jpeg" /></figure><p>Code</p><pre>import airflow<br>from airflow.models import DAG<br>from airflow.operators.dummy_operator import DummyOperator<br>from airflow.utils.task_group import TaskGroup<br><br>DAG_NAME=&quot;test_taskgroup&quot;<br><br>default_args = {<br>    &#39;owner&#39;: &#39;Airflow&#39;,<br>    &#39;start_date&#39;: airflow.utils.dates.days_ago(1)<br>}<br><br>def factory_taskgroup(group_id):<br>    with TaskGroup(group_id=group_id) as tg:<br>        t1 = DummyOperator(task_id=&#39;task1&#39;)<br>        t2 = DummyOperator(task_id=&#39;task2&#39;)<br>    return tg<br><br>with DAG(dag_id=DAG_NAME, default_args=default_args, schedule_interval=&quot;@once&quot;) as dag:<br>    start = DummyOperator(task_id=&#39;start&#39;)<br><br>    with TaskGroup(&quot;group1&quot;) as tg1:<br>        factory_taskgroup(&#39;subgroup1&#39;)<br>        factory_taskgroup(&#39;subgroup2&#39;)<br><br>    some_other_task = DummyOperator(task_id=&#39;check&#39;)<br><br>    with TaskGroup(&quot;group2&quot;) as tg2:<br>        factory_taskgroup(&#39;subgroup1&#39;)<br><br>    end = DummyOperator(task_id=&#39;final&#39;)<br><br>    start &gt;&gt; tg1 &gt;&gt; some_other_task &gt;&gt; tg2 &gt;&gt; end</pre><p>Gannt Chart for the Dag:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NchpDMAWnLNZlCPmjJIZg.jpeg" /></figure><p>Task Groups in Airflow workflows provide:<br>1. <strong>Modularity</strong>: Organize tasks into modular units, enhancing DAG structure comprehension.<br>2. <strong>Code Reusability</strong>: Encapsulate tasks in functions like factory_taskgroup, enabling reuse across Task Groups for improved maintainability.<br>3. <strong>Clarity and Readability</strong>: Group related tasks intuitively, enhancing DAG readability and easing onboarding for new team members.<br>4. <strong>Simplified Maintenance</strong>: Isolate tasks within logical units, streamlining DAG modifications and reducing error risks.</p><p>Feel free to reach out to me on <a href="https://www.linkedin.com/in/hatwar-tushar/">LinkedIn</a> :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcdb1e2be1a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Important airflow Parameters]]></title>
            <link>https://medium.com/@tusharhatwar785/important-airflow-parameters-95f62962cad1?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/95f62962cad1</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 15:58:03 GMT</pubDate>
            <atom:updated>2024-02-07T15:58:03.810Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gcSMIfKOJIfymksu.png" /></figure><p>Understanding airflow.cfg:</p><p>The `airflow.cfg` file serves as the configuration file for your Airflow installation.</p><p>It contains various settings that govern the behavior of your Airflow instance.</p><p>Let’s explore a few critical parameters related to DAG (Directed Acyclic Graph) concurrency, task parallelism, and maximum active runs per DAG.</p><p>1. <strong>dag_concurrency</strong>:</p><p>This parameter controls the maximum number of active DAG runs per DAG.</p><p>In simpler terms, it sets the limit on how many instances of a DAG can run concurrently.</p><p>dag_concurrency = 3</p><p>In this example, we set `dag_concurrency` to 3, meaning only three instances of any DAG can run simultaneously.</p><p>Adjust this value based on the resources available and the desired level of parallelism.</p><p>2. `<strong>parallelism`</strong>:</p><p>`<strong>parallelism</strong>` determines the number of task instances that can run concurrently across all active DAGs.</p><p>It acts as a global limit on the total number of task instances.</p><p>parallelism = 5</p><p>3. max_active_runs_per_dag:</p><p>This parameter restricts the maximum number of active DAG runs for each individual DAG.</p><p>max_active_runs_per_dag = 2</p><p>In this scenario, a specific DAG can have a maximum of two active runs simultaneously. This helps in preventing resource contention and ensures that a DAG doesn’t overwhelm the system by spawning an excessive number of runs.</p><p>Let’s consider a scenario where you have multiple DAGs with varying execution times and dependencies. You want to control the concurrency for each DAG, limit the total parallel task execution, and put a cap on the active runs for individual DAGs.<br>dag_concurrency = 3<br>parallelism = 5<br>max_active_runs_per_dag = 2</p><p>In this setup, each DAG can have a maximum of three concurrent runs (`dag_concurrency`). The total number of parallel task executions across all DAGs is restricted to five (`parallelism`). Additionally, each DAG is constrained to have no more than two active runs simultaneously (`max_active_runs_per_dag`).</p><ul><li>Conclusion:</li></ul><p>Configuring Airflow is a crucial step in optimizing its performance and resource utilization. Understanding parameters like `dag_concurrency`, `parallelism`, and `max_active_runs_per_dag` empowers you to fine-tune your Airflow instance to meet your workflow requirements efficiently.</p><p>Experiment with these parameters, monitor the behavior of your DAGs, and adjust the configuration based on the characteristics of your data pipelines. Finding the right balance ensures that your Airflow deployment operates smoothly and scales effectively with your workload.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95f62962cad1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineer Interview (TMC)]]></title>
            <link>https://medium.com/@tusharhatwar785/the-math-company-data-engineer-interview-odyssey-0ce2d21f4e81?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ce2d21f4e81</guid>
            <category><![CDATA[data-engineer-interview]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 10:38:27 GMT</pubDate>
            <atom:updated>2024-01-12T04:28:58.326Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>&gt; Round 1(</strong>Telephonic<strong>):</strong></h3><p>- Explain job change, aligning aspirations with growth opportunities.<br>- Why did you resign without an offer.<br>- Spark’s architecture, RDDs, transformations, actions, and execution models.<br>- Spark’s Jobs, Stages, Tasks, Partition for parallel processing.<br>- SQL command: DELETE, DROP, TRUNCATE, and their categorization.<br>- Python’s List (mutable) and Tuple (immutable), speed and memory consumption differences.<br>- Rating Python and SQL proficiency on a scale of 5.</p><h3><strong>&gt; Round 2 (Technical)</strong>:</h3><p>Round 2 was focused on technical acumen and problem-solving skills</p><p>Exploration Points:<br>1. Introduction &amp; Project Architecture:<br> — Current project, data sources, targets, table count, and team dynamics.<br>2. Airflow &amp; Data Retrieval:<br> — Airflow’s role in harmonizing data extraction from Oracle databases, orchestrating a seamless data flow.<br>3. Azure’s Secret Scope &amp; Queries:<br> — A deep dive into Azure’s Secret Scope.<br>4. Spark Concepts:<br> — groupByKey, reduceByKey, transformations, and actions, decoding their intricacies.<br>5. Transformation Dynamics:<br> — Difference between wide and narrow transformations.<br>6. Repartition vs. Coalesce:<br> — Differences, practices between repartitioning and coalescing data.<br>7. Optimizations &amp; Exceptions:<br> — Project optimizations and strategies for handling OutOfMemory (OOM) exceptions.</p><p>Problem-Solving Questions:<br>1. Customers with No Purchases:<br> SQL<br> SELECT customer_name<br> FROM customers<br> WHERE customer_id NOT IN (SELECT customer_id FROM sales);<br> <br>2. Top 2 Customers by Total Order Amount:<br> SQL<br> SELECT customer_id, SUM(order_amount) AS total_order_amount<br> FROM sales<br> GROUP BY customer_id<br> ORDER BY total_order_amount DESC<br> LIMIT 2;<br> <br>3. Discovering the Least Quantity Product:<br> SQL<br> SELECT product_name<br> FROM products<br> WHERE quantity = (SELECT MIN(quantity) FROM products);<br> <br>This round focused architecture, Azure Basics, transformations, SQL , and hands-on problem-solving prowess.</p><h3><strong>&gt; Round 3 (Fitment):</strong></h3><ul><li>Personal introduction and recent project specifics.<br>- Clarifying cloud utilization and the architecture for on-prem to cloud migration using Azure Data Factory (ADF).<br>- Airflow’s role in project workflows.<br>- Stressing the distinction between DevOps and Git, emphasizing DevOps’ pivotal role.<br>- Exploring additional cloud platforms and team composition.<br> <br>This round delved into project (architecture, cloud integration), tool significance (Airflow, DevOps vs. Git), diverse cloud platform exploration, and team dynamics.</li></ul><p>All The Best :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ce2d21f4e81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I passed the Databricks Certified Data Engineer Associate Certification]]></title>
            <link>https://medium.com/@tusharhatwar785/how-i-passed-the-databricks-certified-data-engineer-associate-certification-7a03f868bee6?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a03f868bee6</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Thu, 19 Oct 2023 07:04:33 GMT</pubDate>
            <atom:updated>2023-10-19T07:16:41.168Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Databricks Certified Data Engineer Associate" src="https://cdn-images-1.medium.com/max/889/1*NjtgjKAwcLZgGFKeGx-mrw.png" /><figcaption>My Certificate</figcaption></figure><p><strong>Introduction</strong>:<br>The Databricks Certified Data Engineer Associate certification is an examination that evaluates an individual’s proficiency with the Databricks Lakehouse platform.</p><p>This certification is intended for those comfortable writing code in SQL and Python/Scala.</p><p>As the Lakehouse platform is built on top of Apache Spark, a strong knowledge of Spark is essential. Candidates should be capable of building basic ETL (Extract, Transform, Load) pipelines using Spark’s structured streaming and batch processing, creating dashboards, and implementing best security practices.</p><p><strong>Exam Details</strong>:<br>- Number of Questions: 45<br>- Question Format: Multiple-choice questions with a single correct option.<br>- Duration: 90 minutes<br>- Passing Score: 70%<br>- Exam Fee: $200 (Retakes are allowed, but each attempt requires an additional $200 fee).<br>- Result: You will receive immediate pass/fail results on your computer screen. If you pass, you will receive your digital badge within 24 hours after the exam.<br>- Certification Validity: The certification is valid for two years.</p><p>Exam Registration:<br>To register for the certification exam, visit the official registration page at <a href="https://www.webassessor.com/databricks">Link</a>.</p><p><strong>Prerequisites and Learning Pathway</strong>:<br>If you lack prior knowledge of the Databricks Lakehouse platform, completing the “Databricks Lakehouse Fundamental Accreditation” course is highly recommended. This fundamental course overviews the Lakehouse platform’s capabilities and how it addresses common data engineering challenges.</p><h3><strong>Key Exam Topics</strong>:</h3><p>1. <strong>Databricks Lakehouse Platform</strong> — 24% (11/45)<br>2. <strong>ELT with Spark SQL and Python</strong> — 29% (13/45)<br>3. <strong>Incremental Data Processing</strong> — 22% (10/45)<br>4. <strong>Production Pipelines</strong> — 16% (7/45)<br>5. <strong>Data Governance</strong> — 9% (4/45)</p><p><strong>Preparation Resources</strong>:</p><ul><li>“Data Engineering with Databricks” course available on Databricks Academy.</li><li>Read Databricks <a href="https://docs.databricks.com/en/">documentation</a> and blogs.</li><li>I also used the Preparation course for the Databricks Data Engineer Associate certification exam (Versions 2 and 3) with hands-on training by <a href="https://www.udemy.com/user/derar-alhussein/"><strong>Derar Alhussein</strong></a></li><li>YouTube Channels/Videos that I referred to — <a href="https://www.youtube.com/@thedatamaster">The Data Master</a>, <a href="https://youtu.be/259z7481AMc?si=d7phw9XN7c6B0EEu">DE Questions Dumps</a></li><li>Once done with the course you can go through the Practice Exam available in the academy: Databricks Certified Data Engineer Associate Exam</li><li>Practice with a sample practice test: [<a href="https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DataEngineerAssociate.pdf">Practice Exam</a>]</li><li>You can also refer to Databricks Certified Data Engineer Associate Practice Exams by <a href="https://www.udemy.com/course/databricks-certified-data-engineer-associate-practice-tests/#instructor-1"><strong>Akhil R</strong></a></li></ul><p>I am planning to create a cheat sheet in the coming days that you can refer to to get more Clarity on the Exam.</p><p>You can connect with me on <a href="https://www.linkedin.com/in/hatwar-tushar/">LinkedIn</a> for any Query.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a03f868bee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database vs Datawarehouse vs Data Lake vs Delta Lake]]></title>
            <link>https://medium.com/@tusharhatwar785/database-vs-datawarehouse-vs-data-lake-vs-delta-lake-64ada6b3bdbe?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/64ada6b3bdbe</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Sat, 12 Aug 2023 17:08:20 GMT</pubDate>
            <atom:updated>2023-08-24T04:42:34.555Z</atom:updated>
            <content:encoded><![CDATA[<p>Evolution of Data Management: Databases, Data Warehouses, Data Lakes, Delta Lake</p><h3><strong>Introduction</strong></h3><p>In the digital age, data has emerged as the driving force behind innovation, insights, and progress across industries. Data, whether structured, unstructured, or semi-structured, powers decision-making and problem-solving.</p><p>This article explores the evolution of data management, encompassing databases, data warehouses, data lakes, the revolutionary Delta Lake.</p><p>The Dichotomy of Data: Structured, Unstructured, and Semi-Structured</p><figure><img alt="Structured, Unstructured, and Semi-Structured" src="https://cdn-images-1.medium.com/max/1024/0*OKlwktX0ddr113Ul.jpg" /></figure><p>Data comes in three key forms: structured, unstructured, and semi-structured.</p><p>Structured data adheres to predefined schemas, presented as tables with rows and columns.</p><p>Unstructured data lacks a rigid structure and includes sources like news articles and emails.</p><p>Semi-structured data lies in between, offering some organization but without strict schemas — examples include JSON, XML, and log files. These data varieties pose unique challenges and opportunities for data professionals.</p><p>Database: The Foundation of Structure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*Y6vnp-FRibI1BZZK.png" /></figure><p>Databases excel at managing structured data. Consider a company named ABC that employs tables to organize product information and customer transactions. These tables provide a structured framework for record-keeping. Renowned databases like Microsoft SQL Server, Oracle Database, and MySQL facilitate efficient data retrieval and querying.</p><p>Data Warehouses: Unlocking Analytical Insights</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*EZVrdDLFfVcySgFu.png" /></figure><p>Data warehouses specialize in analysis and reporting. They empower organizations to unearth insights from accumulated data. Continuing with ABC company, they might leverage their data warehouse to analyze customer behavior, payment preferences, and trends. This analytical power supports strategic decisions and personalized marketing strategies. The ETL (Extract, Transform, Load) process transfers data from databases to data warehouses, facilitating this transformation.</p><p>Data Lakes: Embracing Unstructured and Semi-Structured Potential</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*DKanEXLZSb2I85Cn.png" /></figure><p>Data lakes have risen as essential resources for storing structured, unstructured, and semi-structured data. Going beyond traditional storage, data lakes accommodate text, images, videos, and diverse file types. They also embrace semi-structured formats like JSON, enabling organizations to capture more diverse data sources. With a “store now, analyze later” approach, data lakes appeal to data scientists and AI applications.</p><p>Delta Lake: The Evolution Continues</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/0*HBzUNGarKqhgg1IO.png" /></figure><p>In the evolving data landscape, Delta Lake takes center stage. Built upon data lakes, Delta Lake introduces reliability to big data with ACID transactions and time travel capabilities. It bridges the gap between data lakes and data warehouses, optimizing the management of streaming and batch data. Delta Lake’s integration ensures data quality, reliability, and performance.</p><p>Accessing Insights: Navigating Data Realms</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*p1R29Yg_2TfkUE98.png" /></figure><p>Gaining insights from these storage paradigms involves distinct methods. Databases employ Structured Query Language (SQL), while data warehouses rely on business intelligence tools like Power BI. Data lakes offer diverse access approaches tailored to specific analysis needs. Delta Lake enhances these capabilities with ACID transactions and support for batch and stream processing.</p><p>Conclusion</p><p>The evolution of data management encompasses structured databases, analytical data warehouses, expansive data lakes, the transformative Delta Lake, and the handling of semi-structured data. Each platform addresses distinct challenges in the data-driven age. Understanding these paradigms, utilizing Delta Lake’s power, and effectively managing semi-structured data enable organizations to unlock the full potential of their data. Informed decisions, innovation, and progress become attainable in the dynamic landscape of modern data management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64ada6b3bdbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ETL Pipeline Using Databricks, AWS S3 & Snowflake]]></title>
            <link>https://medium.com/@tusharhatwar785/etl-pipeline-using-databricks-aws-s3-snowflake-d42718c0e08c?source=rss-3db603ff49fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d42718c0e08c</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tushar Hatwar]]></dc:creator>
            <pubDate>Fri, 30 Jun 2023 08:21:57 GMT</pubDate>
            <atom:updated>2023-07-31T05:48:25.751Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="ETL Pipeline" src="https://cdn-images-1.medium.com/max/1024/1*M6i0X9p_nRm9YYETegGh9A.gif" /></figure><p>The Olympic Games have always been a symbol of unity, and competition, and With the advancements in technology and the availability of vast amounts of data, analyzing Olympic datasets can provide valuable insights into athlete performance, historical trends, and more.</p><p>In this article, we will explore the process of building an Extract, Transform, Load (ETL) pipeline for an Olympic dataset, starting from storing the data in AWS S3, processing and transforming it in Databricks, and finally loading it into Snowflake for further analysis.</p><p><strong>Step 1: Storing the Olympic Dataset in AWS S3</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*w6RlVPe0j68tRDfq2kI_NA.png" /><figcaption>AWS S3 for Data Storage</figcaption></figure><p>To begin, we need to ensure that the Olympic dataset is stored in an accessible and scalable storage solution. Amazon Simple Storage Service (S3) is a highly reliable and cost-effective option for this purpose. You can upload the Olympic dataset files to an S3 bucket, organizing them in a suitable folder structure.</p><p><strong>Step 2: Extracting and Transforming the Data in Databricks</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*1WfX06o4Wzvbi6qSOOQarQ.png" /><figcaption>Databricks for Processing &amp; Transformations</figcaption></figure><p><strong>Databricks </strong>provides a powerful and collaborative environment for big data processing and analytics. You can create a Databricks cluster and connect it to your S3 bucket containing the Olympic dataset. Databricks supports various programming languages such as Python, Scala, and R, allowing you to choose the one that best suits your needs.</p><p><strong>Step 3: Dumping Processed Data into Snowflake</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fPSDaZhq-MhcRIrY.png" /><figcaption>Snowflake for Data Analysis</figcaption></figure><p>Snowflake is a cloud-based data warehousing platform known for its scalability, performance, and ease of use. It provides a SQL-based interface for querying and analyzing data stored in its clusters. To load the processed Olympic dataset into Snowflake, you’ll need to establish a connection between Databricks and Snowflake.</p><p><strong>Step 4: Analyzing the Olympic Dataset in Snowflake</strong></p><p>With the Olympic dataset successfully loaded into Snowflake, you can now unleash the power of Snowflake’s SQL capabilities to perform complex analytical queries, generate insights, and visualize the data. Snowflake’s architecture allows for parallel processing, enabling high-performance analytics even on large datasets</p><p>Follow my <a href="https://github.com/tushar-hatwar/Olympic_Dataset_Analysis">Github</a> Repo to gain more insights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d42718c0e08c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>