<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bharath Raavi on Medium]]></title>
        <description><![CDATA[Stories by Bharath Raavi on Medium]]></description>
        <link>https://medium.com/@braavi24?source=rss-1e489877838f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8-fniTeUykGS6Rxqa1tEsg.jpeg</url>
            <title>Stories by Bharath Raavi on Medium</title>
            <link>https://medium.com/@braavi24?source=rss-1e489877838f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:55:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@braavi24/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Essential Spark Optimization Techniques for Efficient Big Data Processing]]></title>
            <link>https://medium.com/@braavi24/essential-spark-optimization-techniques-for-efficient-big-data-processing-73e238130680?source=rss-1e489877838f------2</link>
            <guid isPermaLink="false">https://medium.com/p/73e238130680</guid>
            <dc:creator><![CDATA[Bharath Raavi]]></dc:creator>
            <pubDate>Sat, 29 Mar 2025 12:16:55 GMT</pubDate>
            <atom:updated>2025-03-29T12:16:55.662Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Spark is a powerful, open-source framework for big data processing, capable of handling a variety of workloads from large-scale data processing to machine learning. However, despite its power, Spark can face performance issues and inefficient resource utilization if not optimized properly. To maximize <strong>Spark </strong>potential, it’s essential to implement optimization techniques that speed up execution, reduce costs, and improve scalability.</p><p>In this article, I will discuss key Spark optimization techniques that every big data engineer should know, and I’ll provide practical examples of how to implement them.</p><p><strong>1. Partitioning: Organizing Data for Efficient Access:</strong></p><p>Partitioning is the practice of dividing large datasets into smaller, manageable parts based on specific columns (e.g., date, region). This can significantly improve query performance by limiting the data that Spark needs to read and process during a job.<br><strong>When to Use:</strong> When filtering on partition columns (e.g., time, region). When working with massive datasets, where partition pruning can be applied.</p><p><strong>For Example:</strong> df.write.partitionBy(“date”).parquet(“/path”)</p><p>In this example, data is partitioned by the date column, which ensures that only relevant partitions are read during subsequent queries or processing.</p><p><strong>2. Bucketing: Optimizing Joins and Aggregations:</strong></p><p>Bucketing distributes data into a fixed number of buckets using hash functions, allowing Spark to perform joins and aggregations more efficiently.<br><strong>When to Use:</strong> When performing frequent joins or aggregations on specific columns.</p><p><strong>For Example:</strong> df.write.bucketBy(100, “user_id”).saveAsTable(“bucketed_table”)</p><p>Here, data is distributed into 100 buckets based on the user_id column, improving the speed of joins and aggregations involving that column.</p><p><strong>3. Broadcast Join: Minimizing Shuffle Costs</strong></p><p>Broadcast join sends small tables to all executors, minimizing network I/O and shuffle operations, leading to better performance.<br><strong>When to Use:</strong> When joining small tables (typically &lt;10MB) with large tables.</p><p><strong>For Example:</strong> spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, “100MB”)</p><p>This setting allows Spark to automatically broadcast smaller tables during joins, improving performance by avoiding shuffle operations.</p><p><strong>4. Repartition vs. Coalesce: Managing Data Distribution</strong></p><p>repartition() increases the number of partitions and triggers a full shuffle, while coalesce() reduces the number of partitions without a full shuffle.<br> <strong>When to Use:</strong> Use repartition() when balancing data across partitions. Use coalesce() when reducing partitions efficiently without triggering a shuffle.</p><p><strong>For Example: (1)</strong>df.repartition(200) #Increases partitions . (2) df.coalesce(10) # Reduces partitions efficiently</p><p>In this example, repartition() increases the number of partitions to 200, while coalesce() reduces the number to 10, avoiding a full shuffle for efficiency.</p><p><strong>5. Caching &amp; Persistence: Storing Intermediate Results</strong></p><p>Caching and persisting datasets in memory or on disk can speed up repetitive operations by reducing the need for redundant computations.<br> <strong>When to Use:</strong> When datasets are reused across multiple stages or queries.</p><p><strong>For Example: (1)</strong>df.cache(). (2)df.persist(StorageLevel.MEMORY_AND_DISK)</p><p>Here, the dataset is cached in memory, and if memory is insufficient, it is stored on disk, reducing redundant computations.</p><p><strong>6. Using Parquet + Compression for Efficient Storage</strong></p><p>Parquet is a columnar storage format that enables efficient compression, reducing storage space and improving query speed.<br><strong>When to Use:</strong> When working with structured data and performing analytical queries.</p><p><strong>For Example: </strong>df.write.option(“compression”, “zstd”).parquet(“/path”)</p><p>This example writes the dataset to Parquet format with zstd compression, which provides efficient storage and faster query execution.</p><p><strong>7. Adaptive Query Execution (AQE): Runtime Optimization</strong></p><p>AQE dynamically optimizes query execution plans based on runtime statistics, allowing Spark to adjust its execution strategy based on data distribution.<br> <strong>When to Use:</strong> When working with Spark 3.0+ for automatic performance optimizations.</p><p><strong>For Example: </strong>spark.sql(“SET spark.sql.adaptive.enabled=true”)</p><p>This setting enables AQE, allowing Spark to optimize shuffle partitions and handle skewed data more efficiently at runtime.</p><p><strong>8. Dynamic Partition Pruning: Skipping Unnecessary Data</strong></p><p>Dynamic partition pruning allows Spark to skip irrelevant partitions during join operations, reducing unnecessary computation.</p><p><strong>When to Use:</strong> When joining partitioned tables, such as fact-dimension joins.</p><p><strong>For Example: </strong>spark.sql(“SET spark.sql.optimizer.dynamicPartitionPruning.enabled=true”)</p><p>This enables dynamic partition pruning, allowing Spark to optimize the data it reads during join operations.</p><p><strong>9. Skew Join Handling: Balancing Uneven Data Distribution</strong></p><p>Skew join handling helps mitigate issues caused by uneven data distribution, particularly when certain keys (e.g., user_id) have disproportionately large amounts of data.<br><strong>When to Use:</strong> When dealing with skewed keys in your dataset.</p><p><strong>For Example: </strong>spark.conf.set(“spark.sql.adaptive.skewJoin.enabled”, “true”)</p><p>This setting helps balance the data distribution during joins, improving performance when dealing with skewed keys.</p><p><strong>10. Z-Ordering: Optimizing Multi-Column Filtering</strong></p><p>Z-Ordering optimizes the layout of data on disk by colocating related data, improving the efficiency of queries that filter on multiple columns.<br> <strong>When to Use:</strong> When performing range queries on multiple columns (e.g., time, user segments).</p><p><strong>For Example:</strong> OPTIMIZE delta_table ZORDER BY (timestamp, user_id)</p><p>This command optimizes the delta_table by colocating the data based on timestamp and user_id, improving query performance for range queries involving both columns.</p><p><strong>Reference Table:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_7xDlVWabsqX1aIcLpMxA.jpeg" /></figure><p><strong>Conclusion:</strong></p><p>Optimizing Spark jobs is essential for maximizing performance, reducing costs, and improving scalability. By incorporating these techniques — partitioning, bucketing, broadcast joins, caching, and more — you can significantly enhance the efficiency of your Spark applications. Whether you’re processing large-scale ETL pipelines or running complex machine learning workflows, these strategies will help you unlock the full potential of Spark.</p><p>If you know something new or other than this please feel free to drop a comment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*SIF6ipnsWPBCXcFM.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73e238130680" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My First Steps into dbt(Data build tool). Journey from Curiosity to Clarity]]></title>
            <link>https://medium.com/@braavi24/my-first-steps-into-dbt-data-build-tool-journey-from-curiosity-to-clarity-5cbcd1cf2521?source=rss-1e489877838f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cbcd1cf2521</guid>
            <dc:creator><![CDATA[Bharath Raavi]]></dc:creator>
            <pubDate>Fri, 07 Mar 2025 14:56:54 GMT</pubDate>
            <atom:updated>2025-03-24T16:50:12.310Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re like me someone diving into the world of data engineering, constantly hearing about tools like dbt but never actually using them.</p><p>Then welcome to my journey. Over the past few days, I’ve been exploring dbt (Data Build Tool), trying to understand what makes it such a game-changer in the modern data stack.</p><h3>What is dbt:</h3><p>Before I even touched the tool, I wanted to understand its purpose. <strong>dbt is not an ETL tool.</strong> It doesn’t extract or load data. Instead, it focuses purely on the <strong>T (Transform) in ELT</strong> allowing you to clean, structure, and optimize data <strong>inside your warehouse using SQL</strong>.</p><p>Think of dbt as <strong>“SQL on steroids.”</strong> It takes your SQL transformations and makes them:<br><strong>Modular: </strong>Break down transformations into smaller, reusable pieces.<br><strong>Version Controlled</strong> : Store everything in Git, making tracking changes easy.<br><strong>Testable: </strong>Run data quality checks before data reaches reports.<br><strong>Automated: </strong>Schedule transformations to run without manual intervention.<br><strong>Documented</strong> : Generate <strong>self-updating documentation and lineage graphs</strong>.</p><p>The idea is simple: Instead of writing <strong>ad-hoc SQL queries in BI tools</strong>, dbt allows you to manage transformations like <strong>code - </strong>scalable, maintainable, and production ready.</p><h3>Installing dbt on Your Local Computer</h3><p>To get started with dbt on your local machine, follow these steps:</p><h3>1. Install Python and dbt</h3><p>Since dbt runs on Python, ensure you have Python installed. You can check by running:</p><pre>python --version</pre><p>If not installed, download and install it from <a href="https://www.python.org/downloads/">Python’s official website</a>.</p><p>Then, install dbt using pip:</p><pre>pip install dbt-core dbt-postgres  # or dbt-snowflake/dbt-bigquery, depending on your database</pre><h3>2. Set Up a dbt Project</h3><p>Once dbt is installed, create your first dbt project:</p><pre>dbt init my_first_project</pre><p>This will create a directory with the necessary folder structure and files for your project.</p><h3>3. Configure Your Database Connection</h3><p>You’ll need to configure dbt to connect to your database. Navigate to the my_first_project directory and update the profiles.yml file (located in ~/.dbt/profiles.yml on Mac/Linux or %USERPROFILE%\.dbt\profiles.yml on Windows) with your database credentials.</p><p>For example, if you’re using <strong>PostgreSQL</strong>, your configuration might look like this:</p><pre>my_first_project:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: postgres<br>      host: localhost<br>      user: my_username<br>      password: my_password<br>      port: 5432<br>      dbname: my_database<br>      schema: public</pre><h3>4. Create and Run dbt Models</h3><p>In the models folder inside your project, create a new SQL file, such as my_first_model.sql, and write a simple SQL transformation:</p><pre>SELECT id, name, created_at<br>FROM my_table<br>WHERE created_at &gt;= &#39;2024-01-01&#39;;</pre><p>Run your transformation using:</p><pre>dbt run</pre><h3>5. Test and Document Your Models</h3><p>To test your data models, add a test in my_first_model.yml:</p><pre>models:<br>  - name: my_first_model<br>    description: &quot;This model selects recent records from my_table.&quot;<br>    tests:<br>      - unique:<br>          column: id</pre><p>Run the test command:</p><pre>dbt test</pre><p>To generate documentation for your dbt project, run:</p><pre>dbt docs generate<br>dbt docs serve  </pre><h3>Advanced Features of dbt</h3><p>Once you’re comfortable with the basics, dbt offers advanced features that enhance your workflow:</p><ol><li><strong>Jinja and Macros: </strong>dbt allows you to write reusable SQL using Jinja templating, making queries dynamic.</li><li><strong>Incremental Models: </strong>Process only new or updated data instead of running transformations on the entire dataset.</li><li><strong>Snapshots: </strong>Track changes in data over time, useful for historical analysis.</li><li><strong>Materializations</strong> : Control whether dbt models are views, tables, or incremental tables for efficiency.</li><li><strong>dbt Cloud vs. dbt Core</strong> : dbt offers a managed cloud version for better collaboration, scheduling, and monitoring.</li></ol><h3>Conclusion:</h3><p>DBT simplifies the transformation process, making it <strong>faster, scalable, and more reliable</strong>.</p><p>If you’re new to data engineering and want to work with modern data tools, dbt is a great place to start. It’s beginner-friendly, open-source, and widely used in the industry.</p><p>Have you used dbt before? Let’s discuss in the comments!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0bcZ7QNHrG2igs5E" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cbcd1cf2521" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Players in the Data Processing. 3FS & SmallPond data frame from deepseek.]]></title>
            <link>https://medium.com/@braavi24/new-players-in-the-data-processing-3fs-smallpond-data-frame-from-deepseek-6e6616e28d54?source=rss-1e489877838f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e6616e28d54</guid>
            <dc:creator><![CDATA[Bharath Raavi]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 01:01:41 GMT</pubDate>
            <atom:updated>2025-03-03T01:02:33.887Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing 3FS &amp; SmallPond: What They Mean for Data Processing</h3><p>On <strong>February 28, 2025</strong>, DeepSeek caught the attention of the open-source community by launching two new tools. <strong>Fire-Flyer File System (3FS)</strong> and <strong>SmallPond Framework</strong>. These tools aim to tackle the increasing demands of modern data processing, particularly in AI training, inference, and large-scale analytics. While it’s still early days, they promise to bring fresh improvements to how we handle massive datasets.</p><h3>3FS (<strong>Fire-Flyer File System</strong>):</h3><p>If you work with large-scale data, you know storage is the backbone of any pipeline. <strong>3FS</strong> is built to provide high-speed access to data while maintaining strong consistency and scalability. Here’s why it stands out:</p><ul><li><strong>Blazing Fast Performance: </strong>Achieves <strong>6.6 TiB/s aggregate read throughput</strong> in a <strong>180-node cluster</strong> and delivers <strong>40+ GiB/s peak throughput per client node</strong>.</li><li><strong>Scalability: </strong>Combines the power of thousands of SSDs with high-bandwidth storage nodes, making it easy to scale from terabytes to petabytes.</li><li><strong>Strong Consistency</strong> : Uses <strong>Chain Replication with Apportioned Queries (CRAQ)</strong> to ensure reliable data integrity.</li><li><strong>Familiar Interface: </strong>Works like a standard file system but is backed by a high-performance <strong>key-value store</strong> (like <strong>FoundationDB</strong>), so there’s no need to learn a new API.</li></ul><h3>SmallPond: A Lightweight Data Processing Framework</h3><p>Alongside <strong>3FS</strong>, DeepSeek also introduced <strong>SmallPond</strong>, a simple but powerful data processing framework designed for speed and scalability. Here’s why it’s worth noting:</p><ul><li><strong>Lightning-Fast Performance</strong> — Uses <strong>DuckDB</strong> under the hood and sorted <strong>110.5 TiB of data across 8,192 partitions in just 30 minutes and 14 seconds</strong>, averaging <strong>3.66 TiB/min</strong>.</li><li><strong>Scalability Without Complexity</strong> — Leverages <strong>3FS</strong> for intermediate storage, making it easier to process petabyte-scale datasets.</li><li><strong>Easy to Use</strong> — No need for complex setup or long-running services — just install and start working. The <strong>Python API</strong> is designed to be intuitive.</li><li><strong>Optimized Data Processing</strong> — Uses a <strong>two-phase sorting approach</strong> to handle large datasets efficiently.</li></ul><h3>The Big Question is will 3FS &amp; SmallPond Make a Real Impact ?</h3><p>3FS and SmallPond look promising for data storage and processing, but we don’t know yet how much of a difference they will make in real-world use. They could improve ETL pipelines, AI training, and real-time analytics, making it easier to work with huge datasets or they might just be useful in specific cases. We’ll have to wait and see.</p><p>If you’re working in <strong>data engineering, AI, or large-scale analytics</strong>, these are definitely tools worth keeping an eye on.</p><p><strong>Source:</strong></p><p>Git Link for 3Fs:<a href="https://github.com/deepseek-ai/3FS"> https://github.com/deepseek-ai/3FS</a></p><p>Git Link For Smallpond: <a href="https://github.com/deepseek-ai/smallpond">https://github.com/deepseek-ai/smallpond</a></p><p><a href="https://www.analyticsvidhya.com/blog/2025/02/deepseek-3fs-and-smallpond-framework/">DeepSeek #OpenSourceWeek Day 5: Launch of 3FS and Smallpond Framework</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x5JxLKxDj_Sxnvbc.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e6616e28d54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Iceberg Scalable Table Formats for Big Data]]></title>
            <link>https://medium.com/@braavi24/apache-iceberg-scalable-table-formats-for-big-data-c3fc8dc7ada4?source=rss-1e489877838f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3fc8dc7ada4</guid>
            <dc:creator><![CDATA[Bharath Raavi]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 16:52:52 GMT</pubDate>
            <atom:updated>2025-02-24T16:52:52.678Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of big data analytics, efficient table formats play a crucial role in managing large-scale datasets. Apache Iceberg has emerged as a powerful solution, offering better scalability, reliability, and performance compared to traditional table formats. But how does it compare to its competitors, like <strong>Delta Lake</strong> and <strong>Apache Hudi</strong>? Let’s dive deep into <strong>Apache Iceberg</strong> and see why it is gaining popularity in modern data architectures.</p><h3>What is Apache Iceberg?</h3><p>Apache Iceberg is an <strong>open-source table format</strong> designed for <strong>huge datasets</strong> in distributed computing environments. It was originally developed at <strong>Netflix</strong> to address the challenges of large-scale data processing and has since been widely adopted by companies dealing with petabyte-scale analytics.</p><p>Unlike traditional Hive tables, Iceberg provides <strong>schema evolution, time travel, ACID transactions, and efficient querying</strong> while reducing metadata overhead. It integrates seamlessly with modern big data tools like <strong>Apache Spark, Trino, Flink, Presto, and Snowflake</strong>.</p><h3>Key Features of Apache Iceberg</h3><p><strong>Schema Evolution</strong>: Modify table schemas dynamically without rewriting data.<br><strong>Hidden Partitioning</strong>: Automates partitioning logic, eliminating the need for manual partition management.<br><strong>ACID Transactions</strong>: Ensures <strong>atomicity</strong>, <strong>consistency</strong>, <strong>isolation</strong>, and <strong>durability</strong> for concurrent read/write operations.<br><strong>Time Travel</strong>: Query historical snapshots of data for auditing and debugging.<br><strong>Efficient File Management</strong>: Uses metadata layers to reduce storage footprint and improve query performance.<br><strong>Open Standard</strong>: Works across multiple processing engines like Spark, Flink, and Trino without vendor lock-in.</p><h3>Apache Iceberg vs. Delta Lake vs. Apache Hudi</h3><p>When choosing a table format for big data, it’s important to understand how Iceberg stacks up against its key competitors, <strong>Delta Lake</strong> (from Databricks) and <strong>Apache Hudi</strong> (from Uber).</p><h3>1. Architecture &amp; Design Philosophy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*9zaCYait1EficQzJ8b0qTg.png" /></figure><p><strong>Iceberg focuses on large-scale, analytic workloads</strong> with optimized metadata and query performance.</p><p><strong>Delta Lake is deeply integrated with Databricks</strong> and is ideal for batch processing and analytics.</p><p><strong>Hudi specializes in incremental updates</strong> and is best suited for streaming and CDC (Change Data Capture) use cases.</p><h3>2. Performance &amp; Scalability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*3GoohkB9QPO7ZJDBMDtuIg.png" /></figure><p><strong>Iceberg is more scalable in terms of metadata management</strong>, as Delta Lake relies on a single JSON log file that grows over time. <strong>Hudi is better for frequent, incremental updates</strong>, but it lacks some of Iceberg’s analytical optimizations.</p><h3>3. Compatibility &amp; Ecosystem Support</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*m5GKR_Ok0SO8TjGVXpW3CA.png" /></figure><p><strong>Iceberg provides broader support across different query engines and cloud storage solutions</strong>. <strong>Delta Lake is tightly integrated with Databricks</strong>, making it a great choice for Databricks users but less flexible elsewhere. <strong>Hudi is best for real-time use cases</strong> but lacks robust analytical optimizations</p><h3>When to Use Apache Iceberg?</h3><p>Apache Iceberg is a great choice if you need:</p><p>A <strong>scalable and open table format</strong> for big data analytics.<br><strong>Efficient schema evolution</strong> without breaking existing queries.<br><strong>ACID transactions</strong> for safe and concurrent data modifications.<br><strong>Time travel and snapshot isolation</strong> for version control and auditing.<br><strong>Better performance on large datasets</strong> with optimized partitioning and metadata management.</p><p><strong>Conclusion:</strong></p><p>As a student exploring big data technologies, learning about Apache Iceberg has been eye-opening. Compared to Delta Lake and Apache Hudi, Iceberg stands out with its efficient metadata management, seamless schema evolution, and broad compatibility across multiple processing engines. These features make it a great choice for handling large-scale analytical workloads in modern data lakes.</p><p>Understanding the differences between these table formats helps in making informed decisions when working on data projects. Iceberg’s flexibility and open-source nature make it an exciting technology to experiment with, and I look forward to implementing it in real-world scenarios.</p><p>What are your thoughts on Apache Iceberg? Let’s discuss!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*gljKIeijyG6lHeKW.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3fc8dc7ada4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Spotify Music Analytics Pipeline with Azure]]></title>
            <link>https://medium.com/@braavi24/how-i-built-a-spotify-music-analytics-pipeline-with-azure-960526a3b527?source=rss-1e489877838f------2</link>
            <guid isPermaLink="false">https://medium.com/p/960526a3b527</guid>
            <dc:creator><![CDATA[Bharath Raavi]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 00:01:02 GMT</pubDate>
            <atom:updated>2025-02-24T00:01:02.227Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever wondered how your Spotify listening habits could turn into real data insights? As a data engineer &amp; analytics enthusiast, I took on the challenge of building an end-to-end Spotify Music Analytics Pipeline using Azure Cloud Services.</p><p><strong>In this article, I’ll walk you through:</strong><br>Extracting data from the Spotify API<br>Storing raw data in Azure Blob Storage<br>Processing it using Databricks &amp; Azure Data Factory<br>Loading transformed data into Azure Synapse Analytics<br>Creating a Power BI dashboard for visualization</p><p>If you’re into <strong>data engineering, analytics, or cloud computing</strong>, this project is a fun way to level up your skills.</p><h3>The Tech Stack</h3><p>To make this happen, I used:</p><p>🔹<strong>Azure Blob Storage</strong> to store raw music data<br>🔹 <strong>Azure Data Factory</strong> to automate data movement<br>🔹 <strong>Databricks</strong> for cleaning and analyzing large datasets<br>🔹 <strong>Azure Synapse Analytics</strong> for SQL-based querying<br>🔹 <strong>Power BI</strong> for stunning visual dashboards</p><p>The beauty of this project is that it <strong>mimics real-world data engineering pipelines</strong>, making it a great addition to my portfolio.</p><h3>What Insights Did I Discover?</h3><p>After setting up the pipeline, I was able to analyze my music trends. Some <strong>fun insights</strong> I found:</p><p><strong>My top 3 most played artists surprised me!</strong> (I thought I listened to different genres, but my data said otherwise 😅)<br><strong>My music preferences change dramatically by season!</strong><br><strong>Most of my listening happens late at night — definitely a night owl!</strong></p><h3>Challenges I Faced</h3><p>No project is without hurdles! Here are some struggles I faced:</p><p><strong>Spotify API limits</strong> — I had to carefully handle request limits to get enough data<br><strong>Azure costs</strong> — Cloud services can be expensive, so optimizing storage and compute usage was crucial<br>Each challenge was a <strong>learning experience</strong>, and overcoming them made the project even more rewarding!</p><h3>Final Thoughts</h3><p>Working on this project <strong>reinforced my passion for data engineering and cloud computing</strong>. It’s incredible how much insight can be unlocked from something as simple as your <strong>music history</strong>!</p><p>Git link: <a href="https://github.com/raavibharath/Spotify_ETL_pipeline_using_Azure">https://github.com/raavibharath/Spotify_ETL_pipeline_using_Azure</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YjTgWSoe3hw76qzhBB3QRQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=960526a3b527" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>