<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SoundaryaSenthilkumar on Medium]]></title>
        <description><![CDATA[Stories by SoundaryaSenthilkumar on Medium]]></description>
        <link>https://medium.com/@soundaryasenthil510?source=rss-8fc964e6f7df------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by SoundaryaSenthilkumar on Medium</title>
            <link>https://medium.com/@soundaryasenthil510?source=rss-8fc964e6f7df------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:10:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@soundaryasenthil510/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Two Databricks concepts that sound alike but solve different problems ]]></title>
            <link>https://medium.com/@soundaryasenthil510/two-databricks-concepts-that-sound-alike-but-solve-different-problems-12801cf86ede?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/12801cf86ede</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 10:21:01 GMT</pubDate>
            <atom:updated>2026-10-17T10:21:01.667Z</atom:updated>
            <content:encoded><![CDATA[<p>1️⃣ What is Lakeflow Connect?</p><p>Databricks’ managed ingestion service. It has built-in connectors for SaaS apps (Salesforce, Workday) and databases (SQL Server, MySQL, PostgreSQL, Oracle).</p><p>🔹 CDC-based incremental loads into Unity Catalog streaming tables</p><p>🔹 Serverless pipelines with automatic schema evolution</p><p>🔹 Replaces custom ingestion code</p><p>👉 Data is copied into the lakehouse</p><p>2️⃣ What is Lakehouse Federation?</p><p>A query federation feature that lets you query external databases from Databricks without moving the data.</p><p>🔹 Works with MySQL, PostgreSQL, SQL Server, Snowflake, Redshift, BigQuery, and more</p><p>🔹 You create a Unity Catalog connection and a foreign catalog, then query it with normal SQL</p><p>🔹 Governance, access control, and lineage come from Unity Catalog</p><p>🔹 Query pushdown sends filters and aggregations to the source system</p><p>🔹 Limitation: read-only, and performance depends on the source system</p><p>👉 Data stays in place</p><p>💡 When to use which?</p><ul><li>Need history, scale, and heavy transformations → Lakeflow Connect (ingest)</li><li>• Need quick, ad hoc access or a proof of concept → Lakehouse Federation (query in place)</li><li>• Files landing in cloud storage → Auto Loader</li></ul><p>Interview tip: The core difference is copy vs. query in place. Say it in one line and you’ve answered half the question.</p><p>More questions coming. Which topic should I cover next? 👇</p><p>#DataEngineering #Databricks #LakeflowConnect #LakehouseFederation #UnityCatalog #InterviewPreparation #Azure #PySpark</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12801cf86ede" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Microsoft Fabric Messy? Or Are We Witnessing a Shift in How Data Platforms Are Built?]]></title>
            <link>https://medium.com/@soundaryasenthil510/is-microsoft-fabric-messy-or-are-we-witnessing-a-shift-in-how-data-platforms-are-built-7816fd46bde8?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/7816fd46bde8</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 09:35:55 GMT</pubDate>
            <atom:updated>2026-06-14T09:35:55.296Z</atom:updated>
            <content:encoded><![CDATA[<p>For years, organizations built data platforms by combining multiple specialized services. A typical Azure-based architecture might include Azure Data Factory for orchestration, Azure Data Lake Storage for storage, Databricks for data engineering, Synapse or Snowflake for warehousing, and Power BI for reporting. Each service excels in its own area, but it also comes with separate administration, security configurations, monitoring, cost management, and integration requirements. As platforms scale, the effort required to manage the ecosystem often grows alongside the data.</p><p>Microsoft Fabric takes a different approach. Rather than offering another ETL tool or another warehouse, Fabric attempts to unify the entire analytics landscape into a single SaaS platform. Data integration, engineering, warehousing, real-time analytics, data science, and business intelligence are all brought together under one experience. This architectural decision is what makes Fabric both compelling and, at first glance, complex.</p><p>At the center of this architecture is OneLake. Unlike traditional approaches where different teams maintain separate storage accounts and copies of datasets, OneLake acts as a centralized data foundation. The idea is simple: data should be stored once and made accessible to multiple workloads. Whether a data engineer is running Spark transformations, an analyst is querying data through SQL, or a business user is building a Power BI dashboard, they can all work from the same underlying data layer. This reduces duplication and encourages a more unified approach to data management.</p><p>One of the most interesting innovations within Fabric is OneLake Shortcuts. In many organizations, moving data has become a default solution. Data is copied from one storage account to another, replicated across environments, and duplicated for different teams. Over time, this creates additional costs, governance challenges, and operational complexity. OneLake Shortcuts challenge that pattern by allowing organizations to reference data where it already exists instead of physically copying it. Existing data in ADLS Gen2, Amazon S3, or even other Fabric workspaces can be accessed through shortcuts. While this may seem like a small feature, it has the potential to significantly reduce data movement and simplify architecture at scale.</p><p>Fabric also embraces the Lakehouse concept, which has become increasingly popular across the industry. Historically, organizations maintained separate data lakes and data warehouses because each solved different problems. Data lakes provided flexibility and scalability for raw data, while warehouses delivered structured analytics and performance. The Lakehouse model attempts to bridge those worlds by allowing multiple workloads to operate on the same data foundation. In Fabric, this means engineers, analysts, and business users can work more closely together without constantly moving data between systems.</p><p>This is where the comparison with platforms like Databricks and Snowflake becomes interesting. Databricks is widely regarded as one of the strongest platforms for advanced data engineering, large-scale Spark processing, and AI workloads. Snowflake has built a reputation around simplicity and high-performance analytics. Fabric’s ambition is different. Rather than focusing on a single workload, it aims to create a connected ecosystem where the boundaries between engineering, analytics, warehousing, and reporting become less visible.</p><p>Of course, that ambition comes with trade-offs. When a platform combines so many capabilities, the experience can initially feel broader than users expect. Someone familiar with dedicated tools may wonder why there are multiple ways to interact with data, multiple experiences within the same workspace, or several options for solving similar problems. However, this complexity often reflects the platform’s scope rather than poor design. Fabric is attempting to serve data engineers, analysts, data scientists, and business users from the same environment.</p><p>Perhaps the biggest advantage of Fabric is not any individual feature, but the reduction of friction between teams. Data engineers no longer need to think only about pipelines, analysts no longer need to think only about reports, and business users no longer need to wait for data to move across multiple platforms. Everything is designed to operate on a shared foundation.</p><p>The more I explore Fabric, the more I see it not as a competitor to a single product, but as Microsoft’s attempt to rethink how modern data platforms should be structured. Instead of connecting best-of-breed services together, Fabric asks a different question: what if the entire analytics ecosystem was designed to work together from the beginning?</p><p>Whether that vision becomes the standard for enterprise analytics remains to be seen. But one thing is certain – Microsoft Fabric has changed the conversation from choosing the best individual tool to rethinking the architecture of the entire data platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*_F7OWt_NQ6Dt4uybzcNXLQ@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7816fd46bde8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Live Tables (DLT) and Lakehouse Jobs]]></title>
            <link>https://medium.com/@soundaryasenthil510/1%EF%B8%8F%E2%83%A3-delta-live-tables-dlt-bd7cc8f36480?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd7cc8f36480</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Sat, 03 Jan 2026 03:24:15 GMT</pubDate>
            <atom:updated>2026-01-14T17:15:43.296Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>DLT:</strong></p><p>Delta Live Tables (DLT) is a managed data pipeline framework in Databricks designed specifically for data engineering pipelines.</p><p>It is built to create reliable, scalable, and maintainable pipelines with minimal operational effort.</p><p>The key idea behind DLT is declarative development:</p><p>•	You define what tables should exist</p><p>•	You define rules and transformations</p><p>•	Databricks decides how and when to run them</p><p>⸻</p><p><strong>How DLT works internally</strong></p><p>When you create a DLT pipeline:</p><p>1.	Databricks scans your DLT code</p><p>2.	It identifies all tables and their dependencies</p><p>3.	It automatically creates a Directed Acyclic Graph (DAG)</p><p>4.	It provisions clusters</p><p>5.	It executes tables in the correct order</p><p>6.	It continuously monitors data quality and failures</p><p>You never explicitly define task order or scheduling logic.</p><p>⸻</p><p><strong>Key components of DLT</strong></p><ol><li>Live Tables</li></ol><p>Live tables are tables managed by DLT. They are stored as Delta tables. DLT controls creation, updates, and refreshes. You do not manually create or drop these tables</p><p>These tables typically represent:</p><p>•	Bronze (raw data)</p><p>•	Silver (cleaned data)</p><p>•	Gold (aggregated/business data)</p><p>⸻</p><p>2. Declarative Pipeline Logic</p><p>In DLT, you do not write:</p><p>•	Cluster creation logic</p><p>•	Scheduling logic</p><p>•	Dependency logic</p><p>Instead, you declare transformations such as:</p><p>•	Read from a source</p><p>•	Apply filters and transformations</p><p>•	Output to another table</p><p>DLT figures out execution order automatically.</p><p>⸻</p><p>3. Built-in Data Quality (Expectations)</p><p>DLT allows you to define data quality rules directly on tables.</p><p>These rules can:</p><p>•	Log invalid records</p><p>•	Drop invalid records</p><p>•	Fail the entire pipeline</p><p>DLT also:</p><p>•	Tracks quality metrics</p><p>•	Displays them in the DLT UI</p><p>•	Helps detect data issues early</p><p>⸻</p><p>4. Streaming and Batch Unification</p><p>DLT supports:</p><p>•	Batch ingestion</p><p>•	Streaming ingestion</p><p>•	Incremental processing</p><p>The same pipeline code can run in:</p><p>•	Triggered mode (on demand or scheduled)</p><p>•	Continuous mode (always running)</p><p>Checkpointing and state management are fully handled by DLT.</p><p>⸻</p><p>5. Monitoring and Lineage</p><p>DLT automatically provides:</p><p>•	End-to-end data lineage</p><p>•	Table dependencies</p><p>•	Execution history</p><p>•	Data quality statistics</p><p>This makes debugging and auditing much easier.</p><p>⸻</p><p><strong>Where DLT is used in real projects</strong></p><p>DLT is mainly used for:</p><p>•	Enterprise-grade data pipelines</p><p>•	Bronze → Silver → Gold architectures</p><p>•	Streaming ingestion use cases</p><p>•	Incremental data processing</p><p>•	Data platforms with strong governance needs</p><p>⸻</p><p><strong>LAKEHOUSE JOBS:</strong></p><p>Lakehouse Jobs are workflow orchestration and scheduling mechanisms in Databricks.</p><p>They are not limited to data pipelines.</p><p>They are designed to run any type of workload.</p><p>The key idea behind Jobs is imperative execution:</p><p>•	You explicitly define what runs first</p><p>•	You explicitly define dependencies</p><p>•	You explicitly manage retries and failures</p><p>⸻</p><p><strong>How Lakehouse Jobs work internally</strong></p><p>When a Job runs:</p><p>1.	The scheduler triggers the job</p><p>2.	A cluster is created or reused</p><p>3.	Tasks run in the defined order</p><p>4.	Each task executes a notebook, script, or SQL</p><p>5.	Failures are handled based on your configuration</p><p>Jobs do not analyze your code to understand dependencies.</p><p>⸻</p><p><strong>Key components of Lakehouse Jobs</strong></p><ol><li>Tasks</li><li>A job consists of one or more tasks.</li><li>Each task can execute:</li></ol><p>•	A notebook</p><p>•	A Python script</p><p>•	SQL</p><p>•	JAR</p><p>•	A DLT pipeline</p><p>Dependencies between tasks are manually configured.</p><p>⸻</p><p>2. Manual Dependency Management</p><p>Unlike DLT, Jobs do not auto-detect dependencies.</p><p>You must explicitly specify:</p><p>•	Task A must complete before Task B</p><p>•	Retry logic</p><p>•	Timeout rules</p><p>This gives flexibility but increases maintenance.</p><p>⸻</p><p>3. Scheduling and Triggers</p><p>Jobs support:</p><p>•	Time-based scheduling (cron)</p><p>•	Manual triggers</p><p>•	Event-based triggers</p><p>This makes Jobs suitable for:</p><p>•	Daily batch jobs</p><p>•	Nightly refreshes</p><p>•	Periodic ML training</p><p>⸻</p><p>4. Error Handling and Retries</p><p>Jobs allow you to configure:</p><p>•	Number of retries</p><p>•	Alerting (email, webhook)</p><p>•	Failure behavior per task</p><p>Data quality and validation must be implemented manually.</p><p>⸻</p><p><strong>Where Jobs are used in real projects</strong></p><p>Lakehouse Jobs are commonly used for:</p><p>•	ML model training and scoring</p><p>•	Power BI or dashboard refresh</p><p>•	API-based workflows</p><p>•	One-time or ad-hoc scripts</p><p>•	Orchestrating non-tabular workloads</p><p>•	Triggering DLT pipelines</p><p>⸻</p><p><strong>Core Difference Between DLT and Jobs</strong></p><p>DLT focuses on what data pipelines should look like</p><p>Jobs focus on how workloads are executed</p><p>DLT manages:</p><p>•	Orchestration</p><p>•	Dependencies</p><p>•	Data quality</p><p>•	Lineage</p><p>Jobs manage:</p><p>•	Scheduling</p><p>•	Task execution</p><p>•	Retries</p><p>•	Workflow control</p><p>⸻</p><p><strong>How DLT and Jobs Work Together :</strong></p><p>In real-world Databricks projects:</p><p>•	DLT builds the data pipelines</p><p>•	Lakehouse Jobs schedule and trigger those pipelines</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*Qo6hOAtgNIP6omDZ7JEqrg@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd7cc8f36480" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Delta Table vs Delta Live Table in Databricks – Simplified!]]></title>
            <link>https://medium.com/@soundaryasenthil510/understanding-delta-table-vs-delta-live-table-in-databricks-simplified-443c1495e1e6?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/443c1495e1e6</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 06:33:31 GMT</pubDate>
            <atom:updated>2025-04-23T07:03:14.268Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TUFGjt0JjNSvnuwA4ujK8Q@2x.jpeg" /></figure><p>If you’re working with Databricks or learning about data engineering, you’ve probably heard of Delta Tables and Delta Live Tables (DLT). They sound similar, but they serve different purposes. Here’s a simple breakdown:</p><p>⸻</p><ol><li>Delta Table – Your Smart Data Storage</li></ol><p>A Delta Table is like a regular data table, but smarter!</p><p>It’s built on Delta Lake, which adds powerful features like:</p><p>•	ACID transactions – so your data stays consistent and reliable</p><p>•	Time travel – you can query older versions of your data</p><p>•	Schema enforcement – avoids messy or corrupt data</p><p>•	High performance – works great even with big data</p><p>You can read, write, update, and delete data easily – just like in a database, but on your data lake!</p><p>⸻</p><p>2. Delta Live Table (DLT) – Your Data Pipeline Assistant</p><p>A Delta Live Table is NOT just a table – it’s a framework to build data pipelines.</p><p>With DLT, you define how data should be transformed using SQL or Python, and Databricks takes care of:</p><p>•	Automatically running the pipeline (batch or streaming)</p><p>•	Creating the Delta Tables for you</p><p>•	Validating your data (with built-in quality checks)</p><p>•	Keeping the pipeline updated and monitored</p><p>Think of DLT as your automated kitchen, where you define the recipe (ETL logic), and it cooks and serves your data into Delta Tables reliably.</p><p>⸻</p><p>In short:</p><p>•	Delta Table = Where data lives</p><p>•	Delta Live Table = How data gets there (safely and reliably)</p><p>⸻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=443c1495e1e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[◾️DATALAKE & DELTALAKE]]></title>
            <link>https://medium.com/@soundaryasenthil510/%EF%B8%8Fdatalake-deltalake-273ea573823c?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/273ea573823c</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:14:02 GMT</pubDate>
            <atom:updated>2025-04-23T07:03:28.428Z</atom:updated>
            <content:encoded><![CDATA[<p>Here’s a step-by-step example of how you can combine Data Lake and Delta Lake:</p><p>📍Step 1: Set Up the Data Lake</p><p>•	Create a centralized storage repository using a distributed file system like Hadoop Distributed File System (HDFS) or cloud-based storage such as Amazon S3 or Azure Data Lake Storage.</p><p>•	Organize your Data Lake by creating folders or directories to store different types of data (e.g., raw, processed, curated).</p><p>•	Ingest raw data from various sources into the Data Lake without imposing any schema. This raw data may include CSV files, JSON documents, Parquet files, or any other format.</p><p>📍Step 2: Implement Delta Lake</p><p>•	Install and configure Delta Lake as an additional layer on top of your existing Data Lake infrastructure. Delta Lake can be deployed on-premises or in the cloud.</p><p>•	Configure Delta Lake to provide ACID transactions, schema enforcement, and time travel capabilities for data consistency and reliability.</p><p>•	Delta Lake integrates seamlessly with Apache Spark, allowing you to leverage Spark’s distributed processing capabilities for data manipulation and analysis.</p><p>📍Step 3: Ingest Data into Delta Lake</p><p>•	Use Apache Spark or other data processing frameworks to ingest raw data from the Data Lake into Delta Lake.</p><p>•	When ingesting data into Delta Lake, Delta Lake automatically enforces schema validation and performs optimizations such as file compaction and data indexing for improved query performance.</p><p>•	Delta Lake supports various data formats, including Parquet, Delta, and JSON, allowing you to ingest data in its original format or convert it to a preferred format for efficient storage and processing.</p><p>📍Step 4: Perform Data Transformation and Analysis</p><p>•	Once the data is ingested into Delta Lake, you can perform data transformation, cleansing, and enrichment using Apache Spark or other data processing tools.</p><p>•	Delta Lake’s transactional capabilities ensure that data modifications are atomic, consistent, isolated, and durable, enabling reliable data transformations without compromising data integrity.</p><p>•	Analyze the transformed data to derive insights, build machine learning models, or generate reports to support business decision-making.</p><p>📍Step 5: Ensure Data Quality and Governance</p><p>•	Implement data quality checks, validation rules, and monitoring mechanisms to ensure the integrity and quality of data stored in Delta Lake.</p><p>•	Establish data governance policies and practices to manage access control, data lineage, metadata management, and compliance requirements.</p><p>•	Delta Lake’s time travel feature allows you to track changes to the data over time and revert to previous versions if needed, enhancing data governance and auditability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*QpjkWsbrPALSyKF9t7wPBg@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=273ea573823c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Daily Doses]]></title>
            <link>https://medium.com/@soundaryasenthil510/daily-doses-e4ff32a03bc9?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4ff32a03bc9</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:11:19 GMT</pubDate>
            <atom:updated>2025-04-23T07:06:58.637Z</atom:updated>
            <content:encoded><![CDATA[<p>⭐️Here’s a step-by-step guide on how to write a PySpark query:</p><p>⚫️Initialize SparkSession:</p><p>. Start by importing the necessary libraries and initializing a SparkSession, which is the entry point to Spark functionality in PySpark.</p><p>⚫️Read Data:</p><p>. Use SparkSession’s `read` methods to load data into a DataFrame from different sources such as CSV files, databases, or other data formats.</p><p>.</p><p>⚫️Perform Data Manipulation:</p><p>. Use DataFrame methods to perform various data manipulation tasks such as filtering, selecting, joining, aggregating, and transforming data.</p><p>.</p><p>⚫️Execute Actions:</p><p>. Use DataFrame actions to trigger the execution of the query and retrieve the results.</p><p>.</p><p>⚫️Stop SparkSession:</p><p>. Once the query is executed and the results are obtained, stop the SparkSession to release the resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*qOw608KLvbEdws0Der6PYA@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4ff32a03bc9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Empowering Insights with Azure Data Flow! ]]></title>
            <link>https://medium.com/@soundaryasenthil510/empowering-insights-with-azure-data-flow-0666edca5023?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/0666edca5023</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:07:37 GMT</pubDate>
            <atom:updated>2025-04-23T07:03:55.924Z</atom:updated>
            <content:encoded><![CDATA[<p>📥 Ingestion: Data enters the Azure ecosystem from diverse sources such as IoT devices, applications, databases, and external platforms. Azure offers a multitude of services like Azure Data Factory, Azure Event Hubs, and Azure Blob Storage for efficient data ingestion, ensuring that data arrives securely and reliably.</p><p>🔄 Transformation: Once ingested, data undergoes transformation processes to cleanse, enrich, and prepare it for analysis. Azure Databricks, Azure HDInsight, and Azure Synapse Analytics are among the services used to perform these transformations, ensuring data quality and consistency.</p><p>💾 Storage: Processed data finds its home in Azure’s robust storage solutions, including Azure Data Lake Storage, Azure SQL Database, and Azure Cosmos DB. These storage services offer scalability, durability, and security, ensuring that data is accessible and protected.</p><p>🔍 Analysis: With the data prepared and stored, it’s time for analysis. Azure provides a suite of powerful analytics tools such as Azure Synapse Analytics, Azure Machine Learning, and Power BI, enabling organizations to derive valuable insights, uncover patterns, and make data-driven decisions.</p><p>📈 Visualization: Insights derived from data analysis are visualized using intuitive dashboards and reports. Power BI, in particular, empowers users to create interactive visualizations that facilitate understanding and decision-making across the organization.</p><p>🤝 Integration &amp; Distribution: Azure facilitates seamless integration and distribution of data insights across various applications, services, and stakeholders. Azure API Management, Azure Logic Apps, and Azure Functions enable efficient data integration and dissemination, ensuring that insights reach the right people at the right time.</p><p>🔒 Governance &amp; Security: Throughout the entire data flow process, Azure prioritizes governance, compliance, and security. Azure Active Directory, Azure Key Vault, and Azure Security Center provide robust identity management, encryption, and compliance monitoring capabilities, safeguarding data integrity and privacy.</p><p>⚙️ Automation &amp; Optimization: Azure offers automation capabilities through services like Azure Logic Apps and Azure Automation, streamlining data flow processes and reducing manual intervention. Additionally, Azure Cost Management helps optimize resource utilization, ensuring cost-effectiveness and efficiency.</p><p>Azure’s powerful suite of data technologies enables businesses to unlock valuable insights and drive growth</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d4SYnCfvBD7P1XHVCfK16Q@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0666edca5023" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JOINS IN PYSPARK:]]></title>
            <link>https://medium.com/@soundaryasenthil510/joins-in-pyspark-b86bd40ed528?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/b86bd40ed528</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:05:14 GMT</pubDate>
            <atom:updated>2025-04-23T07:06:44.756Z</atom:updated>
            <content:encoded><![CDATA[<p>In PySpark, joining two DataFrames is a common operation used for combining data from different sources.</p><p>The syntax for joining DataFrames is as follows:</p><p>joined_df = df1.join(df2, on=&lt;join condition&gt;, how=&lt;join type&gt;)</p><p>Here’s a breakdown of the parameters:</p><p>•	df1, df2: The DataFrames to be joined.</p><p>•	on: The join condition, specifying the column(s) to join on. This can be a column name or a list of column names.</p><p>•	how: The type of join to perform, which can be one of the following:</p><p>◾️”inner”: Returns only the rows that have matching values in both DataFrames.</p><p>◾️	”outer”: Returns all rows from both DataFrames and fills in missing values with nulls where there is no match.</p><p>◾️”left”: Returns all rows from the left DataFrame and the matching rows from the right DataFrame. If there’s no match, it fills in null values.</p><p>◾️	”right”: Returns all rows from the right DataFrame and the matching rows from the left DataFrame. If there’s no match, it fills in null values.</p><p>◾️Left Semi Join: This type of join returns only the rows from the left DataFrame where there is a match with the right DataFrame based on the specified condition.</p><p>◾️Left Anti Join: This type of join returns only the rows from the left DataFrame where there is no match with the right DataFrame based on the specified condition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*-rONENwj5IWa1xyI0FusXg@2x.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*N4A1tmqXhpfenNXKqTM7Zg@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b86bd40ed528" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s explore the differences between Azure Blob Storage, Azure Data Lake Gen 1, and Azure Data…]]></title>
            <link>https://medium.com/@soundaryasenthil510/lets-explore-the-differences-between-azure-blob-storage-azure-data-lake-gen-1-and-azure-data-d388083e3269?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/d388083e3269</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:04:20 GMT</pubDate>
            <atom:updated>2025-04-23T07:04:13.448Z</atom:updated>
            <content:encoded><![CDATA[<h2>🎲Let’s explore the differences between Azure Blob Storage, Azure Data Lake Gen 1, and Azure Data Lake Gen 2! ♟️</h2><p>1️⃣ Azure Blob Storage:</p><p>♟️Pros:</p><p>•	Ideal for storing unstructured data like images, videos, and documents.</p><p>•	Offers scalable, durable storage with various access tiers to optimize cost and performance.</p><p>•	Integration with other Azure services like Azure Functions, Azure Logic Apps, and Azure Event Grid.</p><p>♟️Cons:</p><p>•	Limited support for complex analytics and querying directly within the storage.</p><p>•	Not optimized for big data processing tasks compared to Data Lake offerings.</p><p>2️⃣ Azure Data Lake Gen 1:</p><p>♟️Pros:</p><p>•	Designed specifically for big data analytics workloads.</p><p>•	Supports both structured and unstructured data.</p><p>•	Provides fine-grained access control and integrated analytics with Azure HDInsight.</p><p>♟️Cons:</p><p>. •	Slightly higher costs compared to Azure Blob Storage.</p><p>•	Limited scalability and performance for extremely large datasets compared to Gen 2.</p><p>3️⃣ Azure Data Lake Gen 2:</p><p>♟️Pros:</p><p>•	Improved performance, scalability, and security features compared to Gen 1.</p><p>•	Offers hierarchical namespace for organizing data into folders and subfolders.</p><p>•	Integration with Azure Active Directory for access control.</p><p>♟️Cons:</p><p>•	Higher storage costs compared to Blob Storage and Gen 1.</p><p>•	Complexity in setting up and managing compared to Blob Storage.</p><p>Choosing the right storage solution depends on your specific requirements, such as data volume, access patterns, and security needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*MFyfGNrOp0I4lumukr6JeA@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d388083e3269" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Understanding Azure Storage Redundancy Options ]]></title>
            <link>https://medium.com/@soundaryasenthil510/understanding-azure-storage-redundancy-options-d4058dc3e895?source=rss-8fc964e6f7df------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4058dc3e895</guid>
            <dc:creator><![CDATA[SoundaryaSenthilkumar]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:03:22 GMT</pubDate>
            <atom:updated>2025-04-23T07:04:27.498Z</atom:updated>
            <content:encoded><![CDATA[<p>When storing data in Azure, it’s crucial to choose the right redundancy option to ensure durability and availability. Here’s a quick overview:</p><p>🔸Locally Redundant Storage (LRS):</p><p>•	LRS replicates your data three times within a single storage scale unit in a datacenter.</p><p>•	This replication ensures durability and high availability of your data within a single Azure region.</p><p>•	LRS is the most cost-effective option for storage redundancy in Azure because it doesn’t incur additional charges for data replication outside the primary region.</p><p>•	While LRS provides resilience against hardware failures and ensures data availability, it does not protect against datacenter-wide failures or regional outages.</p><p>. 🔸Zone-Redundant Storage (ZRS):</p><p>•	ZRS replicates your data across multiple availability zones within a region.</p><p>•	Availability zones are physically separate data centers within an Azure region, each with independent power, cooling, and networking infrastructure.</p><p>•	By replicating data across availability zones, ZRS provides higher durability and availability compared to LRS.</p><p>•	ZRS ensures that your data remains available even if one availability zone experiences a failure or outage.</p><p>•	This redundancy is suitable for applications requiring higher levels of resilience and availability, such as mission-critical workloads.</p><p>🔹Geo-Redundant Storage (GRS):</p><p>. •	GRS replicates your data synchronously within the primary region and asynchronously to a paired region, providing additional resilience against regional outages.</p><p>•	Paired regions are located hundreds of miles apart and are designed to withstand major natural disasters and catastrophic events.</p><p>•	In the event of a regional outage, GRS automatically fails over to the paired region, ensuring data availability and business continuity.</p><p>•	GRS is ideal for applications requiring high availability and disaster recovery capabilities across geographic regions.</p><p>🔹Geo-Zone-Redundant Storage (GZRS):</p><p>•	GZRS combines the benefits of GRS and ZRS by replicating data across multiple availability zones in the primary region and asynchronously to a paired region.</p><p>•	This redundancy option offers the highest level of resilience and availability in Azure Storage.</p><p>•	GZRS provides protection against both local failures within a region (via ZRS) and regional outages (via GRS), making it suitable for the most critical and demanding workloads.</p><p>Choose the right redundancy option based on your application’s resilience and availability requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*kMo4hDFbypawqXPg7_8xRw@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4058dc3e895" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>