<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sateeshkumar Ponnaganti on Medium]]></title>
        <description><![CDATA[Stories by Sateeshkumar Ponnaganti on Medium]]></description>
        <link>https://medium.com/@psk.niru?source=rss-44c4432284a2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6rIztpII7UhUWDeorNdJrg.jpeg</url>
            <title>Stories by Sateeshkumar Ponnaganti on Medium</title>
            <link>https://medium.com/@psk.niru?source=rss-44c4432284a2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:28:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@psk.niru/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Senior Data Engineer PySpark (Checklist)]]></title>
            <link>https://medium.com/@psk.niru/senior-data-engineer-pyspark-checklist-f55b37117ad0?source=rss-44c4432284a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f55b37117ad0</guid>
            <dc:creator><![CDATA[Sateeshkumar Ponnaganti]]></dc:creator>
            <pubDate>Wed, 18 Feb 2026 10:30:20 GMT</pubDate>
            <atom:updated>2026-02-18T10:30:20.735Z</atom:updated>
            <content:encoded><![CDATA[<h3>Interview Concepts</h3><ol><li><strong>Spark Architecture (Deep Understanding)</strong></li></ol><ul><li>Driver vs Executor</li><li>Cluster Managers (YARN, Kubernetes, Standalone)</li><li>Spark Application Life Cycle</li><li>Execution DAG: Jobs → stages → tasks</li><li>Narrow vs Wide Transformation</li><li>Shuffles: cost, triggers, how to reduce them</li><li>Broadcast joins vs shuffle joins</li></ul><p><strong>Note</strong>: Expect the interviewer to drill down into how Spark distributes data, Schedule Tasks, and handles failures.</p><p>2. <strong>PySpark DataFrame API (Advanced Level Usage)</strong></p><p>You should be fluent with</p><ul><li>Complex data transformations: “<strong><em>explode</em></strong>”, “<strong><em>posexplode</em></strong>”, “<strong><em>map_from_entries</em></strong>”, “<strong><em>aggregate</em></strong>”, “<strong><em>transform</em></strong>”, “<strong><em>filter</em></strong>” on arrays/maps.</li><li>Window functions (dense_rank, lead, lag, rolling aggregates)</li><li>Working with structs, nested JSON, arrays, and maps.</li><li>Column expressions &amp; functional programming style (df.select(expr(“”))).</li></ul><p>3. <strong>Performance Tuning &amp; Optimisation</strong></p><p>This area is heavily tested by Senior Data Engineers.</p><p><strong><em>Spark Config Tuning:</em></strong></p><ul><li>spark.sql.shuffle.partitions</li><li>spark.default.parallelism</li><li>Adaptive Query Execution (AQE)</li><li>Skew mitigation (salting, skew join hint, broadcast joins)</li><li>Memory management: caching vs Persistence.</li></ul><p><strong><em>Performance Debugging</em></strong>:</p><ul><li>Understanding Spark UI</li><li>How to identify (long-running tasks, shuffle spill, skewed partitions, inefficient joins, unnecessary wide transformations)</li></ul><p><strong><em>Note</em></strong>: Be able to explain how you optimised real pipelines. Interviewers love actual stories.</p><p>4.<strong> PySpark Joins — Deep Concepts</strong>:</p><ul><li>Broadcast hints and auto broadcast</li><li>Sort-merge join</li><li>Shuffle-hash join</li><li>Bucketed Joins</li><li>Detecting and handling data skew</li></ul><p>5. <strong>Partitioning Strategies</strong></p><ul><li>Repartition vs Coalesce</li><li>Partitioning for I/O optimization on storage systems</li><li>Optimizing for downstream consumption</li><li>How partitioning affects parallelism and shuffle</li></ul><p>6. <strong>File Formats &amp; Storage Layers</strong></p><p>Understand the impact on performance and schema behaviour</p><ul><li>Parquet,</li><li>ORC,</li><li>AVRO,</li><li>Delta Lake (very important today)</li><li>Schema Evolution</li><li>Merge-on-read vs copy-on-write</li><li>Z-ordering, file compaction</li></ul><p>7. <strong>Delta Lake (Must know for Databricks interviews)</strong></p><ul><li>Time Travel</li><li>CDC using MERGE</li><li>OPTIMIZE + ZORDER</li><li>VACUUM</li><li>Delta log internals</li><li>Autoloader</li><li>Structured Streaming + Delta</li></ul><p>8. <strong>PySpark SQL (Advanced)</strong></p><ul><li>Catalyst Optimizer</li><li>Query plans (logical &amp; physical)</li><li>Using EXPLAIN</li><li>SQL window functions</li><li>Joins and Subqueries</li></ul><p>9. <strong>Structured Streaming</strong></p><p>For senior roles, they expect deep knowledge</p><ul><li>Trigger types (processing Time, once, continuous)</li><li>Stream-static joins</li><li>Watermarking &amp; late events</li><li>State store &amp; stateful aggregations</li><li>Exactly-once guarantees</li><li>Autoloader</li></ul><p>10. <strong>Distributed Computing Fundamentals</strong>: Be articulate</p><ul><li>Why Spark vs MapReduce</li><li>Fault Tolerance</li><li>Data locality</li><li>Shuffle boundaries</li><li>RDD lineage</li></ul><p>11. <strong>Python UDFs and Performance Impact</strong></p><ul><li>Normal UDFs</li><li>Pandas UDFs</li><li>Vectorised UDFs</li><li>Why are UDFs slower</li><li>When to avoid UDFs</li><li>Using SQL functions instead</li></ul><p>12. <strong>Best Practices for Production Pipelines</strong></p><ul><li>Modularizing PySpark code</li><li>Logging</li><li>Error handling</li><li>Idempotency</li><li>Retry logic</li><li>CI/CD for Spark job</li><li>Job Orchestration (Airflow, ADF, Databricks workflows)</li></ul><p>13. <strong>Cloud &amp; Data Engineering Integration</strong></p><p>Depending on your stack:</p><p><strong><em>Databricks</em></strong>:</p><ul><li>Clusters (job, interactive, all-purpose)</li><li>DBR versions</li><li>Unity Catalog</li><li>Secrets management</li></ul><p><strong><em>AWS</em></strong>:</p><ul><li>S3 (Partitioning, consistency model)</li><li>Glue catalog</li><li>EMR</li></ul><p>14. <strong>System Design for Data Engineering</strong></p><ul><li>Batch vs Stream Architecture</li><li>CDC ingestion patterns</li><li>Lakehouse design</li><li>Medallion architecture (bronze/silver/gold)</li><li>Large-scale ETL pipeline design</li><li>Handling Schema evolution</li><li>High-throughput job design</li></ul><p>15. <strong>Common Interview Coding Patterns</strong></p><ul><li>Windowing</li><li>Explode + Complex JSON transformation — <strong><em>Have to practise</em></strong></li><li>Deduplication (row_number, watermark)</li><li>Handling Data Skews</li><li>Optimizing Joins</li><li>Writing efficient aggregations</li><li>File ingesition + Partition logic</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f55b37117ad0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database vs Datawarehouse vs Data Lake vs Delta Lake]]></title>
            <link>https://medium.com/@psk.niru/database-vs-datawarehouse-vs-data-lake-vs-delta-lake-384164866b68?source=rss-44c4432284a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/384164866b68</guid>
            <dc:creator><![CDATA[Sateeshkumar Ponnaganti]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 12:25:09 GMT</pubDate>
            <atom:updated>2026-11-14T02:07:14.633Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Database</strong>:</p><p>a systematic collection of data or information that is stored electronically.</p><p>As long as your application needs to store data, you will eventually need a database.</p><p>Depending on the nature of your data and how they relate to each other, you may choose between a multitude of the following common database families.</p><p>Different types of databases:</p><ul><li>Relational databases: Oracle, MySQL, Microsoft SQL Server, and PostgreSQL</li><li>Document databases: MongoDB and CouchDB</li><li>Key-value databases: Redis and DynamoDB</li><li>Wide-column stores: Cassandra and HBase</li><li>Graph databases: Neo4j and Amazon Neptune</li></ul><p><strong>Data Warehouse:</strong></p><p>Imagine a bank which has multiple branch locations across the country, each branch runs a separate <strong>database system</strong> to store records of their customers and daily transactions. <strong>How should the HQ branch analyze the data from these multiple branches?</strong></p><p>To accomplish this, the bank might choose to complement their branch databases with a <strong>central data warehouse.</strong></p><p>The goal of using a data warehouse is to combine disparate data sources in order to analyze the data, look for insights, and create business intelligence (BI) in the form of reports and dashboards.</p><p>In other words, a data warehouse is essentially similar to a database, only that it is a <strong>giant database that is optimized for analytics workloads.</strong></p><p>Organizations usually implement the Extract, Transform, Load (ETL) processes to move data from its original sources to the central data warehouse. These processes are usually scheduled on a particular interval (e.g hourly or daily).</p><p>Data warehouses typically have a pre-defined and fixed relational schema, providing a high level of consistency and reliability and making them ideal to be used as storage sources for most of the analytics workloads.</p><p>Once the data is in the warehouse, business analysts can connect data warehouses with Business Intelligence tools such as Power BI or Tableau. These tools allow business analysts and data scientists to explore the data, look for insights, and generate reports for business stakeholders.</p><p>Examples of data warehouses include:</p><ul><li>Amazon Redshift.</li><li>Snowflake</li><li>Google BigQuery</li><li>IBM db2</li><li>Microsoft Azure Synapse.</li><li>Oracle</li><li>Stores Structured and pre-processed data in a columnar format optimized for analytics.</li><li>Data is organized into tables and schemas, with relationships enforced via constraints like Primary Key, Foreign Key, Unique Key, Check, Default etc.</li><li>Requires ETL (Extract, Transform, Load) process to ensure that only clean, structured data is loaded into data warehouse.</li><li>Data pipelines are created and scheduled to ensure data is periodically updated.</li><li>Built-in mechanism for data backups, indexing, and query optimization ensure smooth operations.</li><li>Query performance is significantly faster than a data lake. due to structured and indexed nature of the storage.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*Fbb6yAYdXnTmY79t.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*2enoAf8wfhiSaJYN.png" /></figure><p><strong>Data Lake</strong>:</p><p>While data warehouses are excellent for storing highly structured data for business intelligence type of workloads and guarantees high levels of consistency and reliability; they fall short when handling unstructured data that is required in most of the machine learning and data science workloads.</p><p>Data warehouses are not good at handling the high volume and velocity of big data.</p><p>The main limitations of the data warehouse can be summarized as:</p><ul><li><strong>Scalability </strong>— Data Warehouses are not designed to scale; providing a bottleneck for applications that consumes/stores big volumes of data.</li><li><strong>No support for semi-structured or unstructured data </strong>— While performing perfectly with the structured data, the data warehouse can not handle most of the semi-structured data (e.g csv, json files) and unstructured data(images, videos, audio etc). Whenever you want to store the unstructured data you will want to consider other solutions rather than a data warehouse.</li></ul><p>Data Lake extends the capabilities of the Data Warehouse, by providing scalable object storage that supports storage of vast amounts of raw data (both structured, semi-structured and unstructured) in their original format — without any predefined schema. Data lakes can contain everything from relational data to JSON documents to PDFs to audio files.</p><p>Data lakes are a cost-effective way to store huge amounts of data. Use a data lake when you want to gain insights into your current and historical data in its raw form without having to transform and move it. Data lakes also support machine learning and predictive analytics.</p><p>The following are examples of technology that provide flexible and scalable storage for building data lakes:</p><ul><li>AWS S3</li><li>Azure Blob Storage</li><li>Google Cloud Storage</li><li>HDFS</li></ul><p>Stores raw data in it’s native format (structured, semi-structured, or unstructured) without transformation.</p><p>Data is organized into directories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*ID4Jt70pZJxcqHEu.PNG" /></figure><p><strong>Delta Lake: (Data Lake + Data Warehouse)</strong></p><p>While data lakes are amazing and provide scalability and support for unstructured data (advantages that are missing from the warehouses), the object storage nature of the data lakes makes them lose most of the advantages provided by the warehouses.</p><p>Despite their capabilities to scale and handle different types of data, Data lakes suffers from the following disadvantages:</p><ul><li><strong>No Data consistency</strong> — The reads and writes operations are not synchronized nor isolated. This could lead to FileNotFound Errors in a data analytics system, where downstream business analytics processes try to read data from the data lake while it is being written by an ELT process. Data written to a cloud data lake might not be available instantly.</li><li><strong>No schema enforcement</strong> — Data lakes being object storage, they are not concerned with the structure and schema of data and are happy to store all and any data without performing any checks to make sure that the data is consistent. As a result it is possible to end up with massive amounts of data which are inconsistent or mismatched data types making it hard to be used for analytics.</li><li><strong>Lack of transactional guarantees </strong>— A typical relational database provides transactional guarantees when data is being written. This means that a database operation either completely succeeds or completely fails, and that any consumer trying to read the data simultaneously doesn’t get any inconsistent or incorrect data because of a database operation failure. Data lakes on the other hand does not provide this guarantees and it is up to the developer to clean up and manually rollback half written incomplete data from any failed jobs</li></ul><p>To address these challenges; another layer called delta is implemented on top of the data lake to solve these challenges.</p><p>Delta lake is an open source data storage layer implemented on top of the data lake to help bring reliability, ACID transactional guarantees, schema validation and evolution to the cloud-based data lakes.</p><p>With delta lake, we get both advantages provided by the legacy data warehouse and the modern data lakes.</p><p><strong>SUMMARY:</strong></p><p>Databases, data warehouses, data lakes and delta lakes each have their own purpose.</p><p>every modern application will require a <strong><em>database </em></strong>to store the current application data.</p><p>Organizations that want to analyze their applications’ current and historical data may choose to complement their databases with a data warehouse, a data lake, or delta lake.</p><p>The choice of which one to use will depend on different factors including the nature of your data (structured vs unstructured) and whether your analysis will benefit from having a pre-defined, fixed schema or raw data and cost.</p><p><strong>Wrapping up!</strong></p><p><strong>- Database — </strong>An organized collection of structured information, or data, typically stored electronically in a computer system.</p><p><strong>- Data warehouse</strong> — Supports structured data making them suitable for analytics and BI workloads — does not support unstructured data making them less ideal for ML workloads.</p><p>- <strong>Data lake</strong> — Supports unstructured data and built to scale — but lacks enforcement of ACID transactions.</p><ul><li><strong>Delta lake</strong> — A storage layer on top of a data lake that enables ACID transactions and guarantees consistency and supports batch and streaming workloads.</li></ul><p><strong>Data Lakehouse</strong>:</p><ul><li>Combines Data Lake + Data Warehouse</li><li>Used in Databricks, Delta Lake, AWS Lake Formation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*xkAfwKN1bG6IULgg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*yMLq8nV0KGyF_l6m" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=384164866b68" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Practices for writing SQL queries]]></title>
            <link>https://medium.com/@psk.niru/best-practices-for-writing-sql-queries-9639d8ba9a83?source=rss-44c4432284a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9639d8ba9a83</guid>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Sateeshkumar Ponnaganti]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 18:16:47 GMT</pubDate>
            <atom:updated>2025-08-22T18:16:47.236Z</atom:updated>
            <content:encoded><![CDATA[<p>Writing SQL queries efficiently and clearly is essential for effectively managing and retrieving data.</p><p>By following some best practices you can enhance the performance, readability, and maintainability of your SQL code.</p><p>Here are some key best practices to consider when writing SQL queries, explained in detail.</p><ol><li>Write SQL Keywords in Capital Letters.</li></ol><p>Using capital letters for SQL keywords (like ‘SELECT’, ‘FROM’, ‘WHERE’, etc.). it helps distinguish these commands from table and column names.</p><p>This practice enhances the readability of your code, making it easy to understand and maintain.</p><p><strong><em>Example</em></strong>:</p><pre>SELECT e.id, e.name, e.gender, e.city, e.company, e.salary <br>FROM dbo.tbl_employees AS e <br>WHERE e.gender = &#39;female&#39;;</pre><p>2. Use Table aliases with columns when joining multiple tables.</p><pre>SELECT e.emp_id, e.emp_name, e.emp_gender, d.dept_name<br>FROM tbl_employees AS e<br>INNER JOIN tbl_department AS d<br>ON e.dept_id = d.dept_id;</pre><p>3. Never use ‘SELECT * ‘; always mention list of column in SELECT statement.</p><p>using “ SELECT * ” retrieve all columns from a table, which can lead to unnecessary data fetching and slow down query performance. instead, specify the columns you need.</p><p>This practice not only improves the performance but also makes the query more readable and clear about what data is being retrieved.</p><pre>SELECT e.emp_id, e.emp_name, e.emp_gender<br>FROM dbo.tbl_employees AS e;</pre><p>4. Add useful comments whenever you write complex logic; also at the same time try to avoid too many comments.</p><p>5. Use Joins instead of subqueries whenever possible for better performance.</p><p>6. Create CTEs instead of multiple subqueries, it will make your Query easy to read.</p><p>7. Join tables using JOIN keywords instead of writing JOIN conditions in WHERE clause for better readability.</p><p>Recommended (using JOIN Keyword)</p><pre>SELECT e.emp_id, e.emp_name, e.emp_gender, d.dept_name<br>FROM tbl_employees AS e<br>INNER JOIN tbl_department AS d<br>ON e.dept_id = d.dept_id;</pre><p>Not Recommended (Using WHERE Clause)</p><pre>SELECT e.emp_id, e.emp_name, e.emp_gender, d.dept_name<br>FROM tbl_employees AS e, tbl_department AS d<br>WHERE e.dept_id = d.dept_id;</pre><p>8) Never use ‘ORDER BY’ clause in subqueries, it will unnecessarily increase runtime.</p><p>including ‘ORDER BY’ in subqueries can slow down your queries because the database engine performs unnecessary sorting operations. instead, apply, ‘ORDER BY’ in the main query where the final ordering is required.</p><p>9) If you know there are no duplicates in two Tables, use UNION ALL instead of UNION for better performance.</p><p>‘UNION’ combines the results of two queries and removes duplicates, while ‘UNION ALL’ combines results without removing duplicates. hence, ‘UNION ALL’ is faster and more efficient than ‘UNION’.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9639d8ba9a83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Row-Based vs Column Based]]></title>
            <link>https://medium.com/@psk.niru/row-based-vs-column-based-cd8d088a2449?source=rss-44c4432284a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd8d088a2449</guid>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sateeshkumar Ponnaganti]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 17:33:11 GMT</pubDate>
            <atom:updated>2025-08-22T17:33:11.768Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s take the sample data which is gonna use as a base for the study here. Consider I have employee table with 5 records and 4 columns.</p><figure><img alt="Sample Data" src="https://cdn-images-1.medium.com/max/398/0*Y2bteIpC1euRgqfp.png" /></figure><p><strong>Storage mechanisms</strong>:</p><p>Primarily there are only two ways how the data stores in the files.</p><ol><li>Row oriented storage</li><li>Column oriented storage</li></ol><p><strong><em>Row oriented storage</em></strong>: Store’s data row-wise to the computer disk.</p><p><strong><em>Column oriented storage</em></strong>: Store’s data column-wise to the computer disk.</p><p><strong>Row Oriented Storage</strong>:</p><p>The best example of explaining the row-oriented data is the <strong><em>CSV </em></strong>format. Data get’s persisted in the CSV format s a list of rows. as listed in above example , consider row as the block and that get’s stored as a single unit to the storage. please look at below diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/0*Xe7Vj679aCvR7D8z.png" /><figcaption>CSV Storage Format</figcaption></figure><p><strong>Column Oriented Storage</strong>:</p><p>The best example of explaining the column-oriented data is the <strong><em>Parquet </em></strong>format. The data gets persisted in the parquet format as a list of columns. as listed in above example, consider one column as the block and that get’s stored as a single unit to the storage. please look at below diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/0*UiXQpxQnbq2M7PlO.png" /><figcaption>Parquet Storage Format</figcaption></figure><p><strong>Comparison (Row vs Col)</strong>:</p><p>if the dataset is small or medium then it doesn&#39;t matter whether it is a row-oriented or column-oriented. But the way data is growing and now we’re talking in Petabyte’s and Zettabyte&#39;s of data. Thus, it is crucial to decide what data format should you be considered for your project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*FaR0RfucX_M4nUNH.png" /><figcaption>Comparison</figcaption></figure><p><strong>Why to choose Column as a first choice in today’s era of big data</strong>?</p><ul><li>Compression</li><li>Speed</li></ul><p>Row-Based files (like CSV) are usually larger in size as compare to column based files (like parquet). This is because the columnar storage is the column compressed storage where compression depends on their data types (string, date-time, integer, etc.). Which is not the case with Row-based storage as it stores entire row which many times store on the basis of string types (in case of CSV, for instance).</p><p>Second, Column-based files like parquet have over Row-based is the speed (performance). the columnar based file data is really fast to scan and extract the aggregates. this is because the query runs only to one column and not on all.</p><p><strong>Check Popular Ones</strong>:</p><p>There are many data formats, cloud storages and backend databases (relational and NoSQL).</p><p><strong><em>Column-Based</em></strong>: Parquet, ORC, Delta, Apache Hive, Apache Cassandra, Apache HBase, AWS Redshift, Google Big Query, Snowflake.</p><p><strong><em>Row-Based</em></strong>: CSV, JSON, XML, AVRO, RDBMS Systems, MongoDB, and Redis (Key-Value).</p><p><strong>Explain Parquet File format and it’s advantages</strong>:</p><p>Parquet is a columnar based file format. others like CSV format is a row based format.</p><p>when the data volume is huge, like there are many columns. so when you are doing some aggregations or may be filtering the data of only few columns. these kind of operations are easy for columnar data.</p><p>Also it has like a comparison technique. so the parquet files size is as low as compared to other file formats.</p><p>In Columnar data if we are looking for some kind of aggregations so only we can take all the particular columns for the aggregation purpose. so the rest of the columns will be not considered.</p><p>Due to this reason it is more optimized.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd8d088a2449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differences between ORC, AVRO, and Parquet Files?]]></title>
            <link>https://medium.com/@psk.niru/differences-between-orc-avro-and-parquet-files-c482e93b4a9d?source=rss-44c4432284a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c482e93b4a9d</guid>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sateeshkumar Ponnaganti]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 16:50:26 GMT</pubDate>
            <atom:updated>2025-08-22T16:50:26.449Z</atom:updated>
            <content:encoded><![CDATA[<p>different file formats are used in big data applications. each with its own advantages and disadvantages.</p><p>it is essential for professional working with big data to be familiar with different file formats.</p><ul><li>ORC (Optimized Row Columnar)</li><li>AVRO</li><li>Parquet</li></ul><p><strong>ORC (Optimized Row Columnar)</strong>:<br>A file format that combines the benefits of both row-based and column-based storage for efficient querying and compression.</p><p><strong><em>Advantages</em></strong>:</p><ul><li>Faster when reading and writing of large datasets.</li><li>Efficient compression reduced storage requirements.</li></ul><p><strong><em>Disadvantages</em></strong>:</p><ul><li>Limited support for schema evolution (the process of modifying the logical structure “schema”).</li></ul><p><strong>AVRO</strong>:</p><p>A binary data format that is compact and efficient for storing and exchanging data in big data applications.</p><p><strong><em>Advantages</em></strong>:</p><ul><li>Supports schema evolution for flexible data schemas, compact binary format for efficient data exchange.</li></ul><p><strong><em>Disadvantages</em></strong>:</p><ul><li>Slower than other file formats due to schema parsing.</li></ul><p><strong>Parquet</strong>:</p><p>A columnar storage format. this is optimized for processing large datasets in parallel.</p><p><strong><em>Advantages</em></strong>:</p><ul><li>Columnar storage allows for efficient querying and processing of large datasets.</li><li>efficient compression for reduced storage requirements.</li></ul><p><strong>Disadvantages</strong>:</p><ul><li>Slower writing speed than row-based storage formats.</li><li>Limited support for schema evolution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jyCzN7UPgwZQDCgI.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c482e93b4a9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>