<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Deepa Vasanthkumar on Medium]]></title>
        <description><![CDATA[Stories by Deepa Vasanthkumar on Medium]]></description>
        <link>https://medium.com/@deepa.account?source=rss-96d6ee505787------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Uhc2XU-h9DzKHZCf-nrozQ.png</url>
            <title>Stories by Deepa Vasanthkumar on Medium</title>
            <link>https://medium.com/@deepa.account?source=rss-96d6ee505787------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:36:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deepa.account/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Built a Zero-Cost SQL Explorer with React, DuckDB WASM, GitHub & Vercel]]></title>
            <link>https://medium.com/@deepa.account/i-built-a-zero-cost-sql-explorer-with-react-duckdb-wasm-github-vercel-f3bf802f25d9?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3bf802f25d9</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 05:50:40 GMT</pubDate>
            <atom:updated>2026-08-18T12:32:25.815Z</atom:updated>
            <content:encoded><![CDATA[<p>Run SQL on Parquet, CSV, and JSON directly in your browser — no database server, no Docker, no backend.</p><p>Sometimes the best database is no database at all.</p><p>As data engineers, we constantly spin up databases just to inspect a Parquet file, validate a CSV, or test a SQL query. That usually means PostgreSQL, SQLite, Docker, or cloud resources — even for a five-minute task.</p><p>I wanted something different.</p><p>Try:</p><p><a href="https://data-engineer-toolkit-eta.vercel.app/sql">Data Tool Kit</a></p><blockquote><a href="https://data-engineer-toolkit-eta.vercel.app/sql">https://data-engineer-toolkit-eta.vercel.app/sql</a></blockquote><p>So I built a SQL Explorer that runs entirely inside the browser using a modern, zero-cost stack:</p><ul><li>React for the UI</li><li>DuckDB WASM as the SQL engine</li><li>Monaco Editor for SQL editing</li><li>TanStack Table for results</li><li>GitHub for source control</li><li>Vercel for free deployment</li></ul><p>The result is a browser-native SQL playground where you can upload data files and query them instantly.</p><p>No server. No backend. No infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Om2yQJJqw3GMVRnD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*Fe-JH4M5ArztRoBLB4NuAw.png" /></figure><p>The entire application is essentially a single-page app.</p><p>The browser becomes the <strong>database</strong>.</p><h3>Architecture</h3><p>Instead of sending files to a backend, everything happens locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*QalyhopwQAJmpX4Ui3LHEw.png" /></figure><p>The only thing running remotely is the static website itself.</p><h3>Why DuckDB WASM Changes Everything</h3><p>DuckDB has already become a favorite among data engineers.</p><p>The WASM version takes it one step further.</p><p>Instead of installing DuckDB locally, the engine is compiled to WebAssembly and executed inside the browser.</p><p>That means:</p><ul><li>SQL execution happens locally.</li><li>Files never leave your machine.</li><li>No API calls.</li><li>No backend compute.</li><li>Near-native performance.</li></ul><p>For exploratory analytics, that’s incredibly powerful.</p><h3>Uploading Files Becomes Creating Tables</h3><p>One feature I wanted was making uploaded files behave like real SQL tables.</p><p>When a user uploads a file, the app:</p><p>Detects the file type</p><p>Registers it with DuckDB.</p><p>Creates a SQL view automatically.</p><p>Makes it immediately queryable.</p><p>The component supports:</p><ul><li>Parquet</li><li>CSV</li><li>TSV</li><li>JSON</li></ul><p>From the user’s perspective:</p><blockquote><em>Upload a file → Write SQL.</em></blockquote><p>That’s it.</p><p>The implementation registers the file buffer with DuckDB and creates a view so the uploaded file behaves like a relational table.</p><p>The SQL editor uses <strong>Monaco</strong>, the same editor behind VS Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3UZAl2tLklH3rQcfCYTyeg.png" /></figure><h3>Instant Results with TanStack Table</h3><p>Running SQL is only half the experience.</p><p>Displaying results efficiently is equally important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*M-hYqSVUlPINpN5hyVZHeQ.png" /></figure><p>The results grid includes:</p><ul><li>Resizable columns</li><li>Sticky headers</li><li>Scrollable datasets</li><li>Dynamic column generation</li><li>NULL value handling</li></ul><h3>Deployment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*JxrNV5CnJeKCWgYxDqnwJw.png" /></figure><p>No imports.</p><p>No migrations.</p><p>No database setup.</p><p>The most interesting part of this project wasn’t React or Vercel.</p><p>It was realizing that the browser has become a legitimate analytics platform. DuckDB WASM turns a static frontend into a fully functional SQL engine. Combined with React, Monaco, and TanStack Table, it creates a surprisingly capable data exploration tool without maintaining servers, databases, or cloud infrastructure.</p><p><em>Sometimes the most scalable architecture is simply letting the user’s browser do the work.</em></p><p>#React #DuckDB #WebAssembly #DataEngineering #SQL #MonacoEditor #TanStackTable #Vercel #GitHub #Frontend</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3bf802f25d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas Arrow and Polars]]></title>
            <link>https://medium.com/@deepa.account/pandas-arrow-and-polars-4e89250ffea5?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e89250ffea5</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 07:30:01 GMT</pubDate>
            <atom:updated>2026-07-20T07:30:01.320Z</atom:updated>
            <content:encoded><![CDATA[<p>This Tiny blog shows the execution time comparison between pandas, polars using arrow from Spark Dataframe</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*MxM1VWcA9eZb24jwpw7idw.png" /></figure><p>Try it out, yourself :</p><p>Setting up environment using google colab</p><pre>!pip install pyspark==4.2.0 --no-deps<br>!pip install py4j<br>!pip install -q polars pyarrow<br>!pip install -q duckdb</pre><p>Creating Spark session and sample data</p><pre>from pyspark.sql import SparkSession, functions as F<br>spark=SparkSession.builder.master(&quot;local[*]&quot;).appName(&quot;Spark_4_2_Demo&quot;).getOrCreate()<br>spark<br>df = (<br>    spark.range(1_000_000)<br>         .withColumn(&quot;square&quot;, F.col(&quot;id&quot;) * F.col(&quot;id&quot;))<br>)<br><br>df.show(5)</pre><pre>+---+------+<br>| id|square|<br>+---+------+<br>|  0|     0|<br>|  1|     1|<br>|  2|     4|<br>|  3|     9|<br>|  4|    16|<br>+---+------+<br>only showing top 5 rows<br>    </pre><p>Comparision</p><pre>import time<br><br># Pandas path<br>start = time.time()<br>pdf = df.toPandas()<br>print(&quot;Pandas:&quot;, time.time() - start)<br><br># Arrow path<br>start = time.time()<br>arrow = df.toArrow()<br>print(&quot;Arrow:&quot;, time.time() - start)</pre><pre>Pandas: 1.8471336364746094<br>Arrow: 0.7206161022186279</pre><pre>import polars as pl<br><br>start = time.time()<br>pl_df = pl.from_arrow(arrow)<br>print(&quot;Arrow -&gt; Polars:&quot;, time.time() - start)</pre><pre>Arrow -&gt; Polars: 0.10664725303649902</pre><p>Tip:</p><p>Convert to <strong>Pandas</strong> only when you need compatibility with libraries that expect a Pandas DataFrame (for example, parts of the Python ML ecosystem or visualization libraries).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e89250ffea5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Iceberg Odyssey]]></title>
            <link>https://medium.com/@deepa.account/an-iceberg-odyssey-694eb3c0cf32?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/694eb3c0cf32</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 06:59:47 GMT</pubDate>
            <atom:updated>2026-04-20T07:02:25.901Z</atom:updated>
            <content:encoded><![CDATA[<p>From Iceberg Fundamentals to Modern Lakehouse Design</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*AYiNtkx-P_Bv4ayh32t5Yg.png" /></figure><p>If you’ve worked with data lakes long enough, you’ve likely seen the pattern: what begins as a flexible and scalable storage solution gradually turns into a complex, hard-to-manage system. Files pile up, schemas drift, and maintaining consistency across pipelines becomes a constant struggle.</p><p>This is where Apache Iceberg is changing the game.</p><h3>The Shift from Traditional Data Lakes</h3><p>Traditional data lakes were designed for flexibility, but not necessarily for reliability. Over time, teams run into familiar challenges — lack of version control, difficulty in managing schema changes, and inefficient query performance.</p><p>Modern data platforms are evolving to address these issues, and Iceberg represents a key step in that evolution. It introduces structure and reliability without sacrificing the scalability that made data lakes attractive in the first place.</p><h3>What Makes Apache Iceberg Powerful?</h3><p>At its core, Iceberg brings order to large-scale data systems through a robust table format. Instead of treating data as just files in storage, it adds a powerful metadata layer that:</p><ul><li>Tracks data changes over time</li><li>Enables versioning and time travel</li><li>Improves query planning and performance</li></ul><p>This approach allows teams to manage massive datasets with far greater confidence and control.</p><h3>Working with Modern Data Tools</h3><p>Iceberg fits naturally into today’s data ecosystem. It integrates seamlessly with engines like Apache Spark, allowing you to create, update, and manage datasets efficiently.</p><p>Whether you’re handling batch processing or streaming data, Iceberg provides the consistency guarantees that traditional data lakes lack. Partitioning strategies, optimization techniques, and table management become much more predictable and scalable.</p><h3>Iceberg in the Lakehouse Architecture</h3><p>The rise of the lakehouse paradigm has blurred the lines between data lakes and data warehouses. Iceberg plays a central role in this shift by enabling:</p><ul><li>Reliable table management on top of object storage</li><li>Compatibility with multiple query engines</li><li>Support for both analytical and streaming workloads</li></ul><p>It also stands alongside other modern table formats like Delta Lake and Apache Hudi, each with its own strengths. Understanding where Iceberg fits helps in making better architectural decisions.</p><h3>Beyond Basics: Real-World Data Challenges</h3><p>Building a data platform isn’t just about storing and querying data. Real-world systems demand:</p><ul><li>Seamless schema evolution</li><li>Strong data governance and security</li><li>Cost-efficient storage and compute strategies</li><li>High performance at scale</li></ul><p>Iceberg addresses these concerns with features designed for production-grade environments, making it a strong choice for enterprise data platforms.</p><h3>From Experimentation to Production</h3><p>Adopting a new data technology isn’t just about learning APIs — it’s about making it work reliably in production.</p><p>With Iceberg, that means understanding best practices, avoiding common pitfalls, and designing for resilience. Backup strategies, recovery mechanisms, and operational considerations all play a crucial role in long-term success.</p><h3>Why You Should Care</h3><p>Apache Iceberg isn’t just another tool in the ecosystem — it reflects a broader shift toward more dependable, scalable, and maintainable data systems.</p><p>For data engineers, architects, and platform teams, this shift is becoming increasingly important as data volumes grow and system complexity increases.</p><p>If you’re exploring ways to modernize your data platform or struggling with the limitations of traditional data lakes, Apache Iceberg is worth a closer look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*TLooCfS5R-RTyMFOK_1uEg.png" /></figure><p>And if you want to go deeper — covering everything from fundamentals to advanced implementation strategies — there’s a comprehensive resource available that walks through these concepts step by step. It’s definitely worth checking out if you’re serious about building next-generation data systems.</p><p><a href="https://amzn.in/d/0hTLJxGj">Checkout the book here</a></p><p>📚 <strong>Good news:</strong> There’s an ongoing promotion, so you may be able to <strong>read it for free on Kindle now and can download free promotion day scheduled on 30-April-2026 </strong>— perfect time to dive in and explore without any commitment.</p><p>⭐ If you do check it out, please consider <strong>leaving a rating or review</strong>. Since it’s a new book, your feedback genuinely helps improve visibility and reach others in the community.</p><p>💬 Also, if you’re interested in <strong>next free promotion days</strong>, drop a comment saying <em>“interested”</em>. I’ll DM you when the book is scheduled for upcoming free access windows.</p><p>Happy learning and building!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=694eb3c0cf32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Modern Lakehouse using Datafusion, Parquet and Plotly]]></title>
            <link>https://medium.com/@deepa.account/building-modern-lakehouse-using-datafusion-parquet-and-plotly-1575c0dfb995?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/1575c0dfb995</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[lake-house]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[parquet]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 06:53:20 GMT</pubDate>
            <atom:updated>2026-03-09T06:53:20.142Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDHo8ulNjfPwAfrxRWla5w.png" /></figure><p>A lakehouse combines the flexibility of a data lake with the structured analytics capabilities of a data warehouse.</p><p>Instead of storing data in expensive proprietary databases, data is stored in open formats on object storage and queried using scalable compute engines.</p><p>This architecture enables:</p><ul><li>SQL analytics</li><li>scalable storage</li><li>interactive data exploration</li><li>open data formats</li></ul><h3>Source code repo:</h3><p><a href="https://github.com/deepavasanthkumar/modern_lakehouse_datafusion">GitHub - deepavasanthkumar/modern_lakehouse_datafusion: Simple portable lakehouse implementation using python, datafusion, plotly, arrow</a></p><h3>Core Components of the Stack</h3><h3>Storage Layer — Parquet Data Lake</h3><p>The foundation of the lakehouse is <strong>Apache Parquet</strong>, a highly efficient columnar format designed for analytical workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RRAhSKn_1JGeUqHgosGkaw.png" /></figure><p>Benefits:</p><ul><li>Columnar compression</li><li>Efficient predicate pushdown</li><li>Reduced storage footprint</li><li>Compatible with many engines</li></ul><p>Because Parquet is an <strong>open format</strong>, data can be accessed by many engines without vendor lock-in.</p><h3>Query Engine — DataFusion</h3><p>The compute layer is provided by <strong>Apache DataFusion</strong>, a high-performance SQL query engine built in Rust.</p><p>DataFusion capabilities include:</p><ul><li>ANSI SQL support</li><li>vectorized execution</li><li>query optimization</li><li>predicate pushdown</li><li>Arrow-native processing</li></ul><p>DataFusion operates directly on Parquet files without requiring a traditional database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WlEd3id0_aMxmxooQLPJcw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eC6pi-1BN9nmod6TvyFXyg.png" /></figure><p>Example query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fv5WleMFuZA6ivnHNCeIdQ.png" /></figure><p>Unlike heavy distributed engines, DataFusion can run <strong>embedded applications, notebooks, or microservices</strong>.</p><h3>Analytics Layer — Plotly</h3><p>For visualization, the stack uses <strong>Plotly</strong>.</p><p>Plotly provides:</p><ul><li>interactive dashboards</li><li>web-ready visualizations</li><li>notebook integration</li><li>real-time chart rendering</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0bMYXKP6oZKfvG1ayAu_PQ.png" /></figure><p>Example analytics flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aha8pp9E84tANe5M24Ztog.png" /></figure><p>This enables rapid exploratory analytics without needing a full BI platform.</p><h3>End-to-End Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e9cGmfmqOVfhmESZmbneWQ.png" /></figure><p>This architecture is similar to enterprise lakehouse stacks used by:</p><ul><li><strong>Apache Spark</strong></li><li><strong>Trino</strong></li><li><strong>DuckDB</strong></li></ul><p>but implemented with <strong>much lighter infrastructure</strong>.</p><h3>Cost Advantages Over Traditional Cloud Analytics</h3><p>Traditional cloud analytics platforms often rely on expensive compute services such as managed clusters or proprietary warehouses.</p><p>Example platforms include:</p><ul><li><strong>Google BigQuery</strong></li><li><strong>Amazon Redshift</strong></li><li><strong>Snowflake</strong></li></ul><p>These systems typically charge for:</p><ul><li>compute usage</li><li>query execution</li><li>data scanning</li><li>storage</li><li>concurrency scaling</li></ul><p>Organizations can reduce <strong>analytics compute costs by 70–90%</strong> in exploratory workloads by using embedded engines.</p><h3>Performance Advantages</h3><p>Despite being lightweight, this architecture benefits from:</p><h4>Columnar Processing</h4><p>Parquet stores data by column, allowing queries to scan only relevant columns.</p><p>Example:</p><pre>SELECT country FROM orders</pre><p>Only the <strong>country column</strong> is read from disk.</p><h4>Predicate Pushdown</h4><p>Query filters can be pushed to the storage layer.</p><p>Example:</p><pre>SELECT * FROM orders<br>WHERE country = &#39;US&#39;</pre><p>Only matching row groups are scanned.</p><h4>Vectorized Execution</h4><p>DataFusion processes data in batches using the <strong>Apache Arrow memory format</strong>, improving CPU efficiency.</p><h3>Ideal Use Cases</h3><p>This stack works particularly well for:</p><ul><li>Data exploration notebooks</li><li>Embedded analytics in applications</li><li>Small to medium analytics workloads</li><li>Edge analytics environments</li><li>Data engineering prototypes</li></ul><p>It can also serve as a <strong>local analytics engine for data scientists</strong> working with data lakes.</p><p><strong>Benefits for Data Engineering Teams</strong></p><p>Key advantages of this modern lakehouse stack include:</p><ul><li>open data formats</li><li>minimal infrastructure overhead</li><li>vendor neutrality</li><li>lower operational cost</li><li>rapid experimentation</li></ul><p>It allows engineers to build analytics pipelines without deploying large clusters.</p><p><strong>Future Evolution</strong></p><p>This architecture can evolve into a full enterprise lakehouse by adding:</p><ul><li>table formats such as <strong>Apache Iceberg</strong></li><li>orchestration tools</li><li>metadata catalogs</li><li>governance frameworks</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1575c0dfb995" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Iceberg and problem of Tiny files]]></title>
            <link>https://medium.com/@deepa.account/spark-iceberg-and-problem-of-tiny-files-9d5d369b77cb?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d5d369b77cb</guid>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Sat, 14 Dec 2024 17:52:39 GMT</pubDate>
            <atom:updated>2026-03-06T13:19:16.665Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OsI9plknfMj9dP0O5dX3aw.png" /></figure><p>Just a recap on Iceberg Table format and Layout:</p><p><strong>Data</strong>: Data in Iceberg is stored as Parquet, ORC, or Avro files in the file system (e.g., HDFS, S3, GS or Azure Blob Storage). The table can be organized into partitions, but unlike traditional systems, Iceberg manages partitions, avoiding hardcoding partition paths into query logic.</p><p><strong>Metadata</strong>: Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic swap. The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents. A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table.</p><p>Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file’s partition data, and its metrics. The data in a snapshot is the union of all files in its manifests. Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. Manifests can track data files with any subset of a table and are not associated with partitions.</p><p><strong>Partitions</strong>: The table is organized into partitions, but unlike traditional systems, Iceberg manages partitions transparently, avoiding hardcoding partition paths into query logic. Partitioning is a way to make queries faster by grouping similar rows together when writing. <br>Iceberg uses hidden partitioning, so we don’t need to write queries for a specific partition layout to be fast. Instead, we can write queries that select the data you need, and Iceberg automatically prunes out files that don’t contain matching data.</p><p><strong>Now the problem — Thousands of Tiny (files with few kb) data files written by Spark into Iceberg data/metadata folders.</strong></p><p>Firstly, Why Small data files are so problematic to deal with:</p><p>Tiny files leads to a large number of tasks during data processing, increasing task scheduling and execution overhead. Every tiny data file may be related to row in metadata file, so addition of any data file may be associated with a manifest entry, and there is an additional IO required to handle the metadata operations as well.</p><p>With respect to Storage, these tiny files occupy the default block storage for HDFS, and soon it may eat up the storage, and for cloud file systems, these come with the cost !</p><p>The more the files count, it reflects in the time for query plan creation time as well.</p><p>In short, the more the files in the iceberg table, the more it reflects with working with it, whether it is querying, deleting or even updating fewer records.</p><p>Over time, the maintenance struggle becomes real nightmare. With each tiny data file mapping to a manifest record, and iceberg relying heavily on the metadata information, and with tiny files getting accumulated and amount of metadata for the same these data files together, actual data might be far less than that.</p><p>In the optimal scenario, we need to keep the file count small and manageable.</p><p><strong>Now, How Spark Task maps to file in Iceberg Table Format</strong></p><p>Iceberg manages data storage at the file level, and during a <strong>write</strong> operation, Spark <strong>tasks</strong> are responsible for generating these files. Each Spark task operates on one or more partitions of the data and writes data to Iceberg. These files will be written independent. That is, No two tasks would be writing onto a single data file.</p><p>Iceberg tables have a property (write.target-file-size-bytes) that specifies the target size for data files. If a single Spark task may produce multiple data files if the size of the data exceeds the target file size. We need to also understand that the data in spark task is row based structure in memory where as the iceberg file structure uses parquet which is columnar and optimized in persistent storage. Also note that the property refers to the maximum file size and not the minimum threshold of a file.</p><p>Spark’s parallelism (number of tasks) determines how many files are written simultaneously. The more the number of tasks, the more the input files. The degree of parallelism is also influenced by the number of partitions in the input DataFrame and the spark.sql.shuffle.partitions setting. After all tasks finish, Iceberg updates its metadata and commits the new snapshot.</p><blockquote>The crux here, is that we need to have an optimal Task Size to map it to data file in iceberg table format.</blockquote><p><strong>Points to consider to control tiny files creation:</strong></p><ol><li><strong>Write Distribution Mode </strong><br>(Spark 3.5.0 onwards use write.distribution-mode)<br> defines the write behavior of batch or streaming job</li></ol><p><strong>none</strong>: don’t shuffle rows. It is suitable for scenarios where the rows are located in only few partitions, otherwise that may produce too many small files because each task is writing rows into different partitions randomly.</p><p><strong>hash</strong>: hash distribute by partition key, which is suitable for the scenarios where the rows are located into different partitions evenly.</p><p><strong>range</strong>: range distribute by partition key (or sort key if table has an {<a href="http://twitter.com/link">@link</a> SortOrder}), which is suitable for the scenarios where rows are located into different partitions with skew distribution.</p><p>If the table is partitioned opt for hash distribution. This will ensure that only one writer is responsible for each partition, which will result in fewer files being created within each partition and thus avoid the issue of having many small files. For range distribution, it’s suitable if your table has a sort order. Using ‘None’ may result in the creation of numerous small files, and then need to plan to run compaction process later on to merge these small files.</p><p><strong>2. Repartitioning &amp; Sorting </strong><br>If we already know the table partitioning and before the write, we can ensure that the data is already partitioned and sorted in the way that matches the Iceberg table’s partition spec. This can be done by repartitioning and sortWithinPartitions</p><p>Example, if the partition is based on the event_date for dataframe df, before the write operation,</p><pre>df = df.repartition(&lt;num of Partitions&gt;,&quot;event_date&quot;).sortWithinPartitions(&quot;event_date&quot;)</pre><p>where df is the spark dataframe before using write operation.</p><p><strong>3. Tuning Spark Configurations</strong></p><p>we can also consider tuning the spark configurations matching the required data into consideration.</p><p>spark.conf.set(“spark.sql.shuffle.partitions”, “500”)<br>spark.conf.set(“write.target-file-size-bytes”, “536870912”) # 512 MB<br>spark.conf.set(“write.distribution-mode”, “hash”)<br>spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, “104857600”) # 100 MB</p><p>Adaptive Query Execution umbrella configuration — -&gt; spark.sql.adaptive.enabled “true”</p><p>spark.sql.adaptive.coalescePartitions.enabled — “true” . This feature simplifies the tuning of shuffle partition number when running queries. Spark can pick the proper shuffle partition number at runtime once you set a large enough initial number of shuffle partitions via spark.sql.adaptive.coalescePartitions.initialPartitionNum configuration.</p><p>spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled “true” When true and spark.sql.adaptive.enabled is true, Spark will optimize the skewed shuffle partitions in RebalancePartitions and split them to smaller ones according to the target size (specified by spark.sql.adaptive.advisoryPartitionSizeInBytes), to avoid data skew.</p><p>spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor 0.2. A partition will be merged during splitting if its size is small than this factor multiply spark.sql.adaptive.advisoryPartitionSizeInBytes.</p><h3>Optimizing or Maintaining the Iceberg Tables (already written data)</h3><p>Additionally, there are some maintenance procedures available can be run periodically on the already written data.</p><p>Over the period, this is required and can be tuned on a need basis.</p><p><strong>remove_orphan_file</strong></p><p>Used to remove files which are not referenced in any metadata files of an Iceberg table and can thus be considered “orphaned”.</p><p>CALL &lt;catalog name&gt;.system.remove_orphan_files(table =&gt; ‘db.tablename’, dry_run =&gt; true);</p><p><strong>rewrite data files</strong></p><p>Iceberg can compact data files in parallel using Spark with the rewriteDataFiles action. This will combine small files into larger files to reduce metadata overhead and runtime file open cost.</p><p>CALL &lt;catalog name&gt;.system.rewrite_data_files(‘db.tablename’);<br>Options → strategy string Name of the strategy — binpack or sort. Defaults to binpack strategy</p><p><strong>binpack</strong> : Reorganizes data files by sorting the data based on specified columns. Optimizes data layout for query performance by enabling better compression and efficient range scans. Groups data files together without altering the order of the data within them. Targets files based on size thresholds defined in the table properties (e.g., write.target-file-size-bytes).</p><p><strong>sort: </strong>Reorganizes data files by sorting the data based on specified columns.<br>Optimizes data layout for query performance by enabling better compression and efficient range scans. Reads data from the existing files and rewrites it into new files sorted by the specified sort key(s).<br>Supports single or multiple columns as sort keys.</p><p>binpack or sort?</p><p>binpack: Choose this when the main problem is managing large numbers of small files or overly large files.<br>sort: Choose this when query performance can benefit significantly from sorted data or when specific filtering or range-based queries dominate workload patterns.</p><p><strong>rewrite_manifests</strong><br>Rewrite manifests for a table to optimize scan planning. Data files in manifests are sorted by fields in the partition spec. This procedure runs in parallel using a Spark job.</p><blockquote>CALL &lt;catalog name&gt;.system.rewrite_manifests(‘db.tablename’);</blockquote><p><strong>expire snapshots</strong><br>Iceberg tables maintain multiple snapshots, each representing a point-in-time view of the data. Snapshots include references to metadata, manifest files, and actual data files.</p><p>The expire_snapshots procedure can be used to remove older snapshots and their files which are no longer needed. This procedure will remove old snapshots and data files which are uniquely required by those old snapshots. This means the expire_snapshots procedure will never remove files which are still required by a non-expired snapshot.</p><p>When expire_snapshots is executed, it removes snapshots older than a specified time (retain_last or older_than). It identifies files referenced only by expired snapshots and deletes them. this doesnt remove the live data but rather removes the historical references of the older snapshots</p><p>Example: To retain last 40 snapshots</p><p><strong>CALL &lt;catalog name&gt;.system.expire_snapshots(‘db.tablename’, TIMESTAMP ‘2024–08–20 00:00:00.000’, 40);</strong></p><p>References :</p><p><a href="https://iceberg.apache.org/">https://iceberg.apache.org/</a></p><p>Iceberg slack channel</p><p>Please add on the comments, if there is any other aspect to be considered, while using Apache Iceberg with Apache Spark.</p><p>Thank you and don’t forget to 👏👏 if you find it useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d5d369b77cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An insight into spark.table]]></title>
            <link>https://medium.com/@deepa.account/an-insight-into-spark-table-fbb3df25a4cf?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbb3df25a4cf</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[dataframes]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Tue, 22 Oct 2024 05:38:40 GMT</pubDate>
            <atom:updated>2024-10-22T05:38:40.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JiayTzakd5-yXupKh8VDVg.png" /></figure><p>the <strong>spark.table()</strong> method is used to <strong>access a table</strong> or <strong>view</strong> stored in a metastore (like Hive metastore, or tables registered via catalogs such as Iceberg or Delta Lake) as a <strong>DataFrame</strong>. It allows you to interact with database tables or views in a Spark session without having to write SQL queries directly.</p><h3>spark.table()</h3><ul><li><strong>Retrieve a table</strong> from a metastore or catalog and return it as a Spark <strong>DataFrame</strong>.</li><li>It can be used to access <strong>Hive tables</strong>, <strong>temporary views</strong>, or any other tables registered in the current Spark session.</li><li>The returned <strong>DataFrame</strong> can then be used for further transformations, actions, or SQL queries in PySpark.</li></ul><h3>syntx:</h3><p>spark.table(table_name: str) -&gt; DataFrame</p><p>table_name: A string specifying the fully qualified name of the table. This can include the database name (e.g., &quot;database_name.table_name&quot;) or just the table name if it exists in the default database or session catalog.</p><p>Example:</p><pre># Using spark.sql()<br>df = spark.sql(&quot;SELECT * FROM employee&quot;)</pre><pre># Equivalent using spark.table()<br>df = spark.table(&quot;employee&quot;)</pre><h3>Predicate Pushdown and Projection Pruning (Optimized Query Execution)</h3><p>When you use spark.table(), Spark allows for optimizations like <strong>predicate pushdown</strong> and <strong>projection pruning</strong>:</p><ul><li><strong>Predicate Pushdown</strong>: If you apply a filter to a DataFrame (e.g., df.filter(df.salary &gt; 50000)), Spark attempts to &quot;push down&quot; the filter condition to the data source (e.g., Hive, Parquet, etc.). This means that only the relevant rows are loaded into memory, significantly improving efficiency by reducing the amount of data that needs to be processed.</li></ul><p><strong>Projection Pruning</strong>: If you select only specific columns from the DataFrame, Spark will only load the necessary columns, minimizing data I/O and memory usage. For example:</p><pre>pythondf = spark.table(&quot;employee&quot;)<br># Only the &#39;name&#39; column will be read, skipping unnecessary columns<br>df.select(&quot;name&quot;).show()</pre><p>When using SQL, you would need to ensure that the projection (columns to be selected) is explicitly written in the SQL query. If you don’t project the columns in SQL (e.g. when you useSELECT *), Spark may initially load more columns than needed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbb3df25a4cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A note on Iceberg and Writing Distribution Modes in Spark]]></title>
            <link>https://medium.com/@deepa.account/a-note-on-iceberg-and-writing-distribution-modes-in-spark-3b1000be8003?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b1000be8003</guid>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 14:23:02 GMT</pubDate>
            <atom:updated>2024-10-03T14:23:02.238Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EnW_RABkkOKDSgOwIetlBg.png" /></figure><p>Iceberg’s default Spark writers require that the data in each spark task is clustered by partition values. This distribution is required to minimize the number of file handles that are held open while writing. By default, starting in Iceberg 1.2.0, Iceberg also requests that Spark pre-sort data to be written to fit this distribution. The request to Spark is done through the table property<strong> `write.distribution-mode` with the value `hash`. </strong><br>Note, that Spark doesn’t respect distribution mode in CTAS/RTAS before 3.5.0.</p><p>Let’s go through writing the data :</p><pre>CREATE TABLE prod.db.sample (<br>    id bigint,<br>    data string,<br>    category string,<br>    ts timestamp)<br>USING iceberg<br>PARTITIONED BY (days(ts), category)</pre><p>To write data to the sample table, data needs to be sorted by `days(ts), category` but this is taken care of automatically by the default `hash` distribution. Previously this would have required manually sorting, but this is no longer the case.</p><pre>INSERT INTO prod.db.sample<br>SELECT id, data, category, ts FROM another_table</pre><p>There are 3 options for write.distribution-mode</p><p><strong>none — </strong>This is the previous default for Iceberg. This mode does not request any shuffles or sort to be performed automatically by Spark. Because no work is done automatically by Spark, the data must be manually sorted by partition value. The data must be sorted either within each spark task, or globally within the entire dataset. A global sort will minimize the number of output files. A sort can be avoided by using the Spark write fanout (<strong>write.spark.fanout.enabled </strong>Enables the fanout writer in Spark that does not require data to be clustered; uses more memory) property but this will cause all file handles to remain open until each write task has completed.</p><p><strong>hash</strong> — This mode is the new default and requests that Spark uses a hash-based exchange to shuffle the incoming write data before writing. <br>Practically, this means that each row is hashed based on the row’s partition value and then placed in a corresponding Spark task based upon that value.</p><p><strong>range</strong> — This mode requests that Spark perform a range based exchange to shuffle the data before writing. This is a two stage procedure which is more expensive than the hash mode. The first stage samples the data to be written based on the partition and sort columns. The second stage uses the range information to shuffle the input data into Spark tasks. Each task gets an exclusive range of the input data which clusters the data by partition and also globally sorts. While this is more expensive than the hash distribution, the global ordering can be beneficial for read performance if<br>sorted columns are used during queries. This mode is used by default if a table is created with a sort-order. Further division and coalescing of tasks may take place because of Spark’s Adaptive Query planning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b1000be8003" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Parquet The power of Columnar Storage]]></title>
            <link>https://medium.com/@deepa.account/apache-parquet-the-power-of-columnar-storage-a788663dfaf0?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/a788663dfaf0</guid>
            <category><![CDATA[apache-parquet]]></category>
            <category><![CDATA[parquet]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 13:31:19 GMT</pubDate>
            <atom:updated>2026-03-11T03:53:11.938Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2YJE6wjAxQj7dcKFkV0WLQ.png" /></figure><p>Apache Parquet is a columnar storage file format designed for efficient data processing in big data environments. It offers numerous advantages for data storage and analytics by optimizing data size and access performance. In this blog post, we’ll dive deep into what Apache Parquet is, its architecture, benefits, some disadvantages and how to use it effectively.</p><p>Apache Parquet is an open-source, columnar storage format for Hadoop ecosystems and beyond. Developed as a collaboration between Twitter and Cloudera in 2013, it quickly became one of the most efficient formats for handling large-scale data.</p><p>Parquet is particularly well-suited for analytics workloads and is widely used in distributed computing frameworks such as Apache Hadoop, Apache Spark, Apache Drill, and Presto.</p><p>Parquet’s columnar storage format offers significant benefits over traditional row-based formats like CSV, JSON, or Avro.</p><p>These include:</p><p>High compression rates: Since similar data types are stored together, Parquet allows for better compression, reducing storage costs and improving read/write performance.<br>Efficient scanning: By storing data by column rather than row, only the necessary columns are read, which speeds up queries, especially for analytical workloads.<br>Compatibility with complex data types: Parquet can handle nested structures like arrays, structs, and maps, making it ideal for semi-structured data.<br>Support for schema evolution: Parquet supports adding, removing, or modifying columns without the need to reprocess the entire dataset.</p><p><strong>Parquet File Structure</strong></p><p>Parquet’s file format is structured to store data in column chunks rather than row groups. This structure is organized into three levels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/1*Vxe8XIwyzsPAljEc6WC4Ww.png" /><figcaption>Parquet file structure</figcaption></figure><p>Row Group: A large chunk of data representing multiple rows. Row groups are independent of each other, making them ideal for distributed systems where different nodes process different chunks of data.<br>Column Chunk: Each row group is further divided into column chunks. A column chunk stores the data for a specific column in a row group.<br>Page: Each column chunk is divided into pages, which are the smallest unit of storage. Pages are usually of a fixed size (e.g., 1MB), and each page stores a portion of the data for a single column in a compressed and encoded format.</p><p>Each Parquet file contains the following sections:<br>File header (Magic number and metadata)<br>Data blocks (Organized as row groups and column chunks)<br>Footer (Schema, metadata, and index information)</p><p>The footer contains metadata that enables Parquet to retrieve only the required columns for a query.</p><p><strong>Data Compression and Encoding</strong></p><p>One of Parquet’s strengths is its support for various compression and encoding techniques, which can dramatically reduce the size of datasets.</p><p>Compression Algorithms: Parquet supports several compression algorithms like Snappy, Gzip, LZO, and Zstandard. Snappy is often preferred for its balance between speed and compression ratio.<br> <br>Encoding Schemes: Parquet uses advanced encoding techniques, including:<br> Run-Length Encoding (RLE): Stores consecutive identical values as a single value and a count.<br> Dictionary Encoding: Replaces repeating values with a dictionary reference to minimize storage requirements.</p><p><strong>Bloom Filter</strong></p><p>A Bloom filter is a compact data structure that overapproximates a set. It can respond to membership queries with either “definitely no” or “probably yes”, where the probability of false positives is configured when the filter is initialized. Bloom filters do not have false negatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*kDOsNKPYpxdbdzxF_q3jcg.png" /><figcaption>Bloom filter in parquet</figcaption></figure><p>Because Bloom filters are small compared to dictionaries, they can be used for predicate pushdown even in columns with high cardinality and when space is at a premium.</p><p>The Bloom filters are grouped by row group and with data for each column in the same order as the file schema. The Bloom filter data can be stored before the page indexes after all row groups.</p><p><strong>How Apache Parquet Works with Big Data Ecosystems</strong></p><p>Parquet integrates seamlessly with many big data frameworks:</p><p><strong>Apache Spark:</strong> Spark’s DataFrame API has built-in support for reading and writing Parquet files. Spark’s Catalyst optimizer can push down filters and projections to reduce I/O when querying Parquet data.<br> <br><strong>Apache Hive:</strong> Hive uses Parquet for efficient storage of large datasets. Hive’s Metastore can store metadata about Parquet files, making it easy to run SQL queries on Parquet data.</p><p>Apache Drill, Impala, Presto: These query engines natively support Parquet, enabling interactive and distributed querying on Parquet files with excellent performance.</p><p><strong>Performance Considerations</strong></p><p>When optimizing performance for Parquet, consider the following:</p><p><strong>Row group size</strong></p><p>A larger row group size can reduce the overhead of reading metadata but might increase memory consumption during queries. A good rule of thumb is to set the row group size between 512MB and 1GB.<br> <br>C<strong>ompression ratio:</strong></p><p>Choose a compression algorithm that balances the required compression ratio and speed. For example, Snappy offers fast compression with moderate size reductions, while Gzip provides better compression but is slower.</p><p><strong>Partitioning</strong>: In large datasets, partitioning by column (e.g., date) allows you to skip entire subsets of data, improving query performance.</p><h3><strong>Scenarios where parquet may not be right fit:</strong></h3><p><strong>Complex Schema Evolution</strong><br>Although Parquet supports schema evolution, handling schema changes (e.g., adding, removing, or modifying columns) can be complex and requires careful management. Schema evolution in Parquet involves handling backward compatibility, which means older files must remain readable even if new columns are added. This can introduce complexities when working with highly dynamic or frequently evolving schemas. Increased complexity in managing datasets with frequent schema changes, especially in systems with multiple data producers</p><p><strong>Not Ideal for Frequently Changing Data (Updates/Deletes)</strong><br>Parquet does not natively support updates or deletes within its files. Once data is written to a Parquet file, modifying or removing individual records is difficult. Parquet files are immutable, meaning that to perform an update or delete, you need to rewrite the entire file or use a more complex process involving compaction. Frequent updates or deletes require significant overhead, making Parquet less suitable for workloads where data needs to be modified or deleted regularly.</p><p><strong>File Size Considerations</strong><br>Due to its row group structure, Parquet files work best with large datasets. For smaller datasets, the file size overhead can be significant due to metadata, schema information, and other overheads. Parquet’s columnar format stores extra metadata for each row group and column, which introduces overhead. For small datasets, this metadata can take up a disproportionate amount of storage. Inefficient use of storage for smaller datasets compared to simpler formats like CSV or JSON.</p><p><strong>Difficulty in Handling Small Files</strong><br>Parquet is optimized for large files, and performance suffers significantly when dealing with numerous small files (e.g., generated by streaming systems or frequent batch jobs).Processing small Parquet files results in high overhead due to reading metadata and row groups, which affects performance. Poor performance and potential resource inefficiencies when working with many small files. This is often referred to as the “small file problem” in big data environments.</p><p>While Apache Parquet offers significant benefits for columnar storage and analytics workloads, it’s not a one-size-fits-all solution.</p><p><strong>References:</strong><br>Apache Parquet Documentation - <a href="https://parquet.apache.org/documentation/latest/">https://parquet.apache.org/documentation/latest/</a> <br><br>Schema Evolution in Parquet <a href="https://spark.apache.org/docs/latest/sql-data-sources-parquet.htmlschema-merging">https://spark.apache.org/docs/latest/sql-data-sources-parquet.htmlschema-merging</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a788663dfaf0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark Scenario Problem to find the optimal resource allocation]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@deepa.account/apache-spark-scenario-problem-to-find-the-optimal-resource-allocation-683e5eac2179?source=rss-96d6ee505787------2"><img src="https://cdn-images-1.medium.com/max/2600/1*0VauSFLSm9tyCeGCDgbuSA.png" width="6912"></a></p><p class="medium-feed-snippet">&#x1D00;&#xA731;&#xA731;&#x1D1C;&#x1D0D;&#x1D07; &#x28F;&#x1D0F;&#x1D1C; &#x29C;&#x1D00;&#x1D20;&#x1D07; &#x1D00; &#x1D05;&#x1D00;&#x1D1B;&#x1D00;&#xA731;&#x1D07;&#x1D1B; &#x1D0F;&#xA730; 500 &#x262;&#x299; &#x1D1B;&#x29C;&#x1D00;&#x1D1B; &#x274;&#x1D07;&#x1D07;&#x1D05;&#xA731; &#x1D1B;&#x1D0F; &#x299;&#x1D07; &#x1D18;&#x280;&#x1D0F;&#x1D04;&#x1D07;&#xA731;&#xA731;&#x1D07;&#x1D05; &#x1D0F;&#x274; &#x1D00; &#xA731;&#x1D18;&#x1D00;&#x280;&#x1D0B; &#x1D04;&#x29F;&#x1D1C;&#xA731;&#x1D1B;&#x1D07;&#x280;. &#x1D1B;&#x29C;&#x1D07; &#x1D04;&#x29F;&#x1D1C;&#xA731;&#x1D1B;&#x1D07;&#x280; &#x29C;&#x1D00;&#xA731; 10 &#x274;&#x1D0F;&#x1D05;&#x1D07;&#xA731;, &#x1D07;&#x1D00;&#x1D04;&#x29C; &#x1D21;&#x26A;&#x1D1B;&#x29C; 64 &#x262;&#x299; &#x1D0F;&#xA730; &#x1D0D;&#x1D07;&#x1D0D;&#x1D0F;&#x280;&#x28F; &#x1D00;&#x274;&#x1D05;&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@deepa.account/apache-spark-scenario-problem-to-find-the-optimal-resource-allocation-683e5eac2179?source=rss-96d6ee505787------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@deepa.account/apache-spark-scenario-problem-to-find-the-optimal-resource-allocation-683e5eac2179?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/683e5eac2179</guid>
            <category><![CDATA[sparkcore]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Tue, 03 Sep 2024 05:49:32 GMT</pubDate>
            <atom:updated>2024-09-03T05:49:32.679Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Difference between count(*) and count(Columname)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@deepa.account/difference-between-count-and-count-columname-88c911317cf0?source=rss-96d6ee505787------2"><img src="https://cdn-images-1.medium.com/max/1200/1*WZtSKVcXBzErdE3KuNKAOQ.png" width="1200"></a></p><p class="medium-feed-snippet">COUNT(*) will count the number of rows, while COUNT(expression) will count non-null values in expression and COUNT(column) will count all&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@deepa.account/difference-between-count-and-count-columname-88c911317cf0?source=rss-96d6ee505787------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@deepa.account/difference-between-count-and-count-columname-88c911317cf0?source=rss-96d6ee505787------2</link>
            <guid isPermaLink="false">https://medium.com/p/88c911317cf0</guid>
            <category><![CDATA[spark-sql]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Deepa Vasanthkumar]]></dc:creator>
            <pubDate>Tue, 27 Aug 2024 13:54:26 GMT</pubDate>
            <atom:updated>2024-08-27T14:02:39.526Z</atom:updated>
        </item>
    </channel>
</rss>