Deepa Vasanthkumar·Aug 18I Built a Zero-Cost SQL Explorer with React, DuckDB WASM, GitHub & VercelRun SQL on Parquet, CSV, and JSON directly in your browser — no database server, no Docker, no backend.A response icon3A response icon3
Deepa Vasanthkumar·Jul 20Pandas Arrow and PolarsThis Tiny blog shows the execution time comparison between pandas, polars using arrow from Spark Dataframe
Deepa Vasanthkumar·Mar 9Building Modern Lakehouse using Datafusion, Parquet and PlotlyA lakehouse combines the flexibility of a data lake with the structured analytics capabilities of a data warehouse.
Deepa Vasanthkumar·Dec 14, 2024Spark Iceberg and problem of Tiny filesJust a recap on Iceberg Table format and Layout:A response icon1A response icon1
Deepa Vasanthkumar·Oct 22, 2024An insight into spark.tablethe spark.table() method is used to access a table or view stored in a metastore (like Hive metastore, or tables registered via catalogs…
Deepa Vasanthkumar·Oct 3, 2024A note on Iceberg and Writing Distribution Modes in SparkIceberg’s default Spark writers require that the data in each spark task is clustered by partition values. This distribution is required to…A response icon1A response icon1
Deepa Vasanthkumar·Sep 16, 2024Apache Parquet The power of Columnar StorageApache Parquet is a columnar storage file format designed for efficient data processing in big data environments. It offers numerous…
Deepa Vasanthkumar·Sep 3, 2024Apache Spark Scenario Problem to find the optimal resource allocationᴀꜱꜱᴜᴍᴇ ʏᴏᴜ ʜᴀᴠᴇ ᴀ ᴅᴀᴛᴀꜱᴇᴛ ᴏꜰ 500 ɢʙ ᴛʜᴀᴛ ɴᴇᴇᴅꜱ ᴛᴏ ʙᴇ ᴘʀᴏᴄᴇꜱꜱᴇᴅ ᴏɴ ᴀ ꜱᴘᴀʀᴋ ᴄʟᴜꜱᴛᴇʀ. ᴛʜᴇ ᴄʟᴜꜱᴛᴇʀ ʜᴀꜱ 10 ɴᴏᴅᴇꜱ, ᴇᴀᴄʜ ᴡɪᴛʜ 64 ɢʙ ᴏꜰ ᴍᴇᴍᴏʀʏ ᴀɴᴅ…A response icon2A response icon2
Deepa Vasanthkumar·Aug 27, 2024Difference between count(*) and count(Columname)COUNT(*) will count the number of rows, while COUNT(expression) will count non-null values in expression and COUNT(column) will count all…