<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simbu on Medium]]></title>
        <description><![CDATA[Stories by Simbu on Medium]]></description>
        <link>https://medium.com/@dataxplorer?source=rss-a0dccc038e05------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lnAakSJWu2kS1Pq1lL5tYQ.jpeg</url>
            <title>Stories by Simbu on Medium</title>
            <link>https://medium.com/@dataxplorer?source=rss-a0dccc038e05------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:49:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dataxplorer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why do I like Dhoni so much?]]></title>
            <link>https://dataxplorer.medium.com/why-do-i-like-dhoni-so-much-83b27320c769?source=rss-a0dccc038e05------2</link>
            <guid isPermaLink="false">https://medium.com/p/83b27320c769</guid>
            <category><![CDATA[growth-mindset]]></category>
            <category><![CDATA[sports]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[mindfulness]]></category>
            <category><![CDATA[ms-dhoni]]></category>
            <dc:creator><![CDATA[Simbu]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 15:18:44 GMT</pubDate>
            <atom:updated>2026-11-17T17:26:10.551Z</atom:updated>
            <content:encoded><![CDATA[<p>Where do I even start? Twenty odd years of watching this man and I still don’t have a neat answer, haha.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*L5Z-eG8Cvwdj3o-n" /></figure><p>The swashbuckling wicketkeeper batsman who made his debut for India in 2004? A calm and composed captain who knew about his role to mentor youngsters? An observant leader who observes his players keenly and provides a platform for them to perform or rather express at their natural best? Knows the best way around to extract from his peers, more than they know themselves. This post is definitely not to showcase his achievements, but simply to show my admiration for the greatest guy I came across.</p><p>It’s actually not any of these tangible traits, but I will always like him (even adore/worship him, haha) for making me believe in a simple fact that even if you’re shy, introverted or with less energy or whatever, you can transform yourself the way you want. And out-perform in your life. Growing up watching Sachin play in TV, I always felt it would be extremely difficult to rise up, unless you have some God given talents, so much so that they would literally have to call you “God of cricket” or the “Master blaster”. But Dhoni is different and feels like one of us and that’s his power. He was hard working, but he had his mind and super strong self-belief that catapulted him to where he is now. And that’s a trait anyone can cultivate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*Vavg4n5SxONJfxmc" /></figure><p>He always likes to keep things simple, be mentally present in the moment, controls his anger/emotional outbursts, and believing himself when no one else would. These are not just for a cricketing venue, but your life. If you introspect, consistency and dedication along with these practices, you will see yourself becoming a better version. Being mentally present isn’t a cracking skill, the greatest performers across sports keep returning to this point. Novak Djokovic has obsessed over the mental side of Tennis more openly than almost anyone,</p><blockquote>I think the recovery is more important than actually working hard to stay in the present because it’s almost impossible to stay all the time focused… but then how quickly you go back… for me, conscious breathing is probably the one ingredient that I would take out is the most important.</blockquote><p>I could recall a countless times, the 7 time F1 champ Lewis stressed about the importance of being present to deliver qualy and race laps.</p><p>So its not always easy to be focused but being focused at the moment gets the maximum from you. And that’s pretty evident based on the players who play under him. Many tier-2, tier-3 players under him, felt extremely happy to be part of his squad and we see some of their performances were monumental in the victories he conquered (Joginder Sharma, Shardhul Thakur, amongst many). Stepping back from Dhoni, I always felt this to be quite the hardest part. Literally to be in the moment having control over your wandering mind is something the first step. Meditation and yoga helps you get there, but its important to stay there too. What’s that quote again, haha,</p><blockquote>the beginning is always the hardest and it gets better…</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/0*NivC2pRayLZ3y4bZ.jpg" /></figure><p>So these days, I am a bit cool headed, can easily let go of things, people without giving much of a thought. The old-me used to struggle with that. The important thing is to realize how fast you can get back on track after every fall you take. I also started embracing my quieter, more inward self. I used to think I’m following Robert Frost when it comes to the road less travelled or be my own unique rebel when it comes to certain life-decisions or thought process and never been part of any community. But now, I am completely adoring this phase of life, even more so with MS as my idol or the person to look up to.</p><p><em>P̶.̶S̶.̶:̶ ̶I̶’̶m̶ ̶n̶o̶t̶ ̶h̶e̶r̶e̶ ̶t̶o̶ ̶d̶e̶b̶a̶t̶e̶ ̶w̶h̶e̶n̶ ̶h̶e̶ ̶s̶h̶o̶u̶l̶d̶ ̶r̶e̶t̶i̶r̶e̶ ̶o̶r̶ ̶p̶l̶a̶y̶ ̶t̶h̶i̶s̶ ̶y̶e̶a̶r̶’̶s̶ ̶I̶P̶L̶,̶ ̶t̶h̶i̶s̶ ̶i̶s̶ ̶j̶u̶s̶t̶ ̶t̶o̶ ̶s̶h̶o̶w̶ ̶m̶y̶ ̶u̶n̶d̶e̶t̶e̶r̶r̶e̶d̶ ̶l̶o̶v̶e̶ ̶f̶o̶r̶ ̶t̶h̶e̶ ̶f̶i̶r̶s̶t̶ ̶p̶e̶r̶s̶o̶n̶ ̶o̶u̶t̶s̶i̶d̶e̶ ̶m̶y̶ ̶f̶a̶m̶i̶l̶y̶</em></p><p><em>Edit: He just announced his retirement today (17th Nov, 2026) and that’s it!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83b27320c769" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An intro to Hudi with MinIO — II]]></title>
            <link>https://dataxplorer.medium.com/an-intro-to-hudi-with-minio-ii-b5673677288b?source=rss-a0dccc038e05------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5673677288b</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[apache-hudi]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <dc:creator><![CDATA[Simbu]]></dc:creator>
            <pubDate>Wed, 05 Feb 2025 03:56:01 GMT</pubDate>
            <atom:updated>2025-02-05T03:56:01.040Z</atom:updated>
            <content:encoded><![CDATA[<h3>An intro to Hudi with MinIO — II</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pMx1ZFXlMnmbozlM.png" /></figure><p>Hello again, everyone! Continuing from where I left off last week, this time I’m here exploring the inner workings of Apache Hudi especially within the MOR table type. To start off, please ensure the below requisites are setup locally, in your machine if you want to build a Hudi lakehouse table.</p><p><strong>Prerequisites</strong></p><ol><li><a href="https://dl.min.io/server/minio/release/windows-amd64/minio.exe">MinIO</a> — for local object storage</li><li>Python (3.9+) — I tested with 3.9!</li><li>PyCharm (or any IDE you prefer)</li><li>Apache Spark (3.5.4) — for compute and cataloging (Hive)</li><li>Apache Hudi (1.0.0) — open data lakehouse</li><li>Spark Jars<br>- org.apache.hadoop:hadoop-aws:<strong>3.3.4</strong><br>- com.amazonaws:aws-java-sdk-bundle:<strong>1.12.262</strong><br>- org.apache.hudi:hudi-spark3.5-bundle_2.12:<strong>1.0.0</strong></li><li>Bikeshare data from <a href="https://www.kaggle.com/datasets/hanahorvatovic/bikeshare?resource=download">Kaggle</a></li></ol><p>If you’re using Windows OS, get the version of Hadoop in your spark installation directory (I used PyCharm’s virtual environment and hence I go to this directory where I created the PyCharm project -&gt; venv\Lib\site-packages), and search for hadoop through the site-packages folder. That will give you the exact version of hadoop you have installed, and I suggest you pick this version when you chose the spark jar or else you’ll face nightmare with your Spark-Hudi setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*jJqZC6wI7jtFNN-SaO2Baw.png" /></figure><p>The latest version of Apache Hudi is 1.0.0.</p><p>Since I’m using MinIO as local object storage for Hudi storage layer, and since it has similar semantics as AWS S3, we need to have a AWS jar, and specifically, AWS Java SDK bundle that’s compatible with Spark <strong>3.5.4</strong> and that is <strong>1.12.262</strong>, according to <a href="https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.3.4">this page</a>. (Credits to Koedit — <a href="https://stackoverflow.com/a/77339438">here</a>)</p><p>Now, let’s create the Spark and Hudi config as follows,</p><pre>## Spark config<br>{   <br>    &quot;spark.serializer&quot;: &quot;org.apache.spark.serializer.KryoSerializer&quot;,<br>    &quot;spark.sql.catalogImplementation&quot;: &quot;hive&quot;, <br>    &quot;spark.jars.packages&quot;: &quot;org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0&quot;,  # Prerequisited JARs<br>    &quot;spark.sql.extensions&quot;: &quot;org.apache.spark.sql.hudi.HoodieSparkSessionExtension&quot;,  # spark sql extension support<br>    &quot;spark.sql.catalog.spark_catalog&quot;: &quot;org.apache.spark.sql.hudi.catalog.HoodieCatalog&quot;,  # catalogue within hoodie<br>    &quot;spark.executor.memory&quot;: &quot;3g&quot;,  # adjust as you like<br>    &quot;spark.driver.memory&quot;: &quot;3g&quot;,# adjust as you like<br>    &quot;spark.executor.memoryOverhead&quot;: &quot;512m&quot;,# adjust as you like<br>    &quot;spark.executor.cores&quot;: &quot;2&quot;,<br>    &quot;spark.sql.shuffle.partitions&quot;: &quot;10&quot;,<br>    &quot;spark.default.parallelism&quot;: &quot;10&quot;,<br>    &quot;spark.memory.fraction&quot;: &quot;0.6&quot;,<br>    &quot;spark.sql.autoBroadcastJoinThreshold&quot;: &quot;10MB&quot;,<br>    &quot;fs.s3a.connection.ssl.enabled&quot;: &quot;false&quot;,<br>    &quot;spark.hadoop.fs.s3a.aws.credentials.provider&quot;: &quot;org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider&quot;,  # as we use AWS style access and secret keys to establish connection<br>    &quot;spark.hadoop.fs.s3a.endpoint&quot;: &quot;http://&lt;host-name&gt;:&lt;port-no&gt;&quot;,<br>    &quot;spark.hadoop.fs.s3a.access.key&quot;: &quot;&lt;your-access-key&gt;<br>    &quot;spark.hadoop.fs.s3a.secret.key&quot;: &quot;&lt;your-secret-key&gt;&quot;,<br>    &quot;spark.hadoop.fs.s3a.impl&quot;: &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;,<br>    &quot;spark.hadoop.fs.s3a.path.style.access&quot;: &quot;true&quot;  # if we use s3a://bucket/path/to/hudi/table format<br>}</pre><pre>HUDI_OPTIONS = {<br>    &quot;hoodie.table.name&quot;: &quot;bikeshare&quot;,  # table name for HUDI<br>    &quot;hoodie.datasource.write.table.type&quot;: &quot;MERGE_ON_READ&quot;,  # COPY_ON_WRITE<br>    &quot;hoodie.datasource.write.recordkey.field&quot;: &quot;ride_id&quot;,  # primary key<br>    &quot;hoodie.datasource.write.precombine.field&quot;: &quot;started_at&quot;, # to identify the latest version of a row<br>    &quot;hoodie.database.name&quot;: &quot;hudi_db&quot;,  # optional database for the Hudi table<br>    &quot;hoodie.write.concurrency.mode&quot;: &quot;OPTIMISTIC_CONCURRENCY_CONTROL&quot;,  # default is OCC, where the transactions gets rolled back only when a conflict is detected<br>    &quot;hoodie.datasource.write.partitionpath.field&quot;: &quot;year&quot;,<br>    &quot;hoodie.datasource.write.operation&quot;: &quot;upsert&quot;,  # can be insert, bulk_insert, upsert<br>    &quot;hoodie.parquet.max.file.size&quot;: &quot;26214400&quot;,  # 25MB<br>    &quot;hoodie.parquet.block.size&quot;: &quot;26214400&quot;,# 25MB<br>    &quot;hoodie.copyonwrite.record.size.estimate&quot;: &quot;512&quot;,<br>    &quot;hoodie.parquet.compression.codec&quot;: &quot;snappy&quot;,<br>    &quot;hoodie.clean.automatic&quot;: &quot;true&quot;,<br>    &quot;hoodie.clean.policy&quot;: &quot;KEEP_LATEST_COMMITS&quot;,<br>    &quot;hoodie.cleaner.commits.retained&quot;: &quot;1&quot;,  # for time travel, how many commits to go back to, increase as per need<br>    &quot;hoodie.compact.inline&quot;: &quot;false&quot;,<br>    &quot;hoodie.metadata.enable&quot;: &quot;true&quot;,<br>    &quot;hoodie.index.type&quot;: &quot;RECORD_INDEX&quot;,<br>    &quot;hoodie.metadata.record.index.enable&quot;: &quot;true&quot;,<br>    &quot;hoodie.record.index.update.partition.path&quot;: &quot;RECORD_INDEX_UPDATE_PARTITION_PATH_ENABLE&quot;,<br>    &quot;hoodie.metadata.column.stats.enable&quot;: &quot;true&quot;,  # enabling column level stats<br>    &quot;hoodie.metadata.partition.stats.enable&quot;: &quot;true&quot;,<br>    &quot;hoodie.bloom.index.enable&quot;: &quot;true&quot;,<br>    &quot;hoodie.metadata.expression.index.enable&quot;: &quot;true&quot;,<br>}</pre><p>Apache Spark is the simple solution for interacting with Hudi, however, it comes with its own set of problems that can be contained by choosing the correct versions of the dependencies I shared earlier. Since I previously used <a href="https://medium.com/@dataxplorer/a-lean-data-stack-idea-3abee44ae9a1">DuckDB on MinIO</a>, I thought it would be able to read the Hudi table (along with its metadata) as well, however, it currently supports only Iceberg and Delta tables. That leaves Spark as the best option to read and write Hudi tables when dealing with a local node. And for catalog, I used Hive catalog that come along with the Spark for handling the metadata through the config. Again, I could explore on setting up an alternate catalogs as well, but its beyond the scope of this exploration.</p><p>Next, let’s explore both the MOR (Merge on Read) table type in action, do some upserts and see how the metadata changes with the impact on the transaction timelines. In the bikeshare dataset, the schema goes like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4_yJQ7W-B9NpNbwDY4Idw.png" /><figcaption>schema of the bikeshare data</figcaption></figure><p>ride_id is the unique key in this dataset, and we can use started_at as precombine.field for finding the latest version of a record as its a timestamp field, and let’s use the partition column using year(started_at)</p><pre>def create_table(self, source_path: str, target_path: str):<br>        &quot;&quot;&quot;<br>        Create a Hudi table by reading data from the source path and writing to the target path.<br>        source_path: Input directory of files to read from Spark<br>        target_path: Output directory to create the Hudi table path<br>        &quot;&quot;&quot;<br>        df = self.spark.read.csv(source_path, header=True, inferSchema=True)<br>        df = df.withColumn(&quot;started_year&quot;, f.year(&quot;started_at&quot;))<br>        p_keys = bc.HUDI_OPTIONS[&#39;hoodie.datasource.write.recordkey.field&#39;]<br>        df = df.dropDuplicates([p_keys])  # deduplication in case any duplicate ride_id exists in the given files<br>        self._write_hudi(df, target_path, operation=&#39;insert_overwrite_table&#39;)<br><br>def update_table(self, source_path: str, target_path: str):<br>        &quot;&quot;&quot;<br>        Update a Hudi table by reading incremental data from the source path.<br>        source_path: Input directory of files to read from Spark<br>        target_path: Output directory to create the Hudi table path<br>        &quot;&quot;&quot;<br>        df = self.spark.read.csv(source_path, header=True, inferSchema=True)<br>        df = df.withColumn(&quot;started_year&quot;, f.year(&quot;started_at&quot;))<br>        self._write_hudi(df, target_path, operation=&#39;upsert&#39;)<br><br>def _write_hudi(self, df: DataFrame, target_path: str, operation: str):<br>        &quot;&quot;&quot;<br>        Helper method to write data to Hudi.<br>        &quot;&quot;&quot;<br>        hudi_write_options = self.hudi_options.copy()  # copy the HUDI options<br>        hudi_write_options[&#39;hoodie.datasource.write.operation&#39;] = operation  # &#39;insert_overwrite_table&#39;<br>        if operation == f&#39;insert_overwrite_table&#39;:  # for creating the new table<br>            df.write.format(&#39;hudi&#39;).options(**hudi_write_options).mode(&#39;overwrite&#39;).save(target_path)<br>        else:  # for upserting the existing table<br>            df.write.format(&#39;hudi&#39;).options(**hudi_write_options).mode(&#39;append&#39;).save(target_path)<br><br>hudi_manager.create_table(src_path, tgt_path)    <br>hudi_manager.update_table(src_upd_path, tgt_path)</pre><p>One constraint I noticed when trying with a <strong>2g</strong> memory for driver and executor, especially for a MOR table is that the upsert operation often failed due to Java heap memory issue, prompting me to raise the memory threshold to <strong>3g</strong>. Thereby, I had to split the initial load and the incremental load into multiple runs which allowed me to take a peek at the hoodie directory, especially timeline, metadata/files path</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*ztzNCc2DS9HEg5DcU99GGw.png" /><figcaption>.hoodie/timeline contents post initial commit</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*d_0A7HgchzUFDETzie6g5g.png" /><figcaption>.hoodie/metadata/files contents post initial commit</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Tw7FYHrmSgCUIKWKohjDQ.png" /><figcaption>Hudi table path — consisting of 2 partitions — initial state</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*spSiKxsT_hEBfQFv8XdfpA.png" /><figcaption>2023 partition path and its base files along with .hoodie_partition_metadata</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxkLIl-1hVeaO60lOgzKlw.png" /><figcaption>Querying (Snapshot mode by default) the Hudi table for a ride_id — F74DF9549B504A6B</figcaption></figure><p>These folders contain the metadata about the list of files, along with its partitions, indexes, if any, and log files for a query engine to query the table. Now, I modified the source file for a select ride_id to ensure, I’m doing an update on the existing keys, and changed the started_at so that we will notice the change in partition for these ride_id. And after I run the update (basically changed the started_at to 2025 so I could notice a new partition for 2025 gets created), we notice the changes in metadata as follows,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*gDJO9sObAWmdOvA_E7rCcw.png" /><figcaption>Hudi table path — with a new 2025 partition added — after update</figcaption></figure><p>Yay! The upsert worked by creating a new partition for 2025! And if I go into this newly created partition, I could notice a slightly lesser base file (around 430 KiB, which can be a bit heavy for small # of rows) which is expected because the original base file — containing this particular record (ride_id = F74DF9549B504A6B) is around 4MB</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*CLHQyBxVebMgUWAlK3TNWg.png" /><figcaption>the updated record now moved to 2025 partition</figcaption></figure><p>Now, looking into the original partition directory, i.e., 2023/</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xBC949EkqyVLWxxawTE91g.png" /><figcaption>he delta log file — 1.5 KiB is the delete marker</figcaption></figure><p>Now, an interesting take is that the 2023 older partition, consisting of 4 base files earlier now has a new file with the name .e6492f90–2098–4bcb-b89e-824a03c0fbc5–0_20250126162917570.log.1_0–32–487 This is like a delete marker type avro-log file, since we updated the started_at from 2023 to 2025. In short, this record when queried by any query engine will be ignored and instead, the new record from the 2025 partition would be picked up for this ride_id</p><blockquote>Note: e6492f90–2098–4bcb-b89e-824a03c0fbc5–0_114–19–178_20250126162059363.parquet is the original base file which contains this particular row [amongst others]. Hence the log file, a.k.a the delete marker in this case, also starts with the same name suffixed by the log-id to identify the transaction.</blockquote><p>So the takeaway here, is for incessant upserts or streaming ingestion scenarios, MOR table type would provide the best compromise for building super fast writes at the same time, optimizing the file size for better query performance on read (through table services like compaction).</p><p>Another interesting feature provided by Apache Hudi is the <strong>RECORD_INDEX</strong> type of index. Its an indexing strategy in Hudi that maintains a mapping between individual <strong>record keys</strong> (unique identifiers) and <strong>partition keys </strong>and their <strong>physical locations</strong> (file paths and offsets) in the data files. And a quick peek at the size of this directory (.hoodie/metadata/record_index) is around 96 MB, while the entire dataset is about 500MB, roughly occupying 20% of the size. I hope the future release will do something about this, though it’s not a big deal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8n5do9RJAainu2_8ZvLHg.png" /><figcaption>the contents of the record_index directory</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbYi6Wxg7Zb3Tn_yOVJYJg.png" /><figcaption>Query result — record changed to 2025 partition as expected</figcaption></figure><p>But we’ve to consider a scenario when there are too many small¹ files getting accumulated to a base file for every upsert transaction, often leading to poor read performance, especially in the case of MOR tables. To tackle this, Hudi comes up with a few table services, viz., <strong>compaction</strong> and <strong>cleaning</strong>.</p><p><strong>Compaction²</strong>— helps in mitigating this problem by merging the log files to their respective base files, which leads to rewrite of the base file, either through asynchronous mode or fired inline as part of every commit transaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MbbnikuGwE5MgLxF.jpg" /><figcaption>Base file — before and after compaction</figcaption></figure><p>While this latest file post compaction looks clean, however, the older files still persists on the storage (and is useful for time-travel queries, again, configurable to suit our needs). Now, if our system ends up with way too many of these files, that doesn’t exist in the metadata index anymore, these are often referred as <em>ghost files</em> and they occupy unnecessary space. Hence a <strong>cleaner³</strong> service is an essential service provided by Hudi to remove these ghost files that doesn’t belong to Hudi’s timeline anymore and are safe to be removed. Both these can be invoked as a standalone command as well,</p><pre>hudi_db = bc.HUDI_OPTIONS[&#39;hoodie.database.name&#39;]  # name of your db, per the catalog<br>hudi_tbl = bc.HUDI_OPTIONS[&#39;hoodie.table.name&#39;]# name of your Hudi table, per the catalog<br><br>spark_sess.sql(f&quot;CALL run_compaction(op =&gt; &#39;run&#39;, table =&gt; &#39;{hudi_db}.{hudi_tbl}&#39;)&quot;)<br>spark_sess.sql(f&quot;CALL run_clean(table =&gt; &#39;{hudi_db}.{hudi_tbl}&#39;, clean_policy =&gt; &#39;KEEP_LATEST_COMMITS&#39;)&quot;)</pre><p>While all the above said are applicable only for MOR table, the behavior of COW table isn’t much different either, except that there is no concept of log/delta files as for every upsert/DMLs, the entire base file gets rewritten, at the cost of write latency, though keeping the read performance intact.</p><p>Querying through the <strong>Snapshot</strong> view — to get the latest value of a record,</p><pre>def read_hudi_table(self, target_path, query_type=&quot;snap&quot;, query_ts=None):<br>    if query_type == f&quot;snap&quot;:  # snapshot query <br>        return self.spark.read.format(&quot;hudi&quot;)\<br>            .option(&quot;hoodie.datasource.query.type&quot;, &quot;snapshot&quot;)<br>            .load(target_path)<br>    elif query_type == f&quot;time&quot;:  # time-travel query<br>        return self.spark.read.format(&quot;hudi&quot;)\<br>            .option(&quot;hoodie.datasource.query.type&quot;, &quot;incremental&quot;)\<br>            .option(&quot;hoodie.datasource.read.begin.instanttime&quot;, query_ts)\<br>            .load(target_path)</pre><p>The above code snippet, hoodie.datasource.query.type = snapshot, is the default querying mode in Hudi. It instructs the query engine to use snapshot query mode to fire the query for its result. Alternatively, to use the time-travel query type, you just have to use the keyword, timestamp as of &#39;yyyy-mm-dd hh:mi:ss.sss’ at the end of your SQL FROM keyword before you place WHERE conditions, if any so spark knows this is a time-travel query.</p><p><em>Note: For time-travel queries to work as expected, please ensure you’ve set the below Hudi configs as,</em></p><pre>&quot;hoodie.clean.policy&quot;: &quot;KEEP_LATEST_COMMITS&quot;,  # keeping the latest file - KEEP_LATEST_COMMITS<br>&quot;hoodie.cleaner.commits.retained&quot;: &quot;5&quot;,  # for time travel, how many commits to go back to<br>&quot;hoodie.clean.automatic&quot;: &quot;false&quot;,  # inline cleaning disabling to show changes</pre><p>I hope this gives you a basic understanding of how things work underneath Apache Hudi. The full code I tried is available in git <a href="https://github.com/simbu175/MinIO_DuckDB_Hudi">here</a>.</p><p>Let me know if I have missed to cover anything! Looking to hear from y’all!!</p><p><strong>References</strong></p><ol><li><a href="https://medium.com/@simpsons/apache-hudis-small-file-management-17d8c61b20e6">https://medium.com/@simpsons/apache-hudis-small-file-management-17d8c61b20e6</a></li><li><a href="https://medium.com/@simpsons/apache-hudi-compaction-6e6383790234">https://medium.com/@simpsons/apache-hudi-compaction-6e6383790234</a></li><li><a href="https://medium.com/@simpsons/cleaner-and-archival-in-apache-hudi-9e15b08b2933">https://medium.com/@simpsons/cleaner-and-archival-in-apache-hudi-9e15b08b2933</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5673677288b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An intro to Hudi with MinIO — I]]></title>
            <link>https://dataxplorer.medium.com/an-intro-to-hudi-with-minio-i-75536fe75b4c?source=rss-a0dccc038e05------2</link>
            <guid isPermaLink="false">https://medium.com/p/75536fe75b4c</guid>
            <category><![CDATA[apache-hudi]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[introduction]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[minio]]></category>
            <dc:creator><![CDATA[Simbu]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 06:00:26 GMT</pubDate>
            <atom:updated>2025-01-29T06:00:26.798Z</atom:updated>
            <content:encoded><![CDATA[<h3>An intro to Hudi with MinIO — I</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FDpyQ-E_sqhkBqCy.png" /><figcaption>MinIO + Hudi</figcaption></figure><p>As I keep diving into the open-source wonders of the modern data stack, this time, I landed on something fascinating — Apache Hudi. It’s a complex, yet powerful open lakehouse table format (OLTF), built for lightning-fast upserts, supporting multi-modal and record-level indexing, and top-notch concurrency for reads and writes (across multiple engines!). In this post, I’ll walk you through the fundamentals, features, setup, and a hands-on use case — running Apache Hudi locally with MinIO for storage and Spark for compute.</p><p>First things first, what is a <strong>lakehouse</strong>? According to this awesome <a href="https://hudi.apache.org/blog/2024/07/11/what-is-a-data-lakehouse/">blog</a>¹ by <a href="http://linkedin.com/in/dipankar-mazumdar">Dipankar</a>, “A data lakehouse is a hybrid data architecture that combines the best attributes of data warehouses and data lakes to address their respective limitations. This innovative approach to data management brings the transactional capabilities of data warehouses to cloud-based data lakes, offering scalability at lower costs.” In simple terms, its a perfect concoction of the features offered by almost infinitely scalable cloud storage of data lakes, with the data management features of a typical data warehouse on top, essentially providing us with ACID features, partitioning, clustering, time-travel, concurrency controls, etc.</p><p>And what about <strong>table formats</strong>? A key component of a lakehouse architecture is the table format, which acts as a metadata layer above file formats like Apache Parquet or Apache ORC. Essentially this layer allows different engines to concurrently read and write on the same dataset, supporting ACID transactions. Notable table formats include <a href="https://hudi.apache.org/">Apache Hudi</a>, <a href="https://iceberg.apache.org/">Apache Iceberg</a>, and <a href="https://delta.io/">Delta Lake</a> and these are open sourced (Delta lake a proprietary version too) and hence they’re open table formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MwH7hxYt8-r-c_2L.png" /><figcaption>Layers within the typical data lakehouse¹</figcaption></figure><p><strong>Comparison of OLTF</strong></p><p>As the world of <strong>open lakehouse table formats</strong> evolves, three major players — <strong>Apache Hudi, Delta Lake, and Apache Iceberg</strong> stand out. Each of these table formats aims to bring <strong>ACID transactions, schema evolution, and time-travel capabilities</strong> to data lakes, but they differ in approach and use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*mzaMo52FNUqlVoSvV7iyqw.png" /><figcaption>Feature comparison between Hudi, Iceberg and Delta Lake</figcaption></figure><p><strong>Why Hudi?</strong></p><p>If you’re fascinated by fast upserts, incremental data processing, and real-time analytics, then Apache Hudi is the way to go! Its strong record-level indexing and CDC support make it a great choice for dynamic datasets that require frequent updates. Of course, Delta Lake and Iceberg are strong in their own domains, but if your goal is to efficiently manage constantly evolving datasets without rewriting entire partitions, Hudi stands out!</p><p>Introduction to <strong>Apache Hudi</strong></p><p>Apache Hudi¹ (pronounced as Hoodie) is short for “Hadoop Upserts Deletes and Incrementals”, is an open source <a href="https://hudi.apache.org/docs/hudi_stack">transactional data lakehouse platform</a> built around a database kernel. It provides table-level abstractions over open file formats like Apache Parquet and ORC thereby delivering core warehouse and database functionalities directly in the data lake with the support of transactional capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8UnVgHc4nB1V-1fc.png" /><figcaption>Hudi format layout² — a pictorial representation of read/write workflow</figcaption></figure><p><strong>Components of Hudi tables</strong></p><ul><li>Base files<br> — actual <strong>data</strong> files, usually of the type Parquet / ORC</li><li>Log files<br> — these are the <strong>delta files</strong>, usually stored in Avro format<br> — they capture the incremental <strong>changes</strong>, aka upserts<br> — applicable only for Merge On Read (MOR) table types</li><li>Metadata files<br> — a catalog that keeps track of data and how it’s organized<br> — includes information like file paths, record keys, and partition details</li><li>Timeline<br> — this is a <strong>time-based log</strong> of all actions performed on the table<br> — helps Hudi maintain <strong>ACID transactions</strong> <br> — enables features like <strong>time travel</strong></li><li>Indexes<br> — <strong>mapping</strong> of record keys and their physical locations [base or log] files<br> — helps Hudi quickly locate records during <strong>upserts</strong> and <strong>deletes<br> — </strong>supports <strong>multi-modal</strong> (Non-PK columns), expression based, <strong>record-level</strong> indexing</li><li>Partitions<br> — <strong>logical divisions</strong> of data, often based on a date/categorical column<br> — help organize data for faster queries and better management.</li><li>Table services<br> — these are services that run asynchronously (configurable)<br> — <strong>Compaction</strong>: merges upsert <strong>log files</strong> into base files, for optimizing storage and query performance<br> — <strong>Cleaning</strong>: removes <strong>old or unused files</strong> to free up space</li></ul><p>There are two <strong>table types</strong> supported by Hudi, viz.,</p><ul><li><strong>Copy On Write</strong> (COW) <br> — Updates and deletes rewrite the entire data file. <br> — Best for <strong>read-heavy workloads</strong> where query performance is critical.<br> — Hence no external table services aren’t necessary!</li><li><strong>Merge On Read</strong> (MOR)<br> — Updates and deletes are stored in separate log files. <br> — Best for <strong>write-heavy workloads</strong> where fast writes are more important than immediate query performance.</li></ul><p><strong>Contents of Hoodie Metadata</strong></p><p>For a Hudi table having inserts, updates, deletes transactions, then its timeline would look like,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SS7cFBp_LOrrNE_m.png" /><figcaption>Internal representation of a Hudi metadata²</figcaption></figure><p>You could see every transaction will have its own timeline based logs, that tells whether a particular transaction was successful to commit (and checks for other concurrency based conflicts, in case of multiple concurrent writers). For example, below is the screenshot from my Hudi table, after I first created the table with some records, followed by an update.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xEHIBxsKmS5wN5ExoSe9xw.png" /><figcaption>Inside the hoodie metadata folder</figcaption></figure><p>This Hudi table is of MOR type, so we could see 3 sets of files per transaction, viz., to determine if the transaction went through without any conflicts. These are essential if multiple writers are writing for a table (say from multiple engines) to ensure that each of the transaction is processed with any conflicts. In the event of any conflict being deducted, Hudi automatically rolls back those transaction that couldn’t be committed, hence preserving the ACID nature of a classic OLTP database model.</p><ul><li>&lt;transaction_id&gt;.[replace/delta]<strong>commit.requested</strong> —A request to replace old/insert new data files has been initiated.</li><li>&lt;transaction_id&gt;.[replace/delta]<strong>commit.inflight</strong> — The replacement process is currently in progress, and the new data files are being written. Like a lock in data warehouse, but has added features.</li><li>&lt;transaction_id&gt;.[replace/delta]<strong>commit</strong> — The transaction is completed and marked successful in the metadata/catalog, and the new data files are now active and visible for queries.</li></ul><p>And the base files (parquet) are stored in the partition folder (if applicable) along with its metadata,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ri_PYuTUm5WBT4_TpI0Zjg.png" /><figcaption>A base file with its partition metadata</figcaption></figure><p>The metadata file gets automatically created by Hudi in the partition, “.hoodie_partition_metadata” which has the info about this partition along with the commit time and the depth of the partition as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*2CMtIjtQNYs_8FXSobdkXg.png" /><figcaption>.hoodie_partition_metadata file contents</figcaption></figure><p><strong>Query types </strong>that are supported by Hudi</p><ul><li><strong>Snapshot query</strong><br> — Shows the latest snapshot of the table as of the latest commit.<br> — Similar to SQL queries running on a table in database.<br> — Hudi storage engine accelerates these snapshot queries with indexes whenever possible, on supported query engines.</li><li><strong>Incremental query (Latest state)</strong><br> —Displays only return new data written to the table since an instant on the timeline.<br> — Provides latest value of records inserted/updated (i.e. 1 record output by query for each record key), since a given point.<br> — Can be used to “diff” table states between two points in time.<br> — Can save costs by a lot when dealing with incremental refresh cases</li><li><strong>Incremental query (CDC)</strong><br> — These provides database like change data capture streams out of Hudi tables. (slightly different from the previous type)<br> — Output of a CDC query contain records inserted or updated or deleted since a point in time or between two points in time.</li><li><strong>Read optimized query (only MOR)</strong><br> — These provides excellent snapshot query performance via the base files.<br> — Typically a compaction strategy aligns with a transaction boundary, to provide older consistent views of table/partitions.<br> — This is useful to integrate Hudi tables from data warehouses that typically only query columnar base files as external tables</li><li><strong>Time-travel query<br> </strong>— Shows a snapshot of a table as of a given instant in the past.<br> — Helps access multiple versions of a table on instants in the active timeline or save-points in the past.<br> — Can be used through a timestamp or a commit version</li></ul><p>And that’s a wrap for Part 1 of our deep dive into Apache Hudi! But we’re just getting warmed up. Next time, we’ll roll up our sleeves and get hands-on — setting up a Hudi table, unraveling what really happens when you upsert an existing key, and decoding how Hudi tracks metadata, specifically for MOR tables. Oh, and we’ll also see how this whole process shakes things up for COW tables. Stay tuned — it’s about to get interesting!</p><p><strong>References</strong></p><ol><li><a href="https://hudi.apache.org/blog/2024/07/11/what-is-a-data-lakehouse/">https://hudi.apache.org/blog/2024/07/11/what-is-a-data-lakehouse/</a></li><li><a href="https://blog.min.io/streaming-data-lakes-hudi-minio/">https://blog.min.io/streaming-data-lakes-hudi-minio/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75536fe75b4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Still I rise — Lewis Hamilton]]></title>
            <link>https://dataxplorer.medium.com/still-i-rise-lewis-hamilton-1d0811770924?source=rss-a0dccc038e05------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d0811770924</guid>
            <category><![CDATA[still-i-rise]]></category>
            <category><![CDATA[f1]]></category>
            <category><![CDATA[lewis-hamilton]]></category>
            <dc:creator><![CDATA[Simbu]]></dc:creator>
            <pubDate>Sat, 18 Jan 2025 03:05:43 GMT</pubDate>
            <atom:updated>2025-01-18T03:05:43.637Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Lewis phenomenon</h3><p>What is arguably one of the very few thing that didn’t change since the pandemic? A Mercedes star dominating F1. Since the introduction of V6 hybrid engines in 2014 and till 2019, this guy won a mindboggling 4 out of 5 drivers’ titles. Though since 2022, RedBull is slowly cementing its technical superiority and last year by the arch rivals, McLaren and Ferrari!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*qDpCclfftINC1Bhq.jpg" /><figcaption>An enigmatic “Still I Rise”</figcaption></figure><p>Between 2014 and 2020, Mercedes is unbeaten in both World Championships (Driver and Constructor) while winning the constructors in 2021 and losing the other title by a hair’s width (let’s not go into that #AD21 controversy now, alright?). This is something that has never happened before. The previous best streak was from 1999–2004 achieved by Ferrari under the legendary Michael Schumacher (I could even go as far as to call him the God of F1). Every time the opponents raised the bar; the champions always find a way to push themselves to new heights across all the departments, (race craft, technical feedbacks, communication, strategy, pressure handling, etc) constantly perfecting their machinery. What I like about this amazing team is that their philosophy of “blaming the process and not the person” for failures is a powerful message not only to everyone at the F1 paddock, but also a life lesson to many of us. This is something that paved way for their engineers to innovate at a much cognitive level than their rivals.</p><p>Some of their innovations include (that I remember):</p><ul><li>Split Turbo — V6 Turbo (2014) — the most dominant engine of the hybrid era, ultimately copied by all other PU makers (barring Ferrari)</li><li>Hydraulic rear-suspensions (2016) — a trick that helped them lower their car in straights</li><li>Long wheelbase design (2017) — helps in pushing the sidepods further from the wake of front tires &amp; excellent traction in high speed corners</li><li>T-wings and Shark fins (alongside Ferrari in 2017) — better and stable rear downforce to align the car during slow speeds</li><li>Underneath nose-cape (2018) — better aero flow underneath the nose of the car, also aide the placement of the front suspensions</li><li>Dual Axis Steering a.k.a DAS (2020) — this is so the height of their engineering to quickly warm up the tire temps, that made the governing body to ban it for the subsequent years.</li><li>Zero pod sidepods (2022) — though not successful as their CFD model proved.</li></ul><p>I could write more about their consistent-strive to reach the pinnacle and harmonious work execution week-in week-out, but the 7 trophies lined in their head office wouldn’t be possible without the MAN with the 7 stars decked on his helmet. He delivered consistently without losing motivation, dominating his teammate and the rest of the pack. Lewis Hamilton, the driver, who has gone on to smash most of the existing records (including equaling Michael’s record of being a 7-time world champion) as well as creating new ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/0*8WOKAl2zvzMt4vGH" /><figcaption>The almost unbeatable records of Sir Lewis Hamilton — copy right from X.com</figcaption></figure><p>His champion-esque qualities both on and off the field include:</p><ul><li>Self-belief to win from any circumstance or at least to fight for wins almost at all the races (stats about wins and podiums v finishes).</li><li>Slow built-up from Q1 to the last lap of the race (Very few of his victories were from extreme circumstances, Turkey-2020; Germany-2018)</li><li>Extremely talented (Was only beaten twice — 2011 and 2016, equaled with Alonso on 2007)</li><li>Consistency: Winning at least one race at every championships he competed at (this streak ended in 2022)</li><li>Level Headed under intense pressure especially on-track moments (Monza-2018, Spain-2021)</li><li>Dramatic: Reserves his best at the very last moment to spoil opponent’s party (Singapore 2018 Q3)</li><li>Unafraid to address real-world issues by wearing BLM/Avoid Racism T-shirt on F1 podiums</li><li>Setting up various scholarships to providing support for struggling upcoming drivers who are marginalized due to the lack of funds.</li></ul><p>And the list goes on. Aside from his superior racing skills, his altruism has cemented him forever in people’s hearts.</p><p>He influenced Mercedes’ decision in promoting diversity within the team. I mean, how often do you see a motorsport racing driver, who is upbeat about winning races/championships, think about improving diversity within the team? It’s a no-brainer Lewis is the central pillar around which the Mercedes team was built over the years and it’s so satisfying to see them adopting such diversity programs with a futuristic vision to promote talents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*avVFvw6bHUqd5zYr" /><figcaption>W11 — the modern peak of engineering; the most grippiest car (fastest on corners)</figcaption></figure><p>Taking the knee for “End Racism” and “Black Lives Matter” — Apart from the Covid pandemic in 2020 has given voice to one other major issue, unfair treatment to underprivileged people. Before the start of every race in 2020, we saw a unique pattern with drivers lining up and wearing the shirt at the grid, which either said “END RACISM” or “BLACK LIVES MATTER”. What’s more encouraging is how he puts his power to great use in spreading the need to bring these incidents down by continually fighting against such notions and urging his fellow companions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*M2EcwOkouQkC_45N.jpg" /><figcaption>Lewis &amp; Seb — taking the knee together before the start of a race</figcaption></figure><p>Raising awareness about animal cruelty. I have been following his Instagram account very closely over the years and I have seen many of his stories and posts are about how animals are ill-treated for meat, ornaments, and few odd entertainments. It is very impressive that he holds himself to a high standard by going vegan for their sake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c_dwyZZ89c6rvyt8" /><figcaption>A post about pig slaughter</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*9RSCIGmc3kf4SFsR" /><figcaption>Cruelty to cows — amid his choice for being a happy Vegan</figcaption></figure><p>Educating about disposal of plastics and the harm it poses to our environment. As we are all well aware, plastics don’t degrade easily, even the so-called bio-degradable plastics do not entirely breakdown. I am glad he is using the voice he has built on various social media platforms to highlight these environmental issues. I believe that’s his greatest power.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/0*8Pwem18nFUp-oD1j" /><figcaption>Plastics and its harmful decomposition</figcaption></figure><p>Impact of deforestation and global warming. Over the past couple of years, few major issues have been related to the rapid deforestation of the Amazon forests and the immense Australian bush fires. He was at the forefront of leading the protest against the Amazon deforestation and showcased his generosity during the Australian bush fires by donating half a million dollars. This amount may be a small portion of his wealth but it inspired many others to donate generously as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*6Evr8XKgb7X3Ky7e" /><figcaption>Australian wildfires — needs no introduction</figcaption></figure><p>Way to go champ. Some bonuses,</p><ul><li><a href="https://www.youtube.com/watch?v=s7DJ1rWW9C4">Team X44</a> — Launching electric off-road extreme racing series, in a bid to fulfill his pledge of a creating sustainable planet and achieving carbon neutrality at the earliest.</li><li><a href="https://www.hamiltoncommission.org/">Hamilton commission</a> — in collaboration with Royal Academy of Engineering, is set to bring out the best young racing people from black backgrounds who are otherwise denied such opportunities.</li><li><a href="https://usa.tommy.com/en/tommy-x-lewis">Tommy-X-Lewis</a> — His own fashion line partnered with “Tommy Hilfiger” which also makes <em>sustainable style</em> (Recycled Polyamide aka Nylon: These products contain independently certified recycled nylon from used textiles, carpets or fishing nets reducing its environmental impact) products apart from the regular clothing lines.</li></ul><p>Stitching all, what stands out about him is, how he leverages social media (The most followed F1 celebrity in Instagram/Facebook/Twitter combined) in sharing his vision and impacting his followers positively (manifesting positivity) on how to chase one’s dreams and help them to bring out a better version of themselves.</p><p>My favorite slogans from him, that we hear him say often are, “Stay blessed”, “Sending you hope”, “Best fans”, “Still I rise”, isn’t he a man full of positivity? I wish this 7-time world champion a great success in the years to come with the iconic red as he embarks on a new journey with Ferrari!</p><p>I hope this post helped you gain a little bit about Lewis (as it did for me). And its time for us to say, “Still I rise”, don’t you think?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d0811770924" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A lean data stack idea]]></title>
            <link>https://dataxplorer.medium.com/a-lean-data-stack-idea-3abee44ae9a1?source=rss-a0dccc038e05------2</link>
            <guid isPermaLink="false">https://medium.com/p/3abee44ae9a1</guid>
            <category><![CDATA[exploration]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[minio]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <dc:creator><![CDATA[Simbu]]></dc:creator>
            <pubDate>Thu, 12 Dec 2024 06:14:49 GMT</pubDate>
            <atom:updated>2024-12-12T06:14:49.357Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello data folks! How’re you?</p><p>As 2024 is about to end, I am finally publishing my first tech blog (one of this year’s resolution, which I have been mulling for quite a while). I’m going to start with a little about me, my name is Simbu, working in the field of data engineering for more than a decade. I’m always passionate about learning anything new in the data industry and especially since the AI boom manifested by OpenAI/Nvidia, the demand for data engineering is rapidly expanding across the entire Data and AI landscape.</p><p><strong>Intro</strong></p><p>Today, I’m thrilled to share an interesting data stack I carried out with open source tools, involving <strong>MinIO</strong> (object storage) and <strong>DuckDB</strong> inspired from <a href="https://blog.min.io/duckdb-and-minio-for-a-modern-data-stack/">this blog</a> post. I have been wanting to get my hands into DuckDB for a while, and with the recent exposure I had on MinIO, I thought its a perfect concoction for a great recipe. The limitations of my lightweight laptop, with only 8GB of RAM, prompted me to explore this setup for storing and processing data on a single node. My past challenges with resource-heavy Docker images, which require substantial RAM, further motivated this choice. With the availability of cost-effective storage solutions today, MinIO serves as an excellent choice for developers seeking an S3-compatible object storage solution for both local and cloud setups. DuckDB, on the other hand, shines in OLAP scenarios, offering a combination of speed and simplicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NAiBcsrjj8fGtrAS.jpg" /><figcaption>Harnessing the power of MinIO and DuckDB</figcaption></figure><p><strong>So what is </strong><a href="https://min.io/docs/minio/windows/index.html#minio-object-storage-for-platform"><strong>MinIO</strong></a><strong>?</strong></p><blockquote>MinIO is an object storage solution that provides an Amazon Web Services S3-compatible API and supports all core S3 features. MinIO is built to deploy anywhere — public or private cloud, BareMetal infrastructure, orchestrated environments, and edge infrastructure.</blockquote><p>This means we can use MinIO locally by downloading <a href="https://dl.min.io/server/minio/release/windows-amd64/minio.exe">minio.exe</a> file for Windows, setting it up to run on our localhost, and accessing it through a web browser. This effectively simulates S3 APIs, providing a comparable experience.</p><pre>## navigate to the path where you have the minio.exe file downloaded and use the path where you want to store the console logs <br>.\minio.exe server &lt;/path/to/console-log&gt; --console-address :9001</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAWKkZfJ5iKtaIvFgGyLPA.png" /><figcaption>We can start the MinIO server from the PowerShell/terminal with the above command</figcaption></figure><p>If you’ve successfully configured, you’d be facing a login page ( http:127.0.0.1:9001) similar to the below image, where you can use the default login minioadmin/minioadmin. You could use the same login or create a new user and manage your activities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oi1LcPaS1PC0Og2p5tJhVg.png" /><figcaption>This is the UI login page which gets created from the above command as a result</figcaption></figure><p>In this use case, I created 3 different users on random basis (which can be created under Identity → Users) who has varying accesses in the MinIO buckets. To simplify, the developer role can read and write data to both the buckets (that’s the role I’m using for this exploration) while the consumer has only read access to the buckets and the producer has only write access (never used these two though).</p><figure><img alt="Under identity option, if you select users, you’d be redirected to this page where you can create or edit the current users and vary their permissions in terms of s3 bucket policies" src="https://cdn-images-1.medium.com/max/1024/1*W4USJgXHBwTDaoOkBmQLnQ.png" /><figcaption>This image lists the current users</figcaption></figure><p>And now, for creating a bucket, navigate to the Buckets tab (under Administrator section, hence this is defined with the super user privilege) and click on the “Create Bucket <strong>+</strong>” button at the top. I created two buckets with my own configurations as provided below, one to store the raw data as such and the second one is to store transformations (here the raw-dataset bucket has csv files, whereas the compressed-zone has files in parquet) which we’ll use as a source for analysis through DuckDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p4NVVIX9vzA24igP4DXRuQ.png" /><figcaption>Configuration applied when creating these buckets through the MinIO console</figcaption></figure><p>And now that the buckets are in place, I manually uploaded some files to the raw-dataset bucket through the console UI. For this activity, I downloaded F1 world championship public dataset (since I’ve been following F1 since 2005) from <a href="https://www.kaggle.com/datasets/rohanrao/formula-1-world-championship-1950-2020">Kaggle</a>.</p><p>While all the above are done through the MinIO console (I suggest you could explore many other features through this console and get yourself familiarized), I was also able to achieve the same through the MinIO API for the most frequent use cases involving, creating/removing a bucket, listing its objects or removing them and uploading data (files and folders) to these buckets. I used Python 3.9 for this exercise and the entire code is available in this <a href="https://github.com/simbu175/MinIO_DuckDB">GitHub</a> repo. Do note however, in order to get your hands dirty through these codes, its important to setup your access keys (this is similar to access key and secret access key in AWS) to connect to the MinIO server from your machine. We can generate our keys through the browser and store them locally in the codebase, or via environment variables or somewhere like a vault/secrets manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KmhBOk7CuJRVAF3EgyYpLA.png" /><figcaption>Access key with its secret to access the MinIO server</figcaption></figure><p>Clicking on the pencil icon at the right most, allows me to edit the policies associated with this user (developer who has both read and write access to the buckets) as follows,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*XPqXrvjvPWcWBzJUjq-k5A.png" /><figcaption>In the above JSON, we can edit the Actions under Allow and resource to restrict access on the attribute level</figcaption></figure><p>Diving into the raw-dataset bucket for its content, we see many csv files that has specific information and the same is converted to parquet files using DuckDB and stored in a secondary bucket, compressed-zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ja6iz_mlhLeFproc9auzDA.png" /><figcaption>raw-dataset bucket and its content (csv files)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uIaCemwmkBVEpd6BMUQVXw.png" /><figcaption>compressed-zone bucket and its content — post transformation with DuckDb</figcaption></figure><p>With this, let’s dive into DuckDB for transforming these data to a compressed form with a simplified SQLite based in-process db approach where the computations happen in-memory</p><p><strong>And </strong><a href="https://duckdb.org/"><strong>DuckDB</strong></a><strong>?</strong> As per its official site,</p><blockquote>DuckDB is a fast simple, in-process, analytical, open-source, portable, DBMS supporting a feature-rich SQL dialect complemented with deep integrations into client APIs.</blockquote><p>So why is this efficient to run on a single node? There is no separate DBMS server software to install, update and maintain. It does not run as a separate process, but completely embedded within a host process. For the analytical use case that DuckDB targets, this has the additional advantage of high-speed data transfer to and from the database. In some cases, DuckDB can process foreign data without copying (similar to Snowflake’s zero-copy feature). But unlike Snowflake, DuckDB operates entirely in-process, avoiding remote metadata storage dependencies. This design makes it lightweight and perfect for local or embedded analytics scenarios. Running locally on your PC/laptop, DuckDB can be used in two modes, in-memory or persistent modes.</p><p>By default, its connected with “memory” option,</p><pre>import duckdb<br>con1 = duckdb.connect(&quot;:memory:&quot;)  # in-memory mode<br>con2 = duckdb.connect(r&quot;D:/DuckDB/my_database.duckdb&quot;)  # persistent to disk</pre><p>In the above code snippet, con1 is an in-memory connection and con2 is the persisted connection, meaning the db objects are referenced from this path when we use the later mode.</p><p>A simple Duck SQL command to convert csv files to parquet is as below,</p><pre>COPY (SELECT * FROM &#39;s3://raw-dataset/inputs/circuits.csv&#39;) <br>TO &#39;s3://compressed-zone/inputs&#39; (FORMAT &#39;PARQUET&#39;);</pre><p>That’s how easy it is to convert files from one format to the other using DuckDB. Care must be taken to setup DuckDB to reach the MinIO local server, below is the code I used to connect DuckDB while also reading/writing data from/to MinIO</p><pre>def connect(self):<br>    &quot;&quot;&quot;<br>    Establishes a connection for the duckdb client<br>    &quot;&quot;&quot;<br>    try:<br>        self.con = duckdb.connect(database=self.db_path, read_only=False)<br>        self.con.execute(f&quot;&quot;&quot;<br>            INSTALL httpfs; <br>            LOAD httpfs;<br>            SET s3_endpoint=&#39;{self.minio_host}&#39;;<br>            SET s3_access_key_id=&#39;{self.minio_access_key}&#39;;<br>            SET s3_secret_access_key=&#39;{self.minio_secret_key}&#39;;<br>            SET s3_use_ssl=false; <br>            SET s3_url_style=&#39;path&#39;;<br>        &quot;&quot;&quot;)<br>        return self.con<br>    ...</pre><ul><li>INSTALL/LOAD httpfs; — This DuckDB extension is used to connect to remote resources like S3 (or over HTTPS network) and since this is a local installation of MinIO.</li><li>SET s3_use_ssl=false — Used to avoid using certificates expiration (for SSL connections)</li><li>SET s3_url_style=&#39;path&#39; — Useful to connect to S3 resources with the same way we define paths within S3 (or it could be set as ’vhost’ for S3 URI instead)</li></ul><p>With this setup, we were able to iterate over all the source csv files inside the prefix, s3://raw-dataset/inputs and convert them one by one over the MinIO list_objects iterator (similar to S3&#39;s) to parquet and store in the destination effortlessly by DuckDB. And once these data are ready (mind you since DuckDB can directly read csv files as well, but I wanted to try and its better to use parquet for analytical queries which is the forte of this setup and how this can be leveraged for future activities). We can start with some basic queries, like who’s the driver or constructor with most points, wins, poles etc.</p><p>To understand the data in the source files, I created two schemas — raw_data and derived_data (yes DuckDB supports schema as well, however grants and user management features are something that is not yet supported) I created tables from these parquet files in raw_data schema as these has some dimensional data, like drivers, races, constructors, status etc, the source data model would look like,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jxxVnewS-vR_K5yKvfL6Pg.png" /><figcaption>A simplified logical data model of the source data — included only necessary tables for this analysis</figcaption></figure><p>As we can see from the above, the results &amp; sprint_results table seem like a central fact table and the drivers, constructors, circuits, status are more of a dimensional tables each has its own attributes which seldom change (but in the event they do, the historical values are not preserved and hence these are the SCD type 1 model) and hence they’re part of the classic star schema of a typical warehouse. While the tables like driver_standings, constructor_standings &amp; constructor_results are a quasi-fact tables because they do help in finding some metrics like, number of wins, points got by driver/team, though these doesn’t help in extracting fine grained metrics like # of drivers winning from Pole or drivers who won from most different position on the grid or fastest laps per driver/constructor metrics.</p><p>For extracting those finer metrics from the raw_data, I’m going to create the derived_data schema and model this resultFacts table, with the below definition,</p><pre>USE raw_data;<br><br>--- Note that this derived table is created in derived_data schema<br>DROP TABLE IF EXISTS derived_data.resultFacts;<br>CREATE TABLE derived_data.resultFacts AS <br>SELECT res1.resultId, rac1.raceId, &#39;Normal race&#39; AS race_type, rac1.name||&#39; &#39;||rac1.year as race_name, dr1.forename||&#39; &#39;||dr1.surname AS driver_name, cons1.name AS constructor_name, res1.grid AS starting_grid, res1.position, res1.positionText AS finishPosition, res1.positionOrder AS finishOrder, res1.points, res1.laps, res1.time||&#39;.&#39;||res1.milliseconds AS raced_time, res1.fastestLap, res1.rank, res1.fastestLapTime, res1.fastestLapSpeed AS fastestLapSpeed_kmph, st1.status <br>FROM results res1<br>JOIN status st1 USING (statusId)<br>JOIN races rac1 USING (raceId)<br>JOIN drivers dr1 USING (driverId)<br>JOIN constructors cons1 USING (constructorId)<br>UNION ALL<br>SELECT res2.resultId, rac2.raceId, &#39;Sprint race&#39; AS race_type, rac2.name||&#39; &#39;||rac2.year as race_name, dr2.forename||&#39; &#39;||dr2.surname AS driver_name, cons2.name AS constructor_name, res2.grid AS starting_grid, res2.position, res2.positionText AS finishPosition, res2.positionOrder AS finishOrder, res2.points, res2.laps, res2.time||&#39;.&#39;||res2.milliseconds AS raced_time, res2.fastestLap, NULL AS rank, res2.fastestLapTime, NULL AS fastestLapSpeed_kmph, st2.status<br>FROM sprint_results res2<br>JOIN status st2 USING (statusId)<br>JOIN races rac2 USING (raceId)<br>JOIN drivers dr2 USING (driverId)<br>JOIN constructors cons2 USING (constructorId);</pre><p>This materialization of the resultant cube is cleaned, transformed from the base dimensional points, and is currently loaded in an atomic granularity on the combination of driver, constructor, races thereby any aggregation metrics needed on the race results/driver/constructor can be done as applicable. For ex, to arrive at the Most points per driver over a season,</p><pre>SELECT driver_name, SUM(points) AS driver_points <br>FROM resultFacts --- includeds both Normal races &amp; Sprints<br>WHERE RIGHT(race_name, 4) = 2021<br>GROUP BY driver_name <br>ORDER BY driver_points DESC, driver_name;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*kUBv0G_x5DyGJmZJYNZMaQ.png" /><figcaption>DuckDB driver points data — 2021 season</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*Zo9jq5DJc1R7yshEiwSgXw.png" /><figcaption>F1 site driver points data — 2021 season</figcaption></figure><p>This result includes the points scored by the driver as part of Sprint races as well, as the resultFacts table is inclusive of ‘Normal race’ and ‘Sprint race’ with an UNION ALL clause.</p><p>Another interesting feature I’d like to point here is, IMPORT and EXPORT database command. Export command simply exports the schema definition and data to the local path we’re giving as below,</p><pre>-- Exports the in-memory database to disk<br>EXPORT DATABASE &quot;/path/to/export&quot; <br>  (FORMAT PARQUET, <br>   COMPRESSION ZSTD,  <br>   ROW_GROUP_SIZE 100_000);<br><br>-- Imports the data from disk to memory<br>IMPORT DATABASE &quot;/path/of/exported/folder&quot;;</pre><p>Most fastest laps per driver in career,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*d0lnLYkE97bwGpH6m_lOXA.png" /><figcaption>Data derived from our dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*EZKeSVW8cfGkphyEQbmf9A.png" /><figcaption>Data from google</figcaption></figure><p>While the result above is far from the reality for drivers of the past, I found that in the source data itself, fastest laps metrics aren’t captured for races prior to 2004 and hence Michael Schumacher here is heavily unfavored here. So, this is essential an issue from the dataset and not from DuckDB or MinIO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6k1oDeLfpAMFS1la72yxcA.png" /><figcaption>This is a result of 2003 Australian GP, we could see the fastestLap#, fastestLapTime and rank are by default \N</figcaption></figure><p>I have collated some of simpler metrics and its computations in the <a href="https://github.com/simbu175/MinIO_DuckDB/blob/master/extras/queries.sql">queries.sql</a> Here, I have defined the schema for the views and the tables. I would like to deep dive into the analytical prowess of DuckDB like pivot/unpivot, window/qualify, filter clauses and keep updating this file.</p><p><strong>Pros and Cons — MinIO + DuckDB</strong></p><ul><li><strong>Lightweight</strong> and easy to setup (both MinIO and DuckDB) and open-source (for individuals), need I say more?</li><li>The MinIO APIs are quite <strong>similar to S3,</strong> and this makes it easy to port to S3 in the future, with ease.</li><li>MinIO sets the bar as one of the <a href="https://blog.min.io/tag/benchmarks/">fastest object storage solution available</a>, ensuring data access and retrieval are lightning-fast.</li><li>When integrated with DuckDB, this performance boost enhances the efficiency of data analytics and processing.</li><li>DuckDB’s <strong>zero-copy</strong> integration with data sources like Parquet files allows for high-speed query execution directly without duplicating data.</li><li>While DuckDB is super fast, it is not suitable for live/transactional workloads a.k.a high-frequency writes.</li><li>With DuckDB, schema support is basic compared to other databases and user management is outside its current scope due to its design as an in-process DBMS.</li></ul><p><strong>Next steps?</strong></p><ul><li>Keeping this simplicity in mind, I would like to explore lakehouse tables (Hudi/Iceberg) that DuckDB supports to read/write.</li><li>Exploring the features inside MinIO, viz., replication, encryption (SSE-S3, KMS), IAM, Lifecycle management, versioning as securing this setup is equally important.</li><li>Scouting DuckDB’s features like extensions, ADBC protocols, Indexing and benchmarking for bulkier data volumes (with persisted mode)</li></ul><p><strong>Outro</strong></p><p>With this post, I’d like to highlight the light-weight nature of this setup, along with the advantages and disadvantages within it and if we can counter the same (or at least try to) optimally.</p><p>Suggestions are welcome with your feedback on improvising this stack is hugely appreciated. Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3abee44ae9a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>