<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hemant Marve on Medium]]></title>
        <description><![CDATA[Stories by Hemant Marve on Medium]]></description>
        <link>https://medium.com/@hemant.marve?source=rss-15fe6d933637------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zPqPjRxGBkY944VuriJQVg.jpeg</url>
            <title>Stories by Hemant Marve on Medium</title>
            <link>https://medium.com/@hemant.marve?source=rss-15fe6d933637------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:08:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hemant.marve/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bloom Filters: Preventing Cache Penetration with Just a Few Megabytes of Memory]]></title>
            <link>https://medium.com/@hemant.marve/bloom-filters-preventing-cache-penetration-with-just-a-few-megabytes-of-memory-9ff402a4b7e1?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ff402a4b7e1</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 10:49:30 GMT</pubDate>
            <atom:updated>2026-07-01T10:49:30.116Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A practical, deep dive into one of the most underrated data structures in system design.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aYwQV_EuFZQuGvV_m3qfiw.png" /><figcaption><strong>HLD: Bloom Filter Preventing Cache Penetration</strong></figcaption></figure><p>Imagine your API is receiving <strong>1,000,000 requests per second</strong>. Of those, 800,000 are for real, existing data. The other 200,000 are for IDs, usernames, or URLs that simply don’t exist — bad requests, bots, typos, scrapers probing your system.</p><p>Without any protection, every single one of those 200,000 invalid requests still travels all the way through your stack: hits your load balancer, hits Redis, misses, then hits your database, misses again, and finally returns a 404. You just paid the full cost of a database lookup — for something you already knew wasn’t going to be there.</p><p>This pattern has a name: <strong>cache penetration</strong>. And it’s one of the sneakiest ways a system quietly burns capacity on requests that were never going to succeed in the first place.</p><p>The fix is a small, elegant, decades-old data structure called a <strong>Bloom filter</strong> — and understanding it well is one of those things that consistently pays off, both in production systems and in system design.</p><h3>The Problem: Cache Penetration</h3><p>Picture the normal request path without any safeguard:</p><pre>Client → Redis (cache) → Database → 404 Not Found</pre><p>Every invalid request pays for a Redis round-trip <em>and</em> a database query, just to be told “no.” At scale, this isn’t a rounding error — it’s 200,000 unnecessary database hits, every single second, in our example. That’s the kind of load that takes down databases during traffic spikes, bot attacks, or scraping campaigns.</p><p>What we actually want is a way to answer one very specific question, <em>before</em> any of that expensive work happens:</p><blockquote><em>“Could this key possibly exist?”</em></blockquote><p>If the answer is a confident “no,” we reject instantly — no Redis, no database, no wasted work. That’s the entire job of a Bloom filter.</p><h3>What Is a Bloom Filter?</h3><p>A Bloom filter is a <strong>space-efficient, probabilistic data structure</strong> that answers exactly one question, and it answers it in one of two ways:</p><ul><li><strong>Definitely not present</strong> — we can safely reject the request immediately.</li><li><strong>Possibly present</strong> — we need to go check the real source of truth (Redis or the database).</li></ul><p>The key guarantee that makes this useful: a Bloom filter can produce <strong>false positives</strong>, but it will <strong>never produce a false negative</strong>.</p><p>In other words, it might occasionally say “possibly present” for something that doesn’t actually exist (a false positive — costs you one extra lookup). But it will never say “definitely not present” for something that <em>does</em> exist. That asymmetry is exactly what makes it safe to use as a gatekeeper: the worst case is a wasted lookup, never a wrongly rejected valid request — as long as the filter is fully in sync with reality (more on when that assumption breaks down later).</p><h3>How It Works: A Bit Array and a Few Hash Functions</h3><p>Under the hood, a Bloom filter is just:</p><ul><li>A <strong>bit array</strong> of size m (all bits start at 0)</li><li>k independent <strong>hash functions</strong>, each mapping a key to a position in the array</li></ul><h3>Inserting a key</h3><p>To insert &quot;Apple&quot;, we run it through all k hash functions and set the resulting bit positions to 1.</p><pre>Insert &quot;Apple&quot;   → h1=2, h2=7, h3=11<br>Bit array:  0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0<br>                ↑         ↑       ↑<br>               bit 2     bit 7   bit 11</pre><p>Now insert &quot;Orange&quot; too:</p><pre>Insert &quot;Orange&quot;  → h1=4, h2=7, h3=15<br>Bit array:  0 0 1 0 1 0 0 1 0 0 0 1 0 0 0 1<br>                    ↑                     ↑<br>                new bit 4             new bit 15</pre><p>Notice bit 7 was already set by “Apple” — hash collisions are expected and totally fine.</p><h3>Looking up a key</h3><p>To check whether &quot;Banana&quot; might exist, hash it the same way and look at those bit positions.</p><pre>Lookup &quot;Banana&quot;  → h1=3, h2=9, h3=11<br>Bits at those positions: 0, 0, 1</pre><p>Since <strong>not all three bits are 1</strong>, we know for certain “Banana” was never inserted. <strong>Definitely not present. Reject immediately.</strong></p><p>Now check &quot;Apple&quot;:</p><pre>Lookup &quot;Apple&quot;   → h1=2, h2=7, h3=11<br>Bits at those positions: 1, 1, 1</pre><p>All three bits are set — so “Apple” is <strong>possibly present</strong>. It might genuinely be “Apple,” or it might be a coincidental collision from other inserted keys. Either way, this is where we go check Redis or the database to be sure.</p><p>This lookup runs in <strong>O(k)</strong> time — constant, regardless of how many keys have been inserted — which is what makes Bloom filters so cheap to check on every single request.</p><p><strong>Formula For a Bloom filter with:</strong></p><pre>m = number of bits<br>n = inserted elements<br>k = hash functions<br>False positive probability is approximately:<br>P ≈ (1 − e^(-kn/m))^k<br>The optimal number of hash functions is:<br>k = (m/n) ln 2</pre><blockquote><strong>NOTE</strong>: Standard Bloom Filters do not support deletion because clearing a bit may remove information belonging to other elements. If deletion is required, a Counting Bloom Filter can be used.</blockquote><h3>High-Level Architecture</h3><p>Dropped into a real system, the Bloom filter sits as a gatekeeper right at the front door:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2CuMIkt9H5_v_Eksofmuw.png" /><figcaption><strong>High-Level Design: Single-Server Bloom Filter</strong></figcaption></figure><p>The 90%+ of invalid traffic that the filter catches never touches Redis or the database at all. Only genuinely-plausible requests make it through to the expensive layers.</p><h3>Scaling It Out: Bloom Filters in Distributed Systems</h3><p>A single Bloom filter is easy. The interesting engineering shows up once you have <strong>multiple API servers behind a load balancer</strong>, each handling requests independently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2NGgfsE-enMK153XKXDdFA.png" /><figcaption><strong>High-Level Design: Distributed Bloom Filter Architecture</strong></figcaption></figure><p>Each API server keeps <strong>its own in-memory Bloom filter</strong>, because a lookup needs to be sub-microsecond — adding a network hop just to ask “does this exist?” would defeat the entire purpose.</p><p>The filters stay (mostly) in sync via:</p><ul><li><strong>Event streaming</strong> (Kafka, etc.) — every insert publishes an event, and every server updates its local filter as events arrive.</li><li><strong>Periodic rebuilds</strong> — filters are regenerated from the source of truth on a schedule and redistributed.</li></ul><p>This works well — until you hit a subtle but very real edge case.</p><h3>The Gotcha: What Happens When the Write and the Read Land on Different Servers?</h3><p>Here’s a scenario that trips up a lot of people the first time they think it through carefully:</p><ol><li>A user is created — userId1 — and the write request happens to land on <strong>API Server 1</strong>. Server 1 updates its own local Bloom filter immediately.</li><li>Almost immediately after, a read request for userId1 comes in, and the load balancer happens to route it to <strong>API Server 2</strong>.</li><li>Server 2’s Bloom filter hasn’t received the update yet — the Kafka event is still in flight, or the last periodic rebuild hasn’t run.</li><li>Server 2’s filter says <strong>“Definitely Not Present”</strong> → the request is rejected with a 404 — even though userId1 genuinely exists.</li></ol><p>This is worth sitting with for a second: the Bloom filter itself didn’t lie. Its guarantee — no false negatives — only holds for the data it has actually seen. The <em>system</em> produced a false negative anyway, because one node’s copy of reality was stale. Propagation lag broke the guarantee from the outside.</p><h3>Fixing it: a shared safety-net cache for the propagation gap</h3><p>The cleanest fix doesn’t touch the Bloom filter at all — it adds a small, short-lived, <strong>shared</strong> cache that catches exactly this race window.</p><p>Each server keeps a Bloom filter (slow to update, syncs via Kafka) <em>and</em> every server shares one lightweight store — typically just a Redis SET with a short TTL — that’s updated <strong>synchronously, instantly, on every write</strong>:</p><pre># ON WRITE (Server 1, creating userId1)<br>redis.SADD(&quot;recent_writes&quot;, &quot;userId1&quot;)<br>redis.EXPIRE(&quot;recent_writes&quot;, 10)          # auto-clears in 10 seconds<br>kafka.publish(&quot;key_added&quot;, &quot;userId1&quot;)      # async — for Bloom filter sync<br>db.insert(userId1)</pre><pre># ON READ (any server, checking userId1)<br>if local_bloom_filter.might_contain(&quot;userId1&quot;):<br>    check_redis_or_db(&quot;userId1&quot;)<br>else:<br>    # Bloom filter says &quot;not present&quot; — don&#39;t trust it blindly yet<br>    if redis.SISMEMBER(&quot;recent_writes&quot;, &quot;userId1&quot;):<br>        check_redis_or_db(&quot;userId1&quot;)       # override: treat as possibly present<br>    else:<br>        return_404()                       # safe to reject</pre><p><strong>Walking through the timeline with the fix in place:</strong></p><p>Time Event Result t = 0ms Write lands on Server 1 userId1 added to shared recent_writes set (≈1ms) t = 15ms Read lands on Server 2 Local Bloom filter says &quot;not present&quot; (still stale) t = 15ms Server 2 checks recent_writes Found — overrides the stale Bloom filter result t = 15ms Server 2 queries the DB instead of rejecting Correct data returned ✓ t = 10s recent_writes entry expires No longer needed — the Bloom filter has caught up by now anyway</p><p>This works well in practice because:</p><ul><li>The recent_writes set only ever holds a few seconds&#39; worth of data, so it stays tiny and cheap to check.</li><li>It’s only ever consulted on the “Bloom filter says no” path — which is the minority of traffic — so it barely touches overall throughput.</li><li>It’s self-cleaning: entries expire on their own once the Bloom filter has genuinely synced.</li></ul><p>Other common mitigations worth knowing (often combined with the above):</p><ul><li><strong>Sticky sessions</strong> — route a client’s follow-up requests back to the server that handled their write, for a short window. Cheap, and solves the most common trigger: a user immediately reading back their own write.</li><li><strong>Faster propagation</strong> — tightening the Kafka pipeline so updates reach every node in single-digit milliseconds shrinks the race window without eliminating it.</li><li><strong>A centralized Bloom filter</strong> (e.g., via RedisBloom) — trade the sub-microsecond local lookup for a sub-millisecond network hop, but eliminate cross-node staleness entirely. Many production systems choose this specifically to sidestep the whole class of bug.</li></ul><h3>The Payoff: What This Actually Buys You</h3><p>Back to our original numbers — 1,000,000 requests/second, 200,000 of them invalid:</p><p><strong>Without a Bloom filter:</strong> All 200,000 invalid requests hit Redis, miss, then hit the database. That’s <strong>200,000 unnecessary database lookups every second.</strong></p><p><strong>With a Bloom filter (assuming a 1% false positive rate):</strong></p><ul><li>~198,000 requests (99%) are rejected instantly — no Redis, no database.</li><li>~2,000 requests (1%) are false positives and proceed to a real lookup, which correctly returns nothing.</li></ul><p><strong>Result: only 2,000 database lookups instead of 200,000 — a ~99% reduction in unnecessary load</strong>, for the cost of a data structure that occupies a few megabytes of memory per million keys.</p><h3>Where Bloom Filters Show Up in the Real World</h3><p>Once you know what to look for, Bloom filters turn up everywhere:</p><ul><li><strong>URL shorteners</strong> — instantly check whether a short code is already taken.</li><li><strong>Distributed databases</strong> (Cassandra, HBase, etc.) — skip reading SSTables that provably don’t contain a key.</li><li><strong>Web crawlers</strong> — avoid re-visiting URLs that have already been crawled.</li><li><strong>API gateways</strong> — reject invalid or unknown IDs before they reach downstream services.</li><li><strong>Search engines</strong> — quickly filter out documents that don’t contain a given term.</li><li><strong>Cache penetration prevention</strong> — the exact use case that kicked off this whole post.</li></ul><h3>Key Takeaways (Cheat Sheet)</h3><ul><li>Bloom filters are <strong>probabilistic</strong>: false positives are possible, false negatives are not — <em>as long as the filter’s view of the world is up to date.</em></li><li>They act as a <strong>fast gatekeeper</strong> in front of expensive operations, not a replacement for the source of truth.</li><li>They’re <strong>memory efficient</strong> — bits only, never the actual data.</li><li>Lookup and insert are both <strong>O(k)</strong>, where k is the number of hash functions.</li><li>The size of the bit array (m) and number of hash functions (k) are tunable knobs that control your false positive rate — worth knowing the rough relationship.</li><li>In distributed systems, <strong>staleness between nodes is the real challenge</strong>, not the filter’s math. Sticky routing, fast event propagation, a shared short-TTL safety-net cache, or a centralized filter are the standard fixes.</li></ul><p>Sometimes the biggest performance win isn’t making your database faster — it’s <strong>preventing unnecessary requests from ever reaching it.</strong> A Bloom filter is a small, cheap, elegant way to do exactly that.</p><p><em>Have you used Bloom filters in a system you’ve built? I’d love to hear how you handled the distributed consistency edge cases — drop a comment below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ff402a4b7e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accessing iceberg tables managed by watsonx.data using Databricks Spark]]></title>
            <link>https://medium.com/@hemant.marve/accessing-iceberg-tables-managed-by-watsonx-data-using-databricks-spark-1defe3bb50cb?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/1defe3bb50cb</guid>
            <category><![CDATA[watsonxdata]]></category>
            <category><![CDATA[interoperability]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 08:50:57 GMT</pubDate>
            <atom:updated>2025-05-09T15:16:48.426Z</atom:updated>
            <content:encoded><![CDATA[<p>Integrating <strong>databricks Spark</strong> with w<strong>atsonx.data</strong> via the <strong>Iceberg REST API</strong> enables seamless access to large-scale datasets stored in <strong>Iceberg tables</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lWwjjAWNOw4uFD068FHcvw.png" /><figcaption><strong>Databricks Integration with watsonx.data</strong></figcaption></figure><p><strong>Let’s explore how we can access an Iceberg table created in WXD using databricks spark. For more clarity, please refer to the examples also.</strong></p><p><strong>Please follow below steps:</strong></p><ol><li>In watsonx.data go to infrastructure manager &amp; create an iceberg type catalog associated with any bucket of type S3a or GCS or Azure. You can use already existing iceberg type catalog also.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gYTzKNDD4xR4LPKQrXGMUA.png" /><figcaption><strong>Catalog Details</strong></figcaption></figure><p>2. You will able to see MDS Rest endpoint in catalog details in SAAS &amp; for software please use MDS EXTERNAL endpoint in catalog details.</p><blockquote><strong><em>Skip 3–5 steps if you are using watsonx.data on cloud.</em></strong></blockquote><p>3. In case of Software you need to fetch certificates using below command.</p><pre>echo QUIT | openssl s_client -showcerts -connect &lt;mds-rest-endpoint-host&gt;:443 | awk &#39;/-----BEGIN CERTIFICATE-----/ {p=1}; p; /-----END CERTIFICATE-----/ {p=0}&#39; &gt; mds.cert</pre><p>4. <strong>cat mds.cert</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cA3L5T4T8FoUP1nR.png" /></figure><p>5. Import the both certificates into the Java truststore that Spark will use at runtime.</p><pre>keytool -import -alias &lt;alias_name&gt; -file &lt;certificate_file&gt; -keystore &lt;truststore_file&gt; -storepass &lt;truststore_password&gt;</pre><p>EXAMPLE:</p><pre>sudo keytool -import -alias mycert -file mycert.crt -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit</pre><p>6. We need to install all three jars mentioned below within databricks spark compute library &amp; restart spark compute after installation.</p><pre>org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2<br>org.apache.hadoop:hadoop-aws:3.3.6<br>software.amazon.awssdk:bundle:2.27.14</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFyhONAVhcdqsBEDTxk6iA.png" /><figcaption><strong>Example installing iceberg jar</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTRVpPMDitkiZsUYkI8lZA.png" /><figcaption><strong>After installation(highlited only)</strong></figcaption></figure><p>7. To access Iceberg tables managed by watsonX.data on the Databricks side, you must configure the watsonx.data Iceberg REST catalog within your Databricks notebook. Below is the required configuration to enable seamless access to these tables.</p><blockquote><strong>Note:</strong> The user must have Admin privileges on Databricks to apply this configuration.</blockquote><pre><br>spark.conf.set(&quot;spark.sql.iceberg.vectorization.enabled&quot;,&quot;false&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.s3.endpoint&quot;,&quot;https://&lt;bucket-endpoint&gt;&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.client.region&quot;,&quot;&lt;region&gt;&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.s3.access-key-id&quot;,&quot;&lt;bucket-access-key&gt;&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.s3.secret-access-key&quot;,&quot;&lt;bucket-secret-key&gt;&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;&quot;,&quot;org.apache.iceberg.spark.Sparkcatalog&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.type&quot;,&quot;rest&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.warehouse&quot;,&quot;&lt;wxd-catalog&gt;&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.s3.path-style-access&quot;,&quot;true&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.uri&quot;,&quot;https://&lt;mds-rest-endpoint&gt;/mds/iceberg&quot;);<br>spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.header.Authoirzation&quot;,&quot;Bearer &lt;token&gt;&quot;);<br>#spark.conf.set(&quot;spark.sql.catalog.&lt;wxd-catalog&gt;.header.Authoirzation&quot;,&quot;Basic &lt;base64 encoded username:password&gt;&quot;);<br>spark.conf.set(&quot;spark.sql.defaultCatalog&quot;,&quot;&lt;wxd-catalog&gt;&quot;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i942GyKAwOsrlFULqoeK2w.png" /><figcaption><strong>Iceberg Rest Catalog Configuration</strong></figcaption></figure><blockquote>Sample notebook used on databricks side in examples: <a href="https://github.com/IBM/watsonx-data/blob/main/Tutorials/wxd_databricks_integration.ipynb">https://github.com/IBM/watsonx-data/blob/main/Tutorials/wxd_databricks_integration.ipynb</a></blockquote><p>8. After above configuration , now we can see tables/schemas managed by watsonx.data on databricks side.</p><p><strong>Example</strong>: In the example below, we’ve used the <strong>iceberg_data_demo</strong> catalog, which is present on the watsonx.data side.</p><ol><li><strong>iceberg_data_demo</strong> Catalog Details &amp; Available Schemas/Tables in watsonx.data</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q3TcSpnX4PBYHKloqXY5zw.png" /><figcaption><strong>iceberg_data_demo(wxd side)</strong></figcaption></figure><p>2. Now, let’s execute some queries on the <strong>iceberg_data_demo</strong> catalog within the Databricks notebook using the Spark compute, which we’ve already configured with the required libraries. <strong>Wow!</strong> We can see the same schemas and tables on the Databricks side as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ciNr9Jj48Lps8jAXs0de_Q.png" /><figcaption><strong>Querying same iceberg_data_demo catalog on databricks side</strong></figcaption></figure><p>3. Let’s query table data on watsonx.data side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yo-vq3wPpcuIXQ2giynpHw.png" /><figcaption><strong>Querying table data on wxd side</strong></figcaption></figure><p>4. Now, let’s query the same table data on the Databricks side. <strong>Awesome!</strong> We’re able to fetch the exact same data on Databricks as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eW-QekNoTh37OiiNWScPIQ.png" /><figcaption><strong>Able to query same table data on wxd side</strong></figcaption></figure><p>5. let’s suppose within unity catalog we already have table <strong>dr_unity_table</strong> with column id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eeXlsFvKcgntV2EG8imZdw.png" /><figcaption><strong>Table on Unity catalog side</strong></figcaption></figure><p>6. Let’s perform join between table managed by unity catalog &amp; managed by watsonx.data catalog.</p><blockquote><strong>Note: dbr_unity_table</strong> managed by unity catalog on databricks side &amp; <strong>merge_on_read_table</strong> managed by watsonx.data</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5sk68x2WtQbyXeb-n8S_kg.png" /><figcaption><strong>Performing join between table managed by unity catalog &amp; managed by wxd catalog</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1defe3bb50cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting Apache Spark to watsonx.data Using the Iceberg REST API]]></title>
            <link>https://medium.com/@hemant.marve/connecting-apache-spark-to-watsonx-data-using-the-iceberg-rest-api-f0bb4f0608cc?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0bb4f0608cc</guid>
            <category><![CDATA[icebergs]]></category>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[watsonxdata]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 06:17:41 GMT</pubDate>
            <atom:updated>2025-01-22T04:48:23.141Z</atom:updated>
            <content:encoded><![CDATA[<p>Integrating <strong>Apache Spark</strong> with w<strong>atsonx.data</strong> via the <strong>Iceberg REST API</strong> enables seamless access to large-scale datasets stored in <strong>Iceberg tables</strong>.</p><p>Please follow below steps:</p><ol><li>In watsonx.data go to infrastructure manager &amp; create an iceberg type catalog associated with any bucket of type S3a or GCS or Azure.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b01B3XDb_cMmNoFoQZvJPg.png" /><figcaption><strong>Catalog Details</strong></figcaption></figure><p>2. You will able to see MDS Rest endpoint in catalog details. &amp; for CPD please use MDS EXTERNAL endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sOZ1xeHJOwCKQ5Sg6dyXpw.png" /><figcaption><strong>CPD</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2AG0K3hi1WCIwHYeoMaUDA.jpeg" /><figcaption><strong>SAAS</strong></figcaption></figure><blockquote><strong>Skip 3–5 steps if you are using watsonx.data on cloud.</strong></blockquote><p>3. In case of CPD you need to fetch certificates using below command.</p><pre>echo QUIT | openssl s_client -showcerts -connect &lt;mds-rest-endpoint-host&gt;:443 | awk &#39;/-----BEGIN CERTIFICATE-----/ {p=1}; p; /-----END CERTIFICATE-----/ {p=0}&#39; &gt; mds.cert</pre><p>4. <strong>cat mds.cert</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIn-glHiR0dgnNHvYGkAow.png" /></figure><p>5. Import the both certificates into the Java truststore that Spark will use at runtime.</p><pre>keytool -import -alias &lt;alias_name&gt; -file &lt;certificate_file&gt; -keystore &lt;truststore_file&gt; -storepass &lt;truststore_password&gt;</pre><p>EXAMPLE:</p><pre>sudo keytool -import -alias mycert -file mycert.crt -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit</pre><p>6. Spark configuration.</p><pre>./bin/spark-sql -v \<br>--packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2,org.apache.hadoop:hadoop-aws:3.3.6,software.amazon.awssdk:bundle:2.27.14 \<br>--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \<br>--conf spark.sql.iceberg.vectorization.enabled=false \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.header.Authorization=&#39;Bearer &lt;token&gt; or Basic &lt;base64encoded username:password&gt;&#39; \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;=org.apache.iceberg.spark.SparkCatalog  \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.type=rest \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.warehouse=&lt;iceberg_catalog_name&gt; \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.s3.path-style-access=true \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.s3.access-key-id=&lt;bucket-access-key&gt; \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.s3.secret-access-key=&lt;bucket-secret-key&gt; \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.client.region=&lt;bucket-region&gt; \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.s3.endpoint=&lt;bucket-endpoint&gt; \<br>--conf spark.sql.catalog.&lt;iceberg_catalog_name&gt;.uri=https://&lt;mds-rest-endpoint&gt;/mds/iceberg</pre><p>Example:</p><pre>./bin/spark-sql -v \<br>--packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2,org.apache.hadoop:hadoop-aws:3.3.6,software.amazon.awssdk:bundle:2.27.14 \<br>--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \<br>--conf spark.sql.iceberg.vectorization.enabled=false \<br>--conf spark.sql.catalog.iceberg_data.header.Authorization=&#39;Bearer eyJdhssssbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJJQk1MSCIsInN1YiI6ImhlbWFudC5tYXJ2ZUBpYm0uY29tIiwiYXVkIjpbIkFNUy1VSSJdLCJleHAiOjE3MzI1MTkzNjIsImlhdCI6MTczMjUxMzYzMywidWlkIjoiSUJNaWQtNjk0MDAwODIyUiIsInBlcm1pc3Npb25zIjpbImxha2Vob3VzZS5tZXRhc3RvcmUuYWRtaW4iLCJsYWtlaG91c2UuZGFzaGJvYXJkLnZpZXciLCJsYWtlaG91c2UudXNlcnZwYy5tYW5hZ2UiXSwiZ3JvdXBzIjpbXSwiaW5zdGFuY2UiOnsidXNlcm5hbWUiOiJoZW1hbnQubWFydmVAaWJtLmNvbSIsImluc3RhbmNlX25hbWUiOiJsaDEiLCJpbnN0YW5jZV9pZCI6ImNybjp2MTpzdGFnaW5nOnB1YmxpYzpsYWtlaG91c2U6dXMtc291dGg6YS9jYzFhZTAwOThhMmQ0ZWMzOTQ3ZWNkZWFhOGE2MGQ3MTo1MDYzNjNlYS0zMjczLTQ2NWMtYWU0Ny0zYWEzYTEwMTk2ZGE6OiIsInJvbGVzIjpbImNybjp2MTpibHVlbWl4OnB1YmxpYzppYW06Ojo6cm9sZTpPcGVyYXRvciIsImNybjp2MTpibHVlbWl4OnB1YmxpYzppYW06Ojo6cm9sZTpWaWV3ZXIiLCJjcm46djE6Ymx1ZW1peDpwdWJsaWM6aWFtOjo6OnJvbGU6QXV0aG9yaXphdGlvbkRlbGVnYXRvciIsImNybjp2MTpibHVlbWl4OnB1YmxpYzppYW06Ojo6c2VydmljZVJvbGU6UmVhZGVyIiwiY3JuOnYxOmJsdWVtaXg6cHVibGljOmlhbTo6OjpzZXJ2aWNlUm9sZTpNYW5hZ2VyIiwiY3JuOnYxOmJsdWVtaXg6cHVibGljOmlhbTo6Ojpyb2xlOkFkbWluaXN0cmF0b3IiLCJjcm46djE6Ymx1ZW1peDpwdWJsaWM6aWFtOjo6OnJvbGU6RWRpdG9yIiwiY3JuOnYxOmJsdWVtaXg6cHVibGljOmlhbTo6OjpzZXJ2aWNlUm9sZTpXcml0ZXIiXX0sInVzZXJuYW1lIjoiaGVtYW50Lm1hcnZlQGlibS5jb20iLCJkaXNwbGF5X25hbWUiOiJIZW1hbnQgTWFydmUiLCJjZXJ0X3NlcmlhbF9udW1iZXIiOiIyODkwNjQ1MTMzODU5MTQ5ODcyMDU3NjYyNDgxMjgwNzk3OTQ1NDE4MzIiLCJic3NfYWNjb3VudF9pZCI6ImNjMWFlMDA5OGEyZDRlYzM5NDdlY2RlYWE4YTYwZDcxIn0.JwTOLzbLe1Y-O_9bHNFP1sjJFL7IQzm-VwUJBFJN0hWXl1u0UKnBKLz2K9a0xxh2USGC9mVQ_m07MamsLw__gmpWMeRRyK6mMyeIi_JMIJ_dw0cBInsIQTlmIT8NMRHYfW9cKTnOrvEWlQUCeAxsoXn65xay5qqzCMFc0OWc6H67CFGNqUBhJl5szvz51vF7Lce7UGAWCsz8M80i4_IirnJslD8mpIrYX85G09KfwlNuDOecXQgOJ_Vag3rRkhwfUqgJYli5rySEgRFbCAnIuySbrU07HW_Aj5eN0HA5PrYoISfzI8gC1kR4OkijfcKjcAn05KWm7_3EnQyVQqGoXA&#39; \<br>--conf spark.sql.catalog.iceberg_data=org.apache.iceberg.spark.SparkCatalog  \<br>--conf spark.sql.catalog.iceberg_data.type=rest \<br>--conf spark.sql.catalog.iceberg_data.warehouse=iceberg_data \<br>--conf spark.sql.catalog.iceberg_data.s3.path-style-access=true \<br>--conf spark.sql.catalog.iceberg_data.s3.access-key-id=demo_access_key \<br>--conf spark.sql.catalog.iceberg_data.s3.secret-access-key=demo_secret_key \<br>--conf spark.sql.catalog.iceberg_data.client.region=us-south \<br>--conf spark.sql.catalog.iceberg_data.s3.endpoint=https://s3.us-south.cloud-object-storage.appdomain.cloud/ \<br>--conf spark.sql.catalog.iceberg_data.uri=https://506363ea-3273-465c-ae47-3aa3a10196da.cie9nt2d0bngcm5pd3og.lakehouse.dev.appdomain.cloud:32592/mds/iceberg</pre><p>7. After starting spark session you can try some below basic sql commands.</p><pre>a. USE &lt;catalog-name&gt;; //When you call this sql command first time then by default your &lt;catalog-name&gt; which you gave as warehouse during configuration will overrides prefix with &lt;catalog-name&gt; using config endpoint available in iceberg open spec &amp; by default your catalog will be set for each next operation<br>b. create namespace order;<br>c. USE namespace order;<br>d. create table food(id integer,name string);<br>e. insert into food values(1, &#39;pizza&#39;);<br>f. select * from food;<br>g. alter table add columns(price double)<br>g. drop table food PURGE;</pre><blockquote><strong>SUMMARY</strong>: You can continue using the Native Spark engine with the HMS interface for Metadata, but it can also be configured to work with the Iceberg REST Catalog interface. This blog demonstrates how other Spark-based systems can integrate with the Iceberg REST Catalog of MDS, enabling you to mix and match data, metadata, and execution engines as needed.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0bb4f0608cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connect to Snowflake Open Catalog from watsonx.data Spark]]></title>
            <link>https://medium.com/@hemant.marve/connect-to-snowflake-open-catalog-from-watsonx-data-spark-c2f72bd3d102?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2f72bd3d102</guid>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[snowflake-open-catalog]]></category>
            <category><![CDATA[watsonxdata]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 06:16:45 GMT</pubDate>
            <atom:updated>2025-01-22T04:48:59.237Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>In this blog, we will explore how to connect to Snowflake Open Catalog from watsonx.data using Spark. For detailed instructions on setting up Snowflake Open Catalog, please refer to the official documentation here: <a href="https://other-docs.snowflake.com/en/opencatalog/overview">https://other-docs.snowflake.com/en/opencatalog/overview</a></blockquote><p>Snowflake Open Catalog offers an implementation of the Iceberg REST Catalog, which is distinct from the traditional Snowflake Catalog. Unlike Snowflake’s native catalog, Snowflake Open Catalog is designed specifically for managing Iceberg tables. It supports two types of catalogs: <strong>external</strong> and <strong>internal</strong>.</p><ul><li><strong>External Catalog</strong>: This is a read-only catalog that is not managed by Snowflake Open Catalog itself. Instead, it syncs data from an external catalog, enabling Snowflake to access and query Iceberg tables stored outside of its environment.</li><li><strong>Internal Catalog</strong>: This catalog is fully managed by Snowflake Open Catalog. It allows users to perform full CRUD (Create, Read, Update, Delete) operations on Iceberg tables directly within Snowflake.</li></ul><p>With Snowflake Open Catalog, users can seamlessly integrate and manage Iceberg tables, giving them flexibility in how they work with both externally and internally managed catalog. Service connections we create on snowflake side to connect from external engines.</p><p><strong>Steps for submitting spark application in WXD native spark with Snowflake open catalog:</strong></p><p>1. Sample Python Application(<strong>sample_open_catalog.py</strong>) for Snowflake Open Catalog CRUD operations on Iceberg Tables with watsonx.data Spark Integration.</p><pre>from pyspark.sql import SparkSession<br><br>def init_spark():<br>  spark = SparkSession.builder.appName(&quot;snowflake-open-catalog-test&quot;).getOrCreate()<br>  sc = spark.sparkContext<br>  return spark,sc<br><br>def main():<br>  spark,sc = init_spark()<br>  spark.sql(&quot;USE &lt;catalog_name&gt;&quot;)<br>  spark.sql(&quot;SHOW NAMESPACES&quot;).show()<br>  spark.sql(&quot;CREATE NAMESPACE IF NOT EXISTS &lt;namespace/schema&gt;&quot;).show()<br>  spark.sql(&quot;CREATE TABLE IF NOT EXISTS &lt;catalog_name&gt;.&lt;namespace/schema&gt;.&lt;table-name&gt;(ID INTEGER) USING ICEBERG&quot;).show()<br>  spark.sql(&quot;SHOW TABLES from &lt;catalog_name&gt;.&lt;namespace/schema&gt;&quot;).show()<br>  spark.sql(&quot;INSERT INTO &lt;catalog_name&gt;.&lt;namespace/schema&gt;.&lt;table-name&gt; VALUES(1)&quot;).show()<br>  spark.sql(&quot;SELECT * from &lt;catalog_name&gt;.&lt;namespace/schema&gt;.&lt;table-name&gt;&quot;).show()<br>  spark.stop()<br><br>if __name__ == &#39;__main__&#39;:<br>  main()</pre><p>2. Upload <strong>sample_open_catalog.py</strong> file on bucket.</p><p>3. In the documentation <a href="https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-smbit_nsp_1#nsppk_preq_1">https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-smbit_nsp_1#nsppk_preq_1</a>, while submitting the pyspark application in watsonx.data use below request body for snowflake open catalog.</p><pre>{<br>  &quot;application_details&quot;: {<br>    &quot;application&quot;: &quot;&lt;bucket-file-path&gt;&quot;,<br>    &quot;conf&quot;: {<br>      &quot;spark.hadoop.fs.s3a.bucket.&lt;bucket-name-1&gt;.endpoint&quot;: &quot;&lt;bucket-endpoint&gt;&quot;,<br>      &quot;spark.hadoop.fs.s3a.bucket.&lt;bucket-name-1&gt;.access.key&quot;: &quot;&lt;access_key&gt;&quot;,<br>      &quot;spark.hadoop.fs.s3a.bucket.&lt;bucket-name-1&gt;.secret.key&quot;: &quot;&lt;secret_key&gt;&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.uri&quot;: &quot;https://&lt;open_catalog_account_identifier&gt;.snowflakecomputing.com/polaris/api/catalog&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.warehouse&quot;: &quot;iceberg_open_spec&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.scope&quot;: &quot;PRINCIPAL_ROLE:&lt;principal_role_name&gt;&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.type&quot;: &quot;rest&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;&quot;: &quot;org.apache.iceberg.spark.SparkCatalog&quot;,<br>      &quot;spark.sql.defaultCatalog&quot;: &quot;&lt;catalog_name&gt;&quot;,<br>      &quot;spark.sql.extensions&quot;: &quot;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.credential&quot;: &quot;&lt;client_id&gt;:&lt;client_secret&gt;&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.s3.access-key-id&quot;: &quot;&lt;access_key&gt;&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.s3.secret-access-key&quot;: &quot;&lt;secret-key&gt;&quot;,<br>      &quot;spark.sql.catalog.&lt;catalog_name&gt;.client.region&quot;: &quot;&lt;target_s3_region&gt;&quot;,<br>      &quot;spark.sql.defaultCatalog&quot;: &quot;&lt;catalog_name&gt;&quot;<br>    }<br>  }<br>}</pre><p><strong>Explanation of the Configuration Replacer:</strong></p><ol><li><strong>bucket-name-1</strong> : bucket where you uploaded the <strong>sample_open_catalog.py </strong>file.</li><li><strong>bucket-file-path</strong>: Path where you uploaded file on bucket. <strong>eg</strong>: <strong>s3a://bucket-name/sample_open_catalog.py</strong></li><li><strong>catalog-name</strong> : Specifies the name of the catalog to connect to in snowflake open catalog.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mgs1RdyBLR0NAyekMeU4cg.png" /><figcaption><strong>example-&gt; catalog-name is iceberg_open_spec</strong></figcaption></figure><p><strong>4. client_id</strong>: Specifies the client ID for the service principal to use. <strong>client_secret</strong>: Specifies the client secret for the service principal to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bX7mUt_VyU3zM8sKS3Vxuw.png" /><figcaption><strong>Service connection gives client-id:client-secret</strong></figcaption></figure><p><strong>5. open_catalog_account_identifier</strong>: Specifies the account identifier for your Open Catalog account. Depending on the region and cloud platform for the account, this identifier might be the account locator by itself (for example, xy12345) or include additional segments. For more information, see <a href="https://docs.snowflake.com/en/user-guide/admin-account-identifier#using-an-account-locator-as-an-identifier">Using an account locator as an identifier</a>.</p><p><strong>6. principal_role_name:</strong> Specifies the principal role that is granted to the service principal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4mTfRnXR7hPR7sdUcGNwA.png" /><figcaption><strong>principal_role_name is writer</strong></figcaption></figure><blockquote>You can then assign the principal role to the catalog role.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UvmnGh5Alq6QnIg_opMajQ.png" /><figcaption><strong>Writer principal role granted to catalog writer role</strong></figcaption></figure><p><strong>7. target_s3_region</strong>: Specifies the region code where the S3 bucket containing your Apache Iceberg tables is located. For the region codes, see <a href="https://docs.aws.amazon.com/general/latest/gr/s3.html#s3_region">AWS service endpoints</a> and refer to the Region column in the table.</p><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*xpM2wAzYOC117Fs9SaoQcA.png" /><figcaption><strong>Submitted pyspark application</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24liC2y3bAjrQWpoZ8BghA.png" /><figcaption><strong>Spark Application Finished successfully</strong></figcaption></figure><p><strong>Spark driver logs</strong></p><pre>24/12/10 14:23:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>24/12/10 14:23:51 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties<br>SLF4J: Failed to load class &quot;org.slf4j.impl.StaticLoggerBinder&quot;.<br>SLF4J: Defaulting to no-operation (NOP) logger implementation<br>SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.<br>24/12/10 14:23:52 WARN ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead<br>24/12/10 14:23:57 WARN SparkConf: Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN).<br>24/12/10 14:24:02 WARN COSAPIClient: file status spark/spark350/spark-events returned 404<br>24/12/10 14:24:02 WARN COSAPIClient: file status spark/spark350/spark-events/eventlog_v2_app-20241210142359-0000 returned 404<br>24/12/10 14:24:03 WARN COSAPIClient: file status spark/spark350/spark-events/eventlog_v2_app-20241210142359-0000 returned 404<br>+---------+<br>|namespace|<br>+---------+<br>|mm_schema|<br>+---------+<br><br>++<br>||<br>++<br>++<br><br>++<br>||<br>++<br>++<br><br>24/12/10 14:24:17 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes<br><br>[Stage 0:&gt;                                                          (0 + 1) / 1]<br>24/12/10 14:24:19 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(scavenge), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors<br>24/12/10 14:24:19 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(global, scavenge), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors<br><br>                                                                                <br>+---------+---------------+-----------+<br>|namespace|      tableName|isTemporary|<br>+---------+---------------+-----------+<br>|mm_schema|spark_wxd_table|      false|<br>+---------+---------------+-----------+<br><br><br>[Stage 1:&gt;                                                          (0 + 1) / 1]<br><br>                                                                                <br>++<br>||<br>++<br>++<br><br><br>[Stage 2:&gt;                                                          (0 + 1) / 1]<br><br>                                                                                <br><br>[Stage 3:&gt;                                                          (0 + 1) / 1]<br><br>                                                                                <br>+---+<br>| ID|<br>+---+<br>|  1|<br>|  1|<br>|  1|<br>|  1|<br>|  1|<br>|  1|<br>+---+<br><br>24/12/10 14:24:37 WARN COSAPIClient: file status spark/spark350/spark-events/eventlog_v2_app-20241210142359-0000/appstatus_app-20241210142359-0000 returned 404<br>24/12/10 14:24:37 WARN COSAPIClient: file status spark/spark350/spark-events/eventlog_v2_app-20241210142359-0000/appstatus_app-20241210142359-0000 returned 404</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRpbrb-Pu4bHCjKLO9LX9w.png" /><figcaption><strong>In snowflake we can see table created from WXD spark</strong></figcaption></figure><blockquote>We can query this table from where ever we want using any external engine which snowflake catalog supports.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2f72bd3d102" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Iceberg Rest API on WXD: Java Client PART-3 — Appending Data Files]]></title>
            <link>https://medium.com/@hemant.marve/iceberg-rest-api-on-wxd-java-client-part-3-appending-data-files-23fa80e2eede?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/23fa80e2eede</guid>
            <category><![CDATA[java-client]]></category>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[appending-files]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 06:11:36 GMT</pubDate>
            <atom:updated>2025-01-22T04:51:30.798Z</atom:updated>
            <content:encoded><![CDATA[<h3>Iceberg Rest API on watsonx.data: Java Client PART-3 — Appending Data Files</h3><p>In <a href="https://medium.com/@hemant.marve/iceberg-rest-api-on-wxd-java-client-part-1-ac82028433b7">Part1</a> &amp; <a href="https://medium.com/@hemant.marve/iceberg-rest-api-on-wxd-java-client-part-2-4d3af16ae96c">Part2</a> we covered basic operations &amp; how to configure Iceberg Rest catalog Java client to connect to <strong><em>watsonx.data</em></strong> iceberg catalogs.</p><p>In this blog, we’ll walk through the process of appending data files to an existing Iceberg table. We’ll also explore the Iceberg data module, which offers convenient classes for generating records and handling the actual file-writing process.</p><p><strong>Additional Maven dependencies required</strong></p><pre>&lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.iceberg&lt;/groupId&gt;<br>   &lt;artifactId&gt;iceberg-data&lt;/artifactId&gt;<br>   &lt;version&gt;1.6.1&lt;/version&gt;<br>  &lt;/dependency&gt;<br><br>  &lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.iceberg&lt;/groupId&gt;<br>   &lt;artifactId&gt;iceberg-common&lt;/artifactId&gt;<br>   &lt;version&gt;1.6.1&lt;/version&gt;<br>  &lt;/dependency&gt;<br><br>  &lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.iceberg&lt;/groupId&gt;<br>   &lt;artifactId&gt;iceberg-parquet&lt;/artifactId&gt;<br>   &lt;version&gt;1.6.1&lt;/version&gt;<br>  &lt;/dependency&gt;<br><br>  &lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.parquet&lt;/groupId&gt;<br>   &lt;artifactId&gt;parquet-avro&lt;/artifactId&gt;<br>   &lt;version&gt;1.15.0&lt;/version&gt;<br>  &lt;/dependency&gt;<br><br>  &lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.parquet&lt;/groupId&gt;<br>   &lt;artifactId&gt;parquet-common&lt;/artifactId&gt;<br>   &lt;version&gt;1.15.0&lt;/version&gt;<br>  &lt;/dependency&gt;</pre><p>In the following code, we’ll demonstrate how to append data to an Iceberg table. First, we’ll create an Iceberg table. Then, we’ll generate records and create a data file. Finally, we’ll append the generated data file to the Iceberg table.</p><pre>/***<br>     * Base function to call all other methods<br>     * @throws IOException<br>     */<br>    public void appendingDataFilesIcebergDemo() throws IOException {<br>        Namespace namespace = Namespace.of(&quot;Trek_company&quot;);<br><br>        //if namespace doesn&#39;t exists, let&#39;s create first<br>        if(!catalog.namespaceExists(namespace)) {<br>            catalog.createNamespace(namespace);<br>        }<br><br>        TableIdentifier identifier = TableIdentifier.of(namespace, &quot;trekyaari&quot;);<br>        Table table = null;<br><br>        //if table doesn&#39;t exists, let&#39;s create a table first<br>        if(catalog.tableExists(identifier)) {<br><br>            //loading table if already exists<br>            table = catalog.loadTable(identifier);<br>        }<br>        else{<br>            //Let&#39;s create demo table<br>            table = createDemoTable();<br>        }<br>        ImmutableList&lt;GenericRecord&gt; records = createDemoRecords(table.schema());<br><br>        //Let&#39;s finally write data on file &amp; append Data file to an Iceberg table<br>        appendingFileDemo(table, records);<br>    }<br><br>    /**<br>     *<br>     * @return will create &amp; return demo table<br>     */<br>    public Table createDemoTable(){<br>        Schema schema = new Schema(<br>                Types.NestedField.optional(1, &quot;trek_id&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(2, &quot;trek_name&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(3, &quot;location&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(4, &quot;duration&quot;, Types.DoubleType.get()),<br>                Types.NestedField.optional(5, &quot;difficulty&quot;, Types.StringType.get())<br>        );<br><br>        Namespace trekCompany = Namespace.of(&quot;Trek_company&quot;);<br>        TableIdentifier name = TableIdentifier.of(trekCompany, &quot;trekyaari&quot;);<br>        return catalog.createTable(name, schema, PartitionSpec.unpartitioned());<br>    }<br><br>    /**<br>     * Creating demo records for writing on data file<br>     * @param schema<br>     * @return<br>     */<br>    public ImmutableList&lt;GenericRecord&gt; createDemoRecords(Schema schema){<br>        GenericRecord record = GenericRecord.create(schema);<br>        ImmutableList.Builder&lt;GenericRecord&gt; builder = ImmutableList.builder();<br>        builder.add(record.copy(ImmutableMap.of(&quot;trek_id&quot;, UUID.randomUUID().toString(), &quot;trek_name&quot;, &quot;Triund Trek&quot;, &quot;location&quot;, &quot;Himachal Pradesh&quot;, &quot;duration&quot;, 1.0, &quot;difficulty&quot;, &quot;Intermediate&quot;)));<br>        builder.add(record.copy(ImmutableMap.of(&quot;trek_id&quot;, UUID.randomUUID().toString(), &quot;trek_name&quot;, &quot;Kheerganga Trek&quot;, &quot;location&quot;, &quot;Himachal Pradesh&quot;, &quot;duration&quot;, 1.0, &quot;difficulty&quot;, &quot;Beginner&quot;)));<br>        builder.add(record.copy(ImmutableMap.of(&quot;trek_id&quot;, UUID.randomUUID().toString(), &quot;trek_name&quot;, &quot;Kashmir Great Lakes Trek&quot;, &quot;location&quot;, &quot;Kashmir&quot;, &quot;duration&quot;, 2.0, &quot;difficulty&quot;, &quot;Difficult&quot;)));<br>        builder.add(record.copy(ImmutableMap.of(&quot;trek_id&quot;, UUID.randomUUID().toString(), &quot;trek_name&quot;, &quot;Hampta Paas Trek&quot;, &quot;location&quot;, &quot;Himachal Pradesh&quot;, &quot;duration&quot;, 1.0, &quot;difficulty&quot;, &quot;Moderate&quot;)));<br>        ImmutableList&lt;GenericRecord&gt; records = builder.build();<br>        return records;<br>    }<br><br>    /**<br>     * Let&#39;s finally write data on file &amp; append Data file to an Iceberg table<br>     * @param table<br>     * @param records<br>     * @throws IOException<br>     */<br>    public void appendingFileDemo(Table table, List&lt;GenericRecord&gt; records) throws IOException {<br>        String filepath = table.location() + &quot;/&quot; + UUID.randomUUID().toString();<br>        OutputFile file = table.io().newOutputFile(filepath);<br>        DataWriter&lt;GenericRecord&gt; dataWriter =<br>                Parquet.writeData(file)<br>                        .schema(table.schema())<br>                        .createWriterFunc(GenericParquetWriter::buildWriter)<br>                        .overwrite()<br>                        .withSpec(PartitionSpec.unpartitioned())<br>                        .build();<br><br>        try {<br>            for (GenericRecord record : records) {<br>                //writing records on data file<br>                dataWriter.write(record);<br>            }<br>        } finally {<br>            dataWriter.close();<br>        }<br><br>        //contains metadata of file<br>        DataFile dataFile = dataWriter.toDataFile();<br><br>        //appending &amp; committing data file having records<br>        table.newAppend().appendFile(dataFile).commit();<br>    }</pre><p><strong>Results</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2VEZ-Vz55zEHfl7s0Vs7uQ.png" /><figcaption><strong>Data File Appended successfully using Java Client</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23fa80e2eede" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Iceberg Rest API on watsonx.data: Java Client PART-2]]></title>
            <link>https://medium.com/@hemant.marve/iceberg-rest-api-on-wxd-java-client-part-2-4d3af16ae96c?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d3af16ae96c</guid>
            <category><![CDATA[multitable]]></category>
            <category><![CDATA[java-client]]></category>
            <category><![CDATA[updates]]></category>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 05:45:52 GMT</pubDate>
            <atom:updated>2025-01-22T04:50:37.865Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@hemant.marve/iceberg-rest-api-on-wxd-java-client-part-1-ac82028433b7">Part1</a>, we covered how to connect with Iceberg Type catalog in w<em>atsonx.data</em> &amp; use it to create a namespace &amp; table with a defined schema. In this <strong>second</strong> <strong>part</strong> of the Iceberg Rest Catalog JavaClient, we will explore how to commit updates to multiple tables in a single atomic operation, a feature newly introduced in the Iceberg REST Catalog.</p><blockquote><strong>Note</strong>: Any commit made to transaction object will be a temporary or in-memory commit. So, don’t get confused with actual commit &amp; transaction commit.</blockquote><p>In Below code we are loading two tables from watsonx.data &amp; intializing transaction for each table. With transaction objects we are going to update schema &amp; will do in-memory commit. After in-memory commit successful we are going to call commitTransaction method for multi table commits in atomic manner.</p><pre>/**<br>     * Example to show how to do multi-table commit in single transaction<br>     */<br>    public void multiTableAtomicCommit() {<br>        Namespace namespace = Namespace.of(&quot;Trek_company&quot;);<br><br>        catalog.createNamespace(namespace);<br><br>        TableIdentifier identifier1 = TableIdentifier.of(namespace, &quot;trekyaari&quot;);<br>        TableIdentifier identifier2 = TableIdentifier.of(namespace, &quot;trekIndia&quot;);<br><br>        //schema with which we are creating both tables<br>        Schema schema = new Schema(<br>                Types.NestedField.optional(1, &quot;Trek_id&quot;, Types.LongType.get()),<br>                Types.NestedField.optional(2, &quot;Trek_name&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(3, &quot;Location&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(4, &quot;Duration&quot;, Types.DoubleType.get()),<br>                Types.NestedField.optional(5, &quot;Difficulty&quot;, Types.StringType.get())<br>        );<br><br>        //creating two tables for demo of multi-table atomic commit, will use both tables<br>        catalog.createTable(identifier1, schema);<br>        catalog.createTable(identifier2, schema);<br><br><br>        //loading both tables created above<br>        Table table1 = catalog.loadTable(identifier1);<br>        Table table2 = catalog.loadTable(identifier2);<br><br>        Transaction t1Transaction = catalog.loadTable(identifier1).newTransaction();<br>        //adding column with name minimum_age in table1<br>        t1Transaction.updateSchema().addColumn(&quot;minimum_age&quot;, Types.LongType.get()).commit();<br><br>        //renaming Difficulty column with name as challenge<br>        Transaction t2Transaction = catalog.loadTable(identifier2).newTransaction();<br>        t2Transaction.updateSchema().renameColumn(&quot;Difficulty&quot;, &quot;challenge&quot;).commit();<br><br>        TableCommit tableCommit1 =<br>                TableCommit.create(<br>                        identifier1,<br>                        ((BaseTransaction) t1Transaction).startMetadata(),<br>                        ((BaseTransaction) t1Transaction).currentMetadata());<br><br>        TableCommit tableCommit2 =<br>                TableCommit.create(<br>                        identifier2,<br>                        ((BaseTransaction) t2Transaction).startMetadata(),<br>                        ((BaseTransaction) t2Transaction).currentMetadata());<br><br>        //actual commit for both transaction will happen here<br>        catalog.commitTransaction(tableCommit1, tableCommit2);<br><br>        //loading both tables again after multi-table atomic commit<br>        table1 = catalog.loadTable(identifier1);<br>        table2 = catalog.loadTable(identifier2);<br><br><br>        System.out.println(&quot;Below you can see Table1 updated schema after commit&quot;);<br>        System.out.println(table1.schema());<br>        System.out.println();<br>        System.out.println(&quot;Below you can see Table2 updated schema after commit&quot;);<br>        System.out.println(table2.schema());<br><br>    }</pre><h3>Results:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X6tbG4z8mocWtHn3MdPd7A.png" /><figcaption><strong>watsonx.data</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAO1pbSwj6d2Ud32DjYHLQ.png" /><figcaption><strong>Java Client</strong></figcaption></figure><blockquote><strong>Now, let’s introduce a conflict. As a result, both transactions will fail.</strong></blockquote><p>In the code below, we load two tables from <strong>watsonx.data</strong> and initialize a transaction for each table. Using the transaction objects, we update the schema and perform an in-memory commit. This time, however, before the actual commit, we directly delete a column without involving the transaction. After the column deletion, we attempt to commit both transactions.</p><p>The second transaction, which involves renaming the column, will fail because the column has already been deleted. As a result, the first transaction will also fail, and no changes will be made to <strong>table1</strong> either.</p><pre>public void multiTableAtomicCommitConflict() {<br>        Namespace namespace = Namespace.of(&quot;Trek_company&quot;);<br><br>        catalog.createNamespace(namespace);<br><br>        TableIdentifier identifier1 = TableIdentifier.of(namespace, &quot;trekyaari&quot;);<br>        TableIdentifier identifier2 = TableIdentifier.of(namespace, &quot;trekIndia&quot;);<br><br>        //schema with which we are creating both tables<br>        Schema schema = new Schema(<br>                Types.NestedField.optional(1, &quot;Trek_id&quot;, Types.LongType.get()),<br>                Types.NestedField.optional(2, &quot;Trek_name&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(3, &quot;Location&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(4, &quot;Duration&quot;, Types.DoubleType.get()),<br>                Types.NestedField.optional(5, &quot;Difficulty&quot;, Types.StringType.get())<br>        );<br><br>        //creating two tables for demo of multi-table atomic commit, will use both tables<br>        catalog.createTable(identifier1, schema);<br>        catalog.createTable(identifier2, schema);<br><br><br>        //loading both tables created above<br>        Table table1 = catalog.loadTable(identifier1);<br>        Table table2 = catalog.loadTable(identifier2);<br><br>        Transaction t1Transaction = catalog.loadTable(identifier1).newTransaction();<br>        //adding column with name minimum_age in table1<br>        t1Transaction.updateSchema().addColumn(&quot;minimum_age&quot;, Types.LongType.get()).commit();<br><br>        //renaming Difficulty column with name as challenge<br>        Transaction t2Transaction = catalog.loadTable(identifier2).newTransaction();<br>        t2Transaction.updateSchema().renameColumn(&quot;Difficulty&quot;, &quot;challenge&quot;).commit();<br><br>        // delete the colum Difficulty that is being renamed in the above TX2 to cause a conflict<br>        table2.updateSchema().deleteColumn(&quot;Difficulty&quot;).commit(); //This is actual commit, will call single commit update API<br><br>        TableCommit tableCommit1 =<br>                TableCommit.create(<br>                        identifier1,<br>                        ((BaseTransaction) t1Transaction).startMetadata(),<br>                        ((BaseTransaction) t1Transaction).currentMetadata());<br><br>        TableCommit tableCommit2 =<br>                TableCommit.create(<br>                        identifier2,<br>                        ((BaseTransaction) t2Transaction).startMetadata(),<br>                        ((BaseTransaction) t2Transaction).currentMetadata());<br><br>        //Will fail during commitTransaction<br>        catalog.commitTransaction(tableCommit1, tableCommit2);<br><br>        //loading both tables again after multi-table atomic commit<br>        table1 = catalog.loadTable(identifier1);<br>        table2 = catalog.loadTable(identifier2);<br><br><br>        System.out.println(&quot;Below you can see Table1 updated schema after commit&quot;);<br>        System.out.println(table1.schema());<br>        System.out.println();<br>        System.out.println(&quot;Below you can see Table2 updated schema after commit&quot;);<br>        System.out.println(table2.schema());<br><br>    }</pre><p><strong>Expected Results:</strong> After the commit, you’ll observe that the <strong>Difficulty</strong> column has been deleted from <strong>table2</strong> due to the actual commit. However, no new column will be added to <strong>table1</strong> because the rename operation for the <strong>Difficulty</strong> column in <strong>table2</strong> could not be performed — since the column was already deleted before the transaction could be committed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jVsyOzbWjwCTKm6Vmj47w.png" /><figcaption><strong>watsonx.data</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gpnsvXdSpKXXLgNPHT04eA.png" /><figcaption><strong>Java Client Side Error</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d3af16ae96c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Iceberg Rest API on watsonx.data: Java Client PART-1]]></title>
            <link>https://medium.com/@hemant.marve/iceberg-rest-api-on-wxd-java-client-part-1-ac82028433b7?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac82028433b7</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[crud-operations]]></category>
            <category><![CDATA[iceberg-table]]></category>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <category><![CDATA[java-client]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Thu, 02 Jan 2025 08:11:36 GMT</pubDate>
            <atom:updated>2025-01-22T04:52:01.829Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post, we’ll explore how to use the Iceberg Java client to interact with Iceberg tables in WXD, leveraging the MDS Iceberg REST Catalog implementation for seamless table management and operations.</p><h3>1. The Maven Dependencies</h3><pre>&lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.iceberg&lt;/groupId&gt;<br>   &lt;artifactId&gt;iceberg-api&lt;/artifactId&gt;<br>   &lt;version&gt;1.6.1&lt;/version&gt;<br> &lt;/dependency&gt;<br><br> &lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;<br>   &lt;artifactId&gt;hadoop-aws&lt;/artifactId&gt;<br>   &lt;version&gt;3.4.0&lt;/version&gt;<br> &lt;/dependency&gt;<br><br> &lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.iceberg&lt;/groupId&gt;<br>   &lt;artifactId&gt;iceberg-core&lt;/artifactId&gt;<br>   &lt;version&gt;1.6.1&lt;/version&gt;<br> &lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>   &lt;groupId&gt;org.apache.iceberg&lt;/groupId&gt;<br>   &lt;artifactId&gt;iceberg-aws&lt;/artifactId&gt;<br>   &lt;version&gt;1.6.1&lt;/version&gt;<br>&lt;/dependency&gt;<br></pre><p>2. In WXD.data go to infrastructure manager &amp; create an iceberg type catalog associated with any bucket of type S3a.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z8fehqdFQ3YdHTjg.png" /><figcaption><strong>Catalog Details</strong></figcaption></figure><p>3. You will able to see MDS Rest endpoint in catalog details &amp; for CPD please use MDS EXTERNAL endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PkWDpgdvJuTmRU-C.png" /><figcaption><strong>CPD</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j31__WbXX6ziLaFH.jpeg" /><figcaption><strong>SAAS</strong></figcaption></figure><blockquote><strong><em>Skip 4–6 steps if you are using watsonx.data on cloud.</em></strong></blockquote><p>4. In case of CPD you need to fetch certificates using below command.</p><pre>echo QUIT | openssl s_client -showcerts -connect &lt;mds-rest-endpoint-host&gt;:443 | awk &#39;/-----BEGIN CERTIFICATE-----/ {p=1}; p; /-----END CERTIFICATE-----/ {p=0}&#39; &gt; mds.cert</pre><p>5. <strong>cat mds.cert</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UkEcNOAlX9IkbSPX.png" /></figure><p>6. Import the both certificates into the Java truststore that Spark will use at runtime.</p><pre>keytool -import -alias &lt;alias_name&gt; -file &lt;certificate_file&gt; -keystore &lt;truststore_file&gt; -storepass &lt;truststore_password&gt;</pre><p>EXAMPLE:</p><pre>sudo keytool -import -alias mycert -file mycert.crt -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit</pre><blockquote><strong>Required Roles:</strong> For GET/HEAD endpoints ADMIN/Metastore Admin/Metastore Viewer &amp; for other endpoints required Admin/Metastore Admin</blockquote><p>7.In the code below, we will configure the <strong>Iceberg REST Catalog</strong> Java client, create a <strong>Namespace</strong>, create a <strong>Table</strong>, load the <strong>Table</strong>, and finally, commit updates to a single table.</p><ol><li><strong>Create Namespace</strong>-&gt; In this we will create a namespace which will call schema in <em>watsonx.data</em> in terms of query workspace.</li><li><strong>Create Table</strong> -&gt; In this we will create a Iceberg table within schema/namespace.</li><li><strong>Load Table</strong> -&gt; In this we will see how to load already existing iceberg tables within <em>watsonx.data</em> iceberg type catalog.</li><li><strong>Commit Updates</strong> -&gt; In this we will see a example how to commit updates to single table directly without transaction.</li></ol><pre>import org.apache.iceberg.*;<br>import org.apache.iceberg.aws.s3.S3FileIOProperties;<br>import org.apache.iceberg.catalog.Namespace;<br>import org.apache.iceberg.catalog.TableIdentifier;<br>import org.apache.iceberg.rest.RESTCatalog;<br>import java.util.HashMap;<br>import java.util.Map;<br><br>import static org.apache.iceberg.ManifestFile.SCHEMA;<br><br>public class RestCatalogAPIService {<br><br>    //JAVA CLIENT<br>    private final RESTCatalog catalog = new RESTCatalog();<br><br>    /**<br>     * Configuring Iceberg RestCatalog to connect with watsonx.data Iceberg type catalog<br>     * Please replace &lt;wxd-catalog-name&gt; with watsonx.data Iceberg catalog name<br>     */<br>    public RestCatalogAPIService(){<br>        // Configuration for the REST catalog java api client<br>        Map&lt;String, String&gt; properties = new HashMap&lt;&gt;();<br>        properties.put(CatalogProperties.CATALOG_IMPL, &quot;org.apache.iceberg.rest.RESTCatalog&quot;);<br><br>        //URL pointing to mds-rest service<br>        properties.put(CatalogProperties.URI, &quot;http://&lt;mds-rest-domain&gt;/mds/iceberg&quot;);<br>        properties.put(CatalogProperties.WAREHOUSE_LOCATION, &quot;&lt;wxd-catalog-name&gt;&quot;);<br>        properties.put(&quot;header.Authorization&quot;,&quot;Bearer &lt;bearer-token&gt;&quot;);<br>        //properties.put(&quot;header.Authorization&quot;,&quot;Basic &lt;base64 encoded &lt;username:password&gt;&gt;&quot;);<br>        properties.put(CatalogProperties.FILE_IO_IMPL, &quot;org.apache.iceberg.aws.s3.S3FileIO&quot;);<br>        properties.put(S3FileIOProperties.ACCESS_KEY_ID, &quot;&lt;access-key&gt;&quot;);<br>        properties.put(S3FileIOProperties.SECRET_ACCESS_KEY, &quot;&lt;secret-key&gt;&quot;);<br>        String bucketEndpoint = &quot;&lt;endpoint&gt;&quot;;<br>        properties.put(S3FileIOProperties.ENDPOINT, bucketEndpoint);<br>        System.setProperty(&quot;aws.region&quot;, &quot;&lt;region&gt;&quot;);//Only way to force region properties.put(S3FileIOProperties.PRELOAD_CLIENT_ENABLED, &quot;true&quot;); properties.put(S3FileIOProperties.USE_ARN_REGION_ENABLED, &quot;true&quot;);<br>        catalog.initialize(&quot;&lt;wxd-catalog-name&gt;&quot;, properties);<br>    }<br><br>    /**<br>     * Create a schema in watsonx.data with configured Iceberg type wxd-catalog<br>     * @param namespace name of the schema<br>     */<br>    public void createNamespace(String namespace){<br>        try {<br>            //creating a namespace<br>            catalog.createNamespace(Namespace.of(namespace));<br>            System.out.println(&quot;Namespace created successfully, &quot; + namespace);<br>        } catch(Exception e) {<br>            e.printStackTrace();<br>        }<br>    }<br><br>    /**<br>     * Create a table in watsonx.data with configured Iceberg type wxd-catalog<br>     * @param schema Schema in which you intend to create a table<br>     * @param tableName name of the table<br>     */<br>    public void createTable(String schema, String tableName){<br>        Namespace namespace = Namespace.of(schema);<br>        TableIdentifier identifier1 = TableIdentifier.of(namespace, tableName);<br>        <br>        //table schema definig columns &amp; columns data type<br>        Schema schema = new Schema(<br>                Types.NestedField.optional(1, &quot;Trek_id&quot;, Types.LongType.get()),<br>                Types.NestedField.optional(2, &quot;Trek_name&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(3, &quot;Location&quot;, Types.StringType.get()),<br>                Types.NestedField.optional(4, &quot;Duration&quot;, Types.DoubleType.get()),<br>                Types.NestedField.optional(5, &quot;Difficulty&quot;, Types.StringType.get())<br>        );<br>        //creating a table<br>        catalog.createTable(identifier1, tableSchema);<br>        System.out.println(&quot;Table created successfully, &quot; + schema + &quot;.&quot; + tableName);<br>    }<br><br>    /**<br>     * Load Iceberg table from watsonx.data<br>     * @param schema<br>     * @param tableName<br>     */<br>    public void loadTable(String schema, String tableName){<br>        try {<br>            TableIdentifier identifier = TableIdentifier.of(Namespace.of(schema), tableName);<br><br>            //loading a table<br>            Table table = catalog.loadTable(identifier);<br>            System.out.println(String.format(&quot;%s.%s table loaded successfully %s&quot;, schema, tableName, table));<br>        } catch(Exception e) {<br>            e.printStackTrace();<br>        }<br>    }<br><br>  /**<br>     * Commit updates to a Iceberg table<br>     * @param schema<br>     * @param tableName<br>     */<br>    public void commitUpdatesToSingleTable(String schema, String tableName){<br>        try {<br>            TableIdentifier identifier = TableIdentifier.of(Namespace.of(schema), tableName);<br><br>            //loading a table<br>            Table table = catalog.loadTable(identifier);<br>            System.out.println(String.format(&quot;%s.%s table loaded successfully %s&quot;, schema, tableName, table));<br><br>            //single commit to Iceberg Table<br>            table.updateSchema().addColumn(&quot;new_column&quot;, Types.StringType.get()).commit();<br><br>            System.out.println(String.format(&quot;%s.%s New column added successfully to table %s&quot;, schema, tableName, table));<br><br>            //load table again after updating schema<br>            Table table = catalog.loadTable(identifier);<br><br>            System.out.println(&quot;Below you can see table updated schema after commit&quot;);<br>        } catch(Exception e) {<br>            e.printStackTrace();<br>        }<br>    }<br><br>}</pre><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XIoEsIn2Rkp-LhWsq6nFXw.png" /><figcaption><strong>Rest Catalog configuration example</strong></figcaption></figure><h3><strong>Results:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zZnDjGmd1xQkd03xpeLr8w.png" /><figcaption><strong>Java Client console output</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W2B2AxEkm-BVC81AuAbwnQ.png" /><figcaption><strong>Schema &amp; table created successfully in WXD iceberg_data catalog</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac82028433b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sync watsonx.data-Managed]]></title>
            <link>https://medium.com/@hemant.marve/sync-wxd-managed-iceberg-tables-in-snowflake-using-the-mds-iceberg-open-specification-rest-59d7966f5e62?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/59d7966f5e62</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[iceberg-rest-catalog]]></category>
            <category><![CDATA[wxd]]></category>
            <category><![CDATA[iceberg-table]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Thu, 02 Jan 2025 08:08:50 GMT</pubDate>
            <atom:updated>2025-01-22T04:52:39.288Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sync watsonx.data-Managed Iceberg Tables in Snowflake Using the MDS Iceberg Open Specification REST Implementation- SAAS</h3><p><strong>Parent Topic:</strong> <a href="https://mrudulamadiraju.medium.com/unity-catalog-api-iceberg-rest-catalog-api-in-watsonx-data-a4600cd8fa04">https://mrudulamadiraju.medium.com/unity-catalog-api-iceberg-rest-catalog-api-in-watsonx-data-a4600cd8fa04</a></p><p><strong>Contributors:</strong> <a href="https://mrudulamadiraju.medium.com/">Mrudula Madiraju</a></p><p>We can create a read-only external catalog within Snowflake, which can be integrated with the Iceberg REST Catalog implementation based on the open specification. In MDS, we have already implemented the Iceberg REST Catalog, enabling us to sync Iceberg tables from MDS to Snowflake. Best part you can provide a time interval to refresh synced tables again with new data &amp; metadata. Additionally, you can specify a time interval for refreshing the synced tables, ensuring that both data and metadata are updated automatically at the desired frequency.</p><blockquote>Follow this documentation to set up watsonx.data if you haven’t already: w<a href="https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getstarted-console">atsonx.data Setup Guide</a>.</blockquote><blockquote>Follow this documentation to set up Snowflake if you haven’t already: <a href="https://docs.snowflake.com/user-guide-getting-started">https://docs.snowflake.com/user-guide-getting-started</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuKKap554SFtVWfLhBJTwQ.jpeg" /><figcaption>Sync Data from watsonx.data Iceberg tables into Snowflake Catalog</figcaption></figure><p><strong>Let’s explore how we can sync an Iceberg table created in WXD with Snowflake. For more clarity, please refer to the examples also.</strong></p><p><strong>Example</strong>: In the example, you will see how to sync the Iceberg table iceberg_data.wxd_schema.wxd_table from WXD to Snowflake. I used the Presto engine in WXD to create the Iceberg table, but you can use other methods such as data ingestion, Spark, or alternative engines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZooY63qgYw3zm_7VFUzWPA.png" /><figcaption><strong>WXD Presto Query workspace</strong></figcaption></figure><p><strong>Steps to Sync Iceberg Tables from MDS to Snowflake:</strong></p><blockquote>Execute all below steps on snowflake side using query worksheet.</blockquote><ol><li>Create an External Volume in Snowflake to Point to the Same Bucket as the WXD Iceberg Catalog.</li></ol><pre>CREATE OR REPLACE EXTERNAL VOLUME &lt;my_external_volume&gt;<br>   STORAGE_LOCATIONS =<br>      (<br>         (<br>            NAME = &#39;&lt;bucket-name&gt;&#39;<br>            STORAGE_PROVIDER = &#39;S3COMPAT&#39;<br>            STORAGE_BASE_URL = &#39;S3COMPAT://&lt;bucket-name&gt;/&#39;<br>            CREDENTIALS = (<br>                AWS_KEY_ID = &#39;&lt;access-key&gt;&#39;<br>                AWS_SECRET_KEY = &#39;&lt;secret-key&gt;&#39;<br>                )<br>            STORAGE_ENDPOINT=&#39;&lt;endpoint&gt;&#39;<br>         )<br>      );</pre><p><strong>Example</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDgJQrmMQaYnmZWs9i3hMA.jpeg" /><figcaption><strong>External volume connected with same bucket as WXD catalog</strong></figcaption></figure><p>2. Create a Read-Only External Catalog Integration with the MDS Iceberg REST Catalog in snowflake.</p><blockquote>Please follow blog to check how to setup AWS gateway as proxy for MDS-REST endpoint url. <a href="https://medium.com/@hemant.marve/using-aws-api-gateway-as-a-proxy-between-snowflake-and-wxd-2c3dae879a79">https://medium.com/@hemant.marve/using-aws-api-gateway-as-a-proxy-between-snowflake-and-wxd-2c3dae879a79</a></blockquote><pre>CREATE OR REPLACE CATALOG INTEGRATION &lt;external-catalog-integration-name&gt;<br>  CATALOG_SOURCE = POLARIS<br>  TABLE_FORMAT = ICEBERG<br>  CATALOG_NAMESPACE = &#39;&lt;wxd schema or namespace&gt;&#39;<br>  REST_CONFIG = (<br>    CATALOG_URI = &#39;https://&lt;domain&gt;/mds/iceberg&#39;<br>    WAREHOUSE = &#39;&lt;wxd-catalog&gt;&#39;<br>  )<br>  REST_AUTHENTICATION = (<br>    TYPE = BEARER<br>    BEARER_TOKEN = &#39;&lt;wxd-bearer-token&gt;&#39;<br>  )<br>  ENABLED = TRUE<br>  REFRESH_INTERVAL_SECONDS = 60;</pre><p><strong>Example</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_NcDDjqhA9fCr1lD6-oi3g.jpeg" /><figcaption><strong>Catalog Integration with WXD iceberg_data catalog</strong></figcaption></figure><p>3. Sync a WXD Iceberg Table Using External Catalog Integration.</p><pre>CREATE OR REPLACE ICEBERG TABLE &lt;snowflake-side-table-name&gt;<br>  EXTERNAL_VOLUME = &#39;&lt;my_external_volume&gt;&#39;<br>  CATALOG = &#39;&lt;external-catalog-integration-name&gt;&#39;<br>  CATALOG_TABLE_NAME = &#39;&lt;wxd-iceberg-table-name&gt;&#39;<br>  CATALOG_NAMESPACE = &#39;&lt;wxd namespace or schema&gt;&#39;<br>  AUTO_REFRESH = TRUE;</pre><blockquote>Set AUTO_REFRESH to FALSE if you don&#39;t want the tables to be automatically synced after the specified REFRESH_INTERVAL_SECONDS.</blockquote><p><strong>Example</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*O8eQ5KkvciyxwYjehtdyog.jpeg" /><figcaption><strong>Table creation in snowflake with catalog integration</strong></figcaption></figure><p>4. Test the Iceberg Table Sync.</p><pre>SELECT * FROM &lt;snowflake-side-table-name&gt;;</pre><p><strong>Result</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4YV287GFRueT_5p11YwLw.jpeg" /><figcaption><strong>Iceberg table synced in snowflake from WXD</strong></figcaption></figure><p>5. Update the Iceberg Table on the WXD Side.</p><blockquote>On the WXD side, make any updates to the Iceberg table (e.g., inserting new data or changing metadata).</blockquote><p><strong>Example</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gZGhpJ_G-tBjlVmTIedqHA.png" /><figcaption><strong>Let’s add some more data in same table from WXD</strong></figcaption></figure><p>6. Automatic Data and Metadata Refresh in Snowflake.</p><blockquote>After the update on the WXD side, Metadata and data in Snowflake side table automatically refreshed after a specified <em>REFRESH_INTERVAL_SECONDS</em>.</blockquote><p><strong>Result</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gy3YNHxSAUMsXuvNv6w5gQ.jpeg" /><figcaption><strong>New data also synced in snowflake automatically</strong></figcaption></figure><p>7. Repeat the same steps for all iceberg tables which you want to sync into snowflake.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59d7966f5e62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS API Gateway as a Proxy Between Snowflake and watsonx.data]]></title>
            <link>https://medium.com/@hemant.marve/using-aws-api-gateway-as-a-proxy-between-snowflake-and-wxd-2c3dae879a79?source=rss-15fe6d933637------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c3dae879a79</guid>
            <category><![CDATA[aws-gateway]]></category>
            <category><![CDATA[proxy]]></category>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[https]]></category>
            <category><![CDATA[wxd]]></category>
            <dc:creator><![CDATA[Hemant Marve]]></dc:creator>
            <pubDate>Thu, 02 Jan 2025 08:07:04 GMT</pubDate>
            <atom:updated>2025-01-22T04:52:58.435Z</atom:updated>
            <content:encoded><![CDATA[<p>Currently, we use an API Gateway as a proxy because the Iceberg REST Catalog is available only on the data plane in WXD. To expose the Iceberg REST Catalog endpoint, we’ve employed a NodePort, which results in the endpoint running on port 32671 instead of the standard port 443. Since Snowflake restricts connections to port 443, the API Gateway serves as a proxy to facilitate the connection. While any API Gateway can be used for this purpose, this example demonstrates how to use AWS API Gateway. Looking ahead, we plan to expose all Iceberg REST Catalog APIs directly through console endpoint, then we don’t need API gateway.</p><p><strong>Steps to Fetch the WXD MDS REST Endpoint Exposed via NodePort and Call It from AWS API Gateway Instead of Directly from Snowflake:</strong></p><p>1. Go to infrastructure manager &amp; click on your iceberg catalog for catalog details.</p><p>2. You will able to see MDS Rest endpoint in catalog details. Just copy &amp; use it in AWS HTTP API gateway integration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2AG0K3hi1WCIwHYeoMaUDA.jpeg" /><figcaption><strong>WXD iceberg Catalog</strong></figcaption></figure><p><strong>Steps to setup AWS API Gateway as a Proxy between WXD &amp; snowflake:</strong></p><blockquote>Don’t use AWS Lambda function integration use <strong>HTTP proxy integrations for HTTP APIs</strong></blockquote><blockquote><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop.html">https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-http.html</a></blockquote><ol><li>Create a route with ANY method using {proxy+} in HTTP API gateway.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqGIJgtMs1IO6_sFy9lH7Q.png" /></figure><p>2. Create Http integration to add MDS endpoint. Append {proxy} at the end of wxd rest endpoint. <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-http.html">https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-http.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wQ6IJcEDYRAaiBcttrT8_A.png" /></figure><p>3. Review &amp; create.</p><p>4. Now your API is ready to invoke. Copy the invoke URL which you have to use now for WXD Iceberg Rest Catalog Integration in snowflake. Don’t forget to append <strong><em>/mds/iceberg</em> </strong>at the end of invoke url in snowflake.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c3dae879a79" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>