<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shreshta Balachandar on Medium]]></title>
        <description><![CDATA[Stories by Shreshta Balachandar on Medium]]></description>
        <link>https://medium.com/@shreshta.balachandar?source=rss-1fa856658f58------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*PTvx3tWnKAudHdwq</url>
            <title>Stories by Shreshta Balachandar on Medium</title>
            <link>https://medium.com/@shreshta.balachandar?source=rss-1fa856658f58------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:30:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shreshta.balachandar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running PySpark Splink Workloads Directly on Snowflake]]></title>
            <link>https://medium.com/snowflake/running-pyspark-splink-workloads-directly-on-snowflake-5a99937026d8?source=rss-1fa856658f58------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a99937026d8</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[identity-resolution]]></category>
            <category><![CDATA[snowpark-connect]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[splink]]></category>
            <dc:creator><![CDATA[Shreshta Balachandar]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 19:01:01 GMT</pubDate>
            <atom:updated>2026-07-31T19:01:01.520Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Say goodbye to managing external Spark clusters for identity resolution!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pff8Oi3t6oDf4FMhX_ZfEQ.png" /></figure><p>It’s never been easier to run PySpark workloads directly on Snowflake. The new Snowpark Connect feature lets you bring your existing PySpark Splink workloads to Snowflake without needing to rewrite the Splink identity resolution logic you’ve created. If you’re currently using a separate Spark cluster to run your Splink workloads, you will find it easy to migrate your code to run on Snowflake natively. If you’re exploring Snowflake and currently using Splink with PySpark, you can quickly get up and running on Snowflake.</p><h3><strong>What is Identity Resolution?</strong></h3><p>If you’ve ever needed to de-duplicate customer records across different social channels, link patient records across health systems, or resolve ad impressions to actual users across devices, you’re likely familiar with identity resolution and the challenges it presents. For the uninitiated, identity resolution is the process of determining which records across (or within) datasets refer to the same real-world entity (usually a person).</p><p>One might think this would be straightforward. Why not just match by an email or phone number? But in the real-world names get misspelled, birth years get entered as the current year, and email addresses change. You can’t just do exact matching and hence the challenge is fundamentally probabilistic.</p><p><strong>Splink</strong> is a widely used open-source library for Identity Resolution. Developed by the analytics team at the <strong>UK Ministry of Justice</strong> and used across ad-tech, healthcare, and financial services, Splink implements Fellegi-Sunter probabilistic record linkage with an easy-to-use API. The details behind this algorithm are available <a href="https://moj-analytical-services.github.io/splink/topic_guides/theory/fellegi_sunter.html">here</a>. Officially, Splink supports DuckDB, Apache Spark, AWS Athena, SQLite and Postgres as backends (note: Athena support is being removed in Splink 5) for running its algorithms.</p><h3><strong>The Misconception</strong></h3><p>If you have data in Snowflake, you might be exporting data out to a Spark environment and dealing with the additional complexity of managing another compute plane. In fact, when I ask leading LLMs “How do I run Splink on Snowflake?”, I get a response suggesting exporting my data to a standalone Spark cluster, running Splink and pushing the results back.</p><p>Historically, this was the only option, but this spotlights the challenge of depending on LLMs for accurate answers when their training data contains gaps. Snowpark Connect for Spark is a relatively new capability, and there isn’t as much training data in the wild. When I pushed back on the LLMs and suggested Snowpark Connect, the models reconsider and realize that it’s not only feasible, but also the simpler path.</p><h3><strong>Enter Snowpark Connect for Apache Spark</strong>™</h3><p>Snowpark Connect for Apache Spark™ (SCOS) is a massive engineering accomplishment by the Snowflake team. It implements the <strong>Spark Connect protocol,</strong> the same wire protocol that PySpark 3.4+ uses to communicate with a Spark cluster, and converts the Spark plan into a Snowpark plan that a Snowflake virtual warehouse can understand and execute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*GWclWIIFkI0RV6e2VHSHfQ.png" /><figcaption>Snowpark Connect Architecture</figcaption></figure><p>Imagine being able to run your existing PySpark code on Snowflake with no Spark clusters to manage. Libraries built on top of PySpark become candidates for running on Snowflake with minimal code changes. Splink is one compelling example, but the pattern can apply to other PySpark-native libraries you want to bring along. The implications are broader than convenience.</p><h3><strong>The Blueprint: Running Splink on Snowflake</strong></h3><p>Below, I will cover a step-by-step blueprint for running Splink. The patterns I cover (setup, JAR provisioning, UDF registration, etc.) apply to other PySpark libraries you want to run on SCOS. Running Splink on Snowflake will require setting up 2 types of compute:</p><ul><li>Virtual Warehouse: This will run the transpiled SparkSQL code that Splink generates</li><li>Compute Pool: This will be used to host the Jupyter notebook kernel to run the Splink code interactively. This is optional and with the new Snowflake Code Bundles (currently in private preview), you can entirely bypass this and run it as jobs on a virtual warehouse.</li></ul><p>The full working code is available in the companion <a href="https://github.com/sfc-gh-sbalachandar/splink-on-snowflake">repository</a>. The setup steps below will show you how you can spin up the compute resources you need</p><h3><strong>Setup</strong></h3><p>Before running any notebooks, provision your Snowflake environment with this <a href="https://github.com/sfc-gh-sbalachandar/splink-on-snowflake/blob/main/00_setup.sql">setup script</a>:</p><ol><li><strong>Create a role, warehouse, database, and schema</strong></li><li><strong>Create a compute pool</strong> — this runs your Container Runtime notebook (the PySpark client environment)</li><li><strong>Create an External Access Integration</strong> — Splink downloads sample data from GitHub at runtime, so the compute pool needs egress to `raw.githubusercontent.com`. This is not required if you are operating on data in Snowflake directly</li><li><strong>Add the built-in PyPi artifact repository</strong> to the compute pool so Python packages can be installed</li></ol><p>See the screenshot below for how to set up your notebook container.</p><figure><img alt="Notebook service setup configs" src="https://cdn-images-1.medium.com/max/918/1*sjNlXe6ZSu0RJQ8E2lTBJQ.png" /><figcaption>Snowflake Notebook Service Setup Configuration</figcaption></figure><h3><strong>Setting up the Spark Object</strong></h3><p>Using the following code snippet, you can initialize your Spark object that you can use for the rest of your notebook.</p><pre>import pyspark<br>from snowflake.snowpark_connect import init_spark_session<br><br>conf = pyspark.SparkConf()<br>conf.set(&quot;spark.sql.session.timeZone&quot;, &quot;UTC&quot;)<br>conf.set(&quot;spark.sql.ansi.enabled&quot;, &quot;false&quot;)<br>conf.set(&quot;snowpark.connect.cte.optimization_enabled&quot;, &quot;true&quot;)<br>conf.set(&quot;snowpark.connect.integralTypesEmulation&quot;, &quot;enabled&quot;)<br><br>spark = init_spark_session(app_name=&quot;splink&quot;, conf=conf)</pre><h4><strong>Approach 1: Lift-and-Shift with Java UDF JARs</strong></h4><p>The most direct path. Splink ships with a Scala-compiled JAR containing its fuzzy matching functions — `jaro_winkler`, `damerau_levenshtein`, and others. These register as Java UDFs via Spark’s standard `registerJavaFunction` API. On SCOS, you provision them using `spark.addArtifact()`:</p><pre>from splink.backends.spark import similarity_jar_location<br>spark.addArtifact(similarity_jar_location())</pre><p>There’s one non-obvious requirement: the SCOS Java UDF sandbox doesn’t include the Scala standard library. Since Splink’s similarity JAR is Scala-compiled, you also need to upload the Scala 2.12 runtime:</p><pre>spark.addArtifact(&quot;/path/to/scala-library-2.12.18.jar&quot;)<br><br># Notice I did the following in my sample notebook<br><br>scala_candidates = glob.glob(<br>    os.path.join(os.path.dirname(pyspark.__file__), &quot;jars&quot;, &quot;scala-library-2.12*.jar&quot;)<br>)<br>scala_jar = scala_candidates[0]<br>spark.addArtifact(scala_jar)</pre><p>Without it, every Scala-based UDF fails with NoClassDefFoundError: scala/Predef$. Once both JARs are present, Splink’s entire UDF suite registers and runs unchanged.</p><p>From there, it’s standard Splink:</p><pre>db_api = SparkAPI(<br>  spark_session=spark,<br>  break_lineage_method=&quot;persist&quot;,<br>  register_udfs_automatically=True,<br>  num_partitions_on_repartition=8,<br>)</pre><p>Your existing NameComparison, DateOfBirthComparison, blocking rules, EM training run as is.</p><h4><strong>Approach 2: Python UDFs (The BYO-Library Blueprint)</strong></h4><p>The JAR approach works, but what if you want to use your own similarity logic? Perhaps a custom similarity function for names across languages or a domain-specific edit distance function?</p><p>You can register scalar Python UDFs backed by Python libraries of your choice. By default, Snowflake will resolve Python packages from the curated Anaconda channel. You can also use packages from PyPI or any private Python packages. I’ve included an example below for using RapidFuzz, a high-performance C++ library for string similarity, with the config that controls package resolution:</p><pre>spark.conf.set(&quot;snowpark.connect.udf.packages&quot;, &quot;[rapidfuzz]&quot;)<br><br>@udf(returnType=DoubleType(), useArrow=True)<br>def jw_rapidfuzz(s1: str, s2: str) -&gt; float:<br>  from rapidfuzz.distance import JaroWinkler<br>  if s1 is None or s2 is None:<br>    return None<br>  return JaroWinkler.similarity(s1, s2)<br> <br>spark.udf.register(&quot;jw_rapidfuzz&quot;, jw_rapidfuzz)</pre><p>Then define custom Splink comparisons that reference your UDF names in their SQL conditions. The linked repo has a detailed example of a custom Splink comparison that uses the RapidFuzz UDF we defined above.</p><p>You might be wondering if this pattern of using Python UDFs can be extended outside Splink to other PySpark workloads. Whether the UDF wraps a machine learning model, needs to do large scale time-series with PyKalman or calls a third-party API, you can use your existing Python UDFs with Snowpark Connect via Spark SQL and DataFrames.</p><h3><strong>Comparing the Two Approaches</strong></h3><p>Choosing an approach is not a binary choice. In fact, it’s possible to use both the built in similarity JAR with custom comparators you might need for your use case. I’ve included a comparison below for which option works best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mqaghZJAV8ef3W1cZGQ-5w.png" /><figcaption>Comparing Splink JARs and custom Python UDFs</figcaption></figure><h3><strong>What to Watch For</strong></h3><p>SCOS is powerful and can help you move fast, but there are a few differences worth knowing:</p><ul><li><strong>The </strong><strong>spark.jars configuration is silently ignored. </strong>To set JAR paths in Snowpark Connect, you must use spark.addArtifact() explicitly</li><li><strong>Only PySpark syntax works. Snowflake native functions do not. </strong>Splink uses sqlglot under the hood to convert the logic you’ve written to Spark SQL. Then, Snowflake examines this Spark SQL plan and converts it into a query execution plan that Snowflake can run. However, this means you will not be able to call native Snowflake functions like JAROWINKLER_SIMILARITY directly from Spark SQL, and you will need to use PySpark specific syntax. For JaroWinkler, this means using either the provided Splink JAR or a custom UDF as shown above with RapidFuzz</li><li><strong>No </strong><strong>spark.default.parallelism.</strong> Note that Snowflake virtual warehouses manage parallelism under the hood and don’t expose this Spark property. However, libraries that read it (like Splink) will crash. Make sure to set the num_partitions_on_repartition explicitly when configuring the Spark backend, so that Splink has a value to work with, even though the Snowflake warehouse controls actual execution parallelism.</li></ul><h3><strong>Bottom Line</strong></h3><p>Organizations that currently use Snowflake and Splink now have another option to run their identity resolution workloads directly on Snowflake with minimal changes thanks to Snowpark Connect. You no longer need to provision separate Spark clusters, have a separate S3 bucket, and copy data across.</p><p>This extends further to other PySpark workloads that your team relies on. The Spark ecosystem represents hundreds of libraries and years of community investment. Snowpark Connect gives that entire ecosystem a path to using Snowflake without requiring any of those libraries to be rewritten.</p><p>If you’re curious and want to learn more about leveraging Snowpark Connect to run your Spark workloads, here are a couple of articles where my colleagues go into detail.</p><ul><li><a href="https://medium.com/@umeshpatel_us/run-apache-spark-on-snowflake-the-future-of-scalable-data-workloads-dc1c3c00e955">Run Apache Spark on Snowflake: The Future of Scalable Data Workloads</a></li><li><a href="https://medium.com/snowflake/ways-to-adapt-spark-workloads-for-snowflake-spark-connector-snowpark-and-snowpark-connect-3fbc16b2b46a">Ways to Adapt Spark Workloads for Snowflake: Spark Connector, Snowpark, and Snowpark Connect Explained</a></li><li><a href="https://medium.com/snowflake/run-existing-spark-code-natively-on-snowflake-with-snowpark-connect-8cb91dc1443f">Run Existing Spark Code Natively on Snowflake with Snowpark Connect</a></li></ul><p>I’m excited for what your team can do with Snowpark Connect! Our team is focused on reducing friction for our customers every day, and we would love to hear your feedback and experience with Snowpark Connect.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a99937026d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/running-pyspark-splink-workloads-directly-on-snowflake-5a99937026d8">Running PySpark Splink Workloads Directly on Snowflake</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>