<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Empower Data Platform - Medium]]></title>
        <description><![CDATA[We are a group of data professionals dedicated to making Data and AI solutions FASTER and less challenging. - Medium]]></description>
        <link>https://medium.com/empoweranalytics?source=rss----1fb0812c90c3---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Empower Data Platform - Medium</title>
            <link>https://medium.com/empoweranalytics?source=rss----1fb0812c90c3---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:44:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/empoweranalytics" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Practitioner’s Insight: Databricks AI Suite vs Snowflake’s 3rd-party Requirements]]></title>
            <link>https://medium.com/empoweranalytics/practitioners-insight-databricks-ai-suite-vs-snowflake-s-3rd-party-requirements-8d69f8f2a7a4?source=rss----1fb0812c90c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/8d69f8f2a7a4</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Michael Cerny Green]]></dc:creator>
            <pubDate>Mon, 30 Jan 2023 20:32:29 GMT</pubDate>
            <atom:updated>2023-02-16T00:23:48.952Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was co-authored by </em><a href="https://www.linkedin.com/in/micah-groh-929794228/"><em>Micah Groh</em></a><em> (Data Scientist), </em><a href="https://www.linkedin.com/in/michael-cerny-green/"><em>Michael Green</em></a><em>, (Senior Data Scientist), and </em><a href="https://www.linkedin.com/in/fred-heller-18822a155/"><em>Fred Heller</em></a><em> (Senior Director of Data Science).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*tQwRFG3ssUAmiICx.png" /></figure><p><em>In this post, we reference a </em><a href="https://global.hitachi-solutions.com/downloads/ai-benchmark-study/"><em>data sheet</em></a><em> which details our benchmark methods and results!</em></p><p>Hitachi Solutions built the Empower Data Platform to offer customers a fully managed, subscription-based modern data estate. At its core, Empower’s analytics and artificial intelligence models are driven by Delta Lake, a transaction-based open format. Empower uses Databricks and Spark to prepare data for analytical business insights.</p><p>To optimize best practices and drive value for our customers, the Empower team continues to explore and experiment with big data processing data platforms — such as Databricks and Snowflake — to gauge their real-world effectiveness for data science. We look beyond the hype to focus on understanding each platform’s ability to augment technical teams seeking to maximize performance and minimize costs.</p><p><a href="https://www.databricks.com/">Databricks</a> (by itself) and <a href="https://www.snowflake.com/en/">Snowflake</a> (collectively with 3rd party platforms) provide data science toolkits for machine learning workflows using fundamentally different approaches. Snowflake follows the traditional approach by centralizing all data into its proprietary super-powered SQL cloud database while augmenting any gaps with additional products. Conversely, Databricks has adopted the modern paradigm of a separate data lake for storage, while natively incorporating open-source software like Spark, Kafka, and Delta Lake into a core engine that supports many programming languages.</p><p>Based on our findings, there are a limited number of configurations and ML solutions that Snowflake can support, as it is mainly a data processing and warehousing product. To facilitate ML training, Snowflake users must use 3rd party services to make up for what the platform lacks, as well as commit a significant amount of time and effort to deploy and manage those additional services. Databricks, which has been built from the ground up as a Data and AI platform, allows users to perform any use case and configuration end-to-end — from data processing and feature engineering, to building, deploying, and managing models. The Hitachi Solutions Empower team tested which platform would be easier, faster, and cheaper for both user experience and business outcomes for our customers. To do this, we designed and conducted experiments from the <a href="https://www.tpc.org/tpcx-ai/default5.asp">TPCx-AI benchmark standard</a>. This blog post is meant to be read alongside an accompanying <a href="https://global.hitachi-solutions.com/downloads/ai-benchmark-study/">data sheet</a> containing more details about our experiments’ methods and metrics.</p><p>Read on to discover what we found and how it translates to AI development advantages for our customers using the <a href="https://global.hitachi-solutions.com/technology/empower/">Empower Analytics Platform</a>.</p><h3>Our Benchmark and Takeaways</h3><p>To assess the capabilities of Snowflake and Databricks for natural language processing (NLP), computer vision (CV), and classic machine learning problems (CML), we tested each platform using four different data generators and models from the TPCx-AI benchmark.</p><p>We developed single-node and multi-node versions of these models, which required us to deploy separate compute infrastructure outside the platform, as was the case for Snowflake. We also built datasets of various sizes (1GB, 10GB, and 100GB) to assess how each platform scales.</p><p>While we built and ran our experiments, we discovered several key insights which we want to share. Here are just a few callouts:</p><p><strong>Takeaway #1:</strong></p><p><strong>Databricks is cheaper and more dependable than Snowflake for single-node workflows.</strong></p><p>We were surprised by this finding, as we expected Snowflake’s data transformation speed would result in fast and cheap model training. However, Snowflake failed on two workflows which required Conda libraries or Linux packages for NLP and CV. Fortunately, successfully completed the Price Prediction CML workflow. Broadly, we noted that Snowflake’s stored procedures have memory and time limits, causing it to time out on large datasets or run out of memory for CML workflows. Below are the results of the one workflow we were able to run on Snowflake involving price prediction. Snowflake failed to run on the 100GB dataset because of time restrictions, telegraphed to us with the following <em>SnowparkSQLException</em> message: <em>“Executing Stored Procedure progress timed out after 3600 seconds…”</em> Even on the 10GB dataset, our Snowflake pipeline triggered a stored procedure timeout exception. The bar you see is a result of an extrapolation on our part from clocking it on 1/3 of the data and multiplying it to get an estimate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glVk0TRl1O4Z8_WjPygBPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJy9oWWZKC5gjCY-7-KHpQ.png" /></figure><p><strong>Takeaway #2:</strong></p><p><strong>Databricks is faster to train AI models than Snowflake for single-node workflows</strong>.</p><p>This becomes more pronounced as the dataset sizes grow larger, as illustrated in the above graph. Databricks has the added benefit of native GPU support in both single-node and multi-node configurations, which means that scientists can take advantage of accelerated computing for even faster training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZb73-SQ-evLQQgjcDZw1g.png" /></figure><p><strong>Takeaway #3:</strong></p><p><strong>Using third-party Spark platforms to fill in for Snowflake’s missing multi-node functionality results in higher costs than just using Databricks.</strong></p><p>Snowflake has no native multi-node capability for AI. To perform multi-node workflows, you must use a secondary compute cluster. Understandably, the technical debt introduced by self-managed Spark environments materially drive-up infrastructure, deployment and management costs. If you use Snowflake along with a managed Spark service, it will be more expensive than just using Databricks on its own, which can easily handle multi-node workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AhLeMtDLhU1JVr5Qq1fKiQ.png" /><figcaption>Figure 1: Above you can see the costs of Databricks vs Snowflake for all four workflows. Whereas Databricks contains a single cost metric (Azure spend + DBUs), Snowflake contains the costs of Snowflake warehouses, a Spark cluster on Azure, and licensing costs for managed Spark</figcaption></figure><p><strong>Takeaway #4:</strong></p><p><strong>Databricks is simpler to deploy and maintain than Snowflake for multi-node workflows.</strong></p><p>With only one compute cluster to worry about, Databricks users have no trouble scaling up their models and datasets for demanding workflows. MLOps tools are natively integrated into the Databricks environment, and scaling is not challenging. Snowflake requires the user to use external compute to scale, requiring external compute integration and introducing more complexity.</p><p><strong>Takeaway #5:</strong></p><p><strong>Databricks is easier to use, especially when considering MLOps and workspaces.</strong></p><p>Databricks enhances a data science workflow rather than inhibits it. The workspace is set up to mimic the look and feel of Jupyter notebooks, which will be intuitive to data scientists. The Databricks ecosystem of natively supported tools like <a href="https://mlflow.org/">MLFlow</a> for MLOps makes training, maintaining, and deploying models easy.</p><h3>Artificial Intelligence Strategies</h3><p>Snowflake and Databricks recommend different approaches to implement AI workflows in terms of compute and solution space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W27fboJgreVwAyI5pmKjpg.png" /></figure><p>Snowflake’s recommended strategy is to use single-node compute for AI/ML scenarios. Understandably, however, this fails for tabular data over 1M rows using any model due to memory and time constraints. Snowflake’s compute environment cannot handle the TPCx-AI’s NLP or computer vision benchmarks because of library requirements since Snowflake limits what a user can install within a warehouse. To perform workloads requiring these kinds of models or multi-node workloads, Snowflake users must utilize an external compute environment. This also means that data will flow outside the Snowflake warehouse during training and possibly across cloud vendors and geographical regions, which can be slower, complex, expensive and introduce governance risks.</p><p>In contrast, Databricks is multi-node by default and easily scales to single-node. The user fully controls library installation, instance type, and cluster size. In general, Databricks succeeds in all use cases. All data flow, data transformation, model training, and model serving occur within the Databricks’ compute environment, and data does not need to leave the tenant. This is a significant advantage.</p><h3>Training &amp; Serving</h3><p>Let’s dig into how to build an AI training and inference pipeline to contrast each vendor’s approach.</p><h3>Snowflake</h3><p>Snowflake users can train models using one of the following two methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMsZs4NL6td4IFOYyrNeFg.png" /><figcaption>Snowflake ML Training Pipeline Options</figcaption></figure><p><strong>Snowflake Compute</strong></p><p>Models can be trained using Snowflake compute using UDFs or stored procedures. In both cases, <a href="https://www.snowflake.com/snowpark/">Snowpark</a> can be used to build the training function in the user’s development environment and send the function to Snowflake to execute. The data and the model are contained/trained entirely within the Snowflake execution environment. Still, this method is limited to a single node based on how SQL manages UDFs and stored procedures. UDFs are recommended only for models and datasets that would take less than a minute to train.</p><p><strong>Third-Party Compute</strong></p><p>Snowflake does not support distributed, multi-node training within its compute warehouse. Users seeking to scale their machine learning workflows with more data or larger models will need to use a separate compute cluster, the industry-standard being Spark. Models can be trained in a Spark cluster, which uses Snowpark or the Snowflake Spark connector to query Snowflake for training data, which is then sent back to the Spark Cluster from the Snowflake data store. We commonly see customers deploying self-managed Spark clusters to reduce costs without realizing the complex infrastructure and sophisticated expertise required to succeed. Subsequently, we highly recommend using a third-party platform to manage the Spark clusters, such as Sagemaker, Dataiku, Databricks, etc.</p><p><em>In single or multi-node cases, MLOps services must be found externally (Dataiku, Datarobot, etc.) as Snowflake does not natively support MLOps functionality like model registries or experiment tracking.</em></p><p>Like training, model serving in Snowflake can be done with two different methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmqQHhXP0qtpOuDHgpMz3w.png" /><figcaption>Snowflake ML Serving Options</figcaption></figure><p><strong>In-Platform Serving</strong></p><p>Within Snowflake, models can be loaded from either the Snowflake staging area or an external model registry. Inference can be executed on a table using a <strong>SQL UDF</strong> to evaluate the model one row at a time. Snowpark UDFs also support batch inference of the model for efficiency on large datasets.</p><p><strong>Model API</strong></p><p>Models can also be hosted in inference clusters outside of the Snowflake environment using partner platforms. Snowflake UDFs can be written in Snowpark to call the trained model’s API and send it data to infer upon. The partner platform will return the model’s results for that data, which the UDF can write back to the data warehouse.</p><h3>Databricks</h3><p>By default, Databricks can operate in multi-node format since the underlying compute infrastructure is <a href="https://spark.apache.org/">Apache Spark</a>. Users can develop using <strong>Spark dataframes and Delta Tables with</strong> <strong>SparkML or </strong><a href="https://horovod.ai/"><strong>Horovod</strong></a><strong> </strong>versions of learning libraries (Tensorflow, Keras, Pytorch, Scikitlearn, XGBoost).</p><p>Databricks can easily shift to single-node workflows to leverage Pandas dataframes, NumPy arrays, or Spark dataframes with usual versions of learning libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*huYbsGqoPiy1kwIvloHk5Q.png" /><figcaption>Databricks ML Training Options</figcaption></figure><p><em>In either case, single or multi-node, MLOps services exist in the natively integrated MLFlow, which supports model registry and feature stores, experiment tracking, and model serving features.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*rZHtpTrNvFyOZK2ExtsoGA.png" /><figcaption>Databricks ML Serving Options</figcaption></figure><p>Databricks manages model serving in-house using the natively integrated MLflow. By using MLflow to store and host models, users can host trained models with several supported configurations:</p><ul><li><strong>Batch Inference</strong>: Models can infer on a table using the model’s API (Spark, Tensorflow, PyTorch, etc). Integration with MLflow supports loading any flavor of ML model into a <strong>SQL UDF </strong>for inference.</li><li><strong>Server Endpoint</strong>: The Databricks model registry can create a server REST endpoint with the cluster that can scale to meet demand.</li><li><strong>Serverless Endpoint</strong>: Databricks can create a Databricks-managed serverless REST endpoint.</li></ul><p><strong>Conclusion</strong></p><p>At Hitachi Solutions, we build innovative products to generate value for our customers. With Empower, it is imperative for us to use technology stacks that facilitate ML development, training, and serving for our models.</p><p>Based on our findings, Databricks is, on average, faster, cheaper, and easier to use for developing machine learning models. Snowflake’s reliance on third-party resources for distributed training is a major drawback. The need to combine disparate products to scale up training brings complexity and, in our view, is an unnecessary complication to drive business value.</p><p>Databricks’ framework of a Spark platform built on top of a data Lakehouse is a proven formula that is being adopted across the industry. That’s why team Empower uses Databricks for our data science needs. If you’d like to read more about our benchmarking, check out our <a href="https://global.hitachi-solutions.com/downloads/ai-benchmark-study/">datasheet</a>. <a href="https://global.hitachi-solutions.com/contact-us/">Contact us</a> to learn more about Hitachi Solutions, and how our experts can accelerate your modern data estate initiatives today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d69f8f2a7a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/empoweranalytics/practitioners-insight-databricks-ai-suite-vs-snowflake-s-3rd-party-requirements-8d69f8f2a7a4">Practitioner’s Insight: Databricks AI Suite vs Snowflake’s 3rd-party Requirements</a> was originally published in <a href="https://medium.com/empoweranalytics">Empower Data Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practitioner’s Experience: A Comparison of Training ML Models using SnowPark vs Databricks]]></title>
            <link>https://medium.com/empoweranalytics/practitioners-experience-a-comparison-of-training-ml-models-using-snowpark-vs-databricks-f29d3f79fb6d?source=rss----1fb0812c90c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/f29d3f79fb6d</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Michael Cerny Green]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 14:31:54 GMT</pubDate>
            <atom:updated>2022-11-01T14:31:54.149Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was co-authored by </em><a href="https://www.linkedin.com/in/micah-groh-929794228/"><em>Micah Groh</em></a><em>, Data Scientist, and </em><a href="https://www.linkedin.com/in/michael-cerny-green/"><em>Michael Green</em></a><em>, Senior Data Scientist.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NoZPsFbxtjoFOHfu6-BrpQ.png" /></figure><p>Artificial intelligence. Deep learning. Big data. Google any of these phrases (or, gulp, all three at once), and you’ll immediately be inundated with results claiming to give you the know-how, the capability, or both, to analyze and train on your data using their “all-in-one” platform. We won’t claim to know everything, but with our experience building the <a href="https://global.hitachi-solutions.com/technology/empower/">Empower Data Platform</a>, we can point to many examples where we have helped our customers wrangle their bytes to produce results. Our blog posts aim to guide data scientists and engineers in making an informed choice.</p><p><a href="https://spark.apache.org/">Apache Spark</a> is widely accepted as the open-source gold standard for data engineering and artificial intelligence research and development in the business world [<a href="https://www.databricks.com/blog/2022/06/15/apache-spark-and-photon-receive-sigmod-awards.html">1</a>, <a href="https://medium.com/plumbersofdatascience/why-you-should-get-into-apache-spark-and-kafka-f16b4e4fd193">2</a>, <a href="https://blog.devgenius.io/modern-data-engineering-building-a-data-lakehouse-with-apache-spark-vol-1-853f0882862b">3</a>]. We’ve recently seen articles about moving away from Spark to a different data platform like <a href="https://www.snowflake.com/en/"><strong>Snowflake</strong></a> [<a href="https://medium.com/snowflake/migrating-from-pyspark-to-snowpark-python-series-part-1-a75058c1e579">1</a>, <a href="https://blog.devgenius.io/snowpark-vs-snowflake-connector-5c6baa44bde2">2</a>]. Since <a href="https://www.snowflake.com/blog/snowpark-is-now-generally-available/">the release of Snowpark at the beginning of the year</a>, which was built to augment AI/ML workflows, we haven’t yet found a thorough assessment of Snowflake’s AI/ML functionality. To remedy this, we’ve decided to provide the critical eye<strong> </strong>with a side-by-side comparison scoped to training artificial intelligence models between Databricks (developers of Apache Spark’s latest evolution) and Snowflake. In this post, we want to reason whether Snowflake with Snowpark is a strong contender regarding artificial intelligence workflows.</p><p><strong>TLDR;</strong> <strong>it is not. </strong><em>Keep reading for our breakdown.</em></p><h3>Workspace Comparison</h3><h4>Databricks</h4><p>Databricks uses managed Spark clusters built on top of a <a href="https://www.databricks.com/glossary/data-lakehouse#:~:text=What%20is%20a%20Data%20Lakehouse,(ML)%20on%20all%20data.">data Lakehouse</a>. This architecture allows for common data warehousing, data analytics, data science, data engineering, and machine learning to be unified within a single platform. The primary developer interface is the all-too-familiar notebook, which has native Git integration and can be edited by multiple collaborators simultaneously. In addition to Python, notebooks allow for executing markdown, R, SQL, Scala, and Shell, all with integrated documentation. The workspace supports the complete PyPI ecosystem of Python libraries, and Linux packages can be installed using customized <a href="https://docs.databricks.com/clusters/init-scripts.html">initialization scripts</a>. It’s worth mentioning that users can also develop within their favorite IDE or notebook server and use <a href="https://docs.databricks.com/dev-tools/databricks-connect.html">Databricks Connect</a> to schedule Spark jobs in the Databricks cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6o71DHxa28zwLUNQapTnCw.gif" /><figcaption>The Databricks workspace UI is a collaborative Jupyter notebook, with support for Python, SQL, Shell, and Scala</figcaption></figure><h4>Snowflake/Snowpark</h4><p>Snowflake is a data warehouse whose primary interface is a SQL-only worksheet which can execute commands on SQL tables. They do not have native Git integration, and multiple users cannot edit a worksheet collaboratively like in Databricks. The Snowpark API allows users to define data transformations within their local environment to be used on tables stored in Snowflake. <em>This means that data manipulated with Snowpark remains within Snowflake.</em> Snowpark also allows custom-made functions to be uploaded to the Snowflake execution environment and then called to run within a Snowflake warehouse. We explored the Snowpark Python library, but Snowpark also supports SQL, Scala, and Java. Snowflake clusters support several third-party Python packages provided and built by Anaconda, but the list is limited, and not all library versions are supported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*R7LP1gyafPZzNNVseupEqw.png" /><figcaption>The Snowflake workspace UI lets you write code only in SQL</figcaption></figure><h4>Winner: Databricks</h4><p>Snowpark gets points for letting you develop from the comfort of your own development environment/language. At the same time, so does Databricks using Databricks Connect. Databricks’ collaborative notebooks are a huge plus, as is native Git integration. The variety of languages supported by Databricks’ notebooks translates to extreme convenience for different developers and use cases, and debugging is a breeze. Furthermore, Databricks has better package support.</p><h3>Hardware Management</h3><h4>Databricks</h4><p>Databricks clusters can be built to handle both CPU and GPU workloads. In addition to the standard Databricks UI, there are also <a href="https://docs.databricks.com/dev-tools/api/latest/index.html">Databricks API</a> and <a href="https://docs.databricks.com/dev-tools/index-cli.html">CLI</a> tools that can manipulate Databricks resources and modules within a workspace from another machine. You can select type and node count for your clusters within Azure. Admins can setup templated cluster policies to reduce option overload for regular users. Clusters can be specialized for generalized or ML workloads. Jobs clusters can be used to optimize costs, as can spot instances.</p><h4>Snowflake/Snowpark</h4><p>Snowflake, on the other hand, only supports CPU clusters. You can <a href="https://docs.snowflake.com/en/user-guide/warehouses-overview.html#warehouse-size">select t-shirt-defined cluster sizes</a>, but beyond that, there are not many choices. <a href="https://docs.snowflake.com/en/user-guide/snowsql.html">SnowSQL</a>, the Snowflake CLI client, allows you to make API calls to your cluster to move data, query, and manipulate warehouses remotely. In Snowflake, you can set up scheduled tasks to execute (the equivalent to job clusters in Databricks).</p><h4>Winner: Databricks</h4><p>Choice paralysis is a real thing, but not when you need to optimize a cluster for machine learning training. Each company has a different philosophy about configuration: Snowflake removes many choices for you, while Databricks offers you many. Depending on your needs, you may prefer one over the other. However, Databricks supports GPUs within their environment, which is critical for big-model AI training, especially in the computer vision and natural language processing spaces.</p><h3>Single-Node Training</h3><h4>Databricks</h4><p>Single-node training in Databricks feels no different than training on a Jupyter notebook in your local environment. You can develop a model and run it inside the notebook using any popular ML and DL frameworks, such as Scikit-Learn, PyTorch, or Tensorflow. You can also launch your notebook as a job, and Databricks will spin up and down resources to handle the training. All data within the Databricks workspace is available to your notebook. Note that the default use case of Spark is to do multi-node computation. Databricks users looking to do distributed large scale training or single node training are equally at home. We discuss distributed training in the next comparison section.</p><h4>Snowflake/Snowpark</h4><p>Snowflake does single-node training a bit differently. There are two options that we are aware of to do this <strong>within</strong> Snowflake clusters: user-defined functions and stored procedures.</p><p><strong><em>User Defined Functions (UDFs)</em></strong></p><p>UDFs are valuable for defining complex transformations on Dataframes. Snowpark supports scalar UDFs[<a href="https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs.html">1</a>, <a href="https://database.guide/what-is-a-scalar-udf-in-sql-server/">2</a>], which can take a row as input and output a new single value. UDFs are not a standard technique for training models, but <a href="https://github.com/Snowflake-Labs/sfguide-citibike-ml-snowpark-python/blob/main/03_ML_Engineering.ipynb">Snowflake provides an example</a> of how to use a UDF to train many models in parallel for time series forecasting. In this example, they flatten training and inference data into lists before passing them to a UDF with Snowpark. Once within the UDF, the data is unflattened back into Dataframes and passed to the model’s training loop. Once trained, the model can then infer on the inference data, and the UDF can insert predictions into the desired table.</p><p>If the above paragraph seems convoluted to you, you are not alone. It took us quite a few tries to understand and build this ourselves. The process of flattening Dataframes and reverting them back is susceptible to human error and is not intuitive. Debugging can be challenging because the code is executed within the Snowflake execution environment, which, being meant for SQL, isn’t always helpful when outputting Python errors. Furthermore, UDFs are limited to a one-minute execution time, restricting this method to the smallest of models and/or datasets. UDFs cannot return objects, so the trained model must infer within the UDF before it is thrown away.</p><p><strong><em>Stored Procedures</em></strong></p><p>Stored procedures allow for the execution of a series of commands to be reused. They enable the execution of arbitrary Python code, making them better suited for training models for a wider variety of use cases than UDFs. Snowflake tables can be loaded from the Snowflake data warehouse into Snowpark, and transformations on these tables using Snowpark will be pushed down to SQL statements to execute on the tables within the Snowflake warehouse. Tables can also be converted to Pandas Dataframes and operated with usual Pandas Dataframe functions. As with UDFs, stored procedures separate the development and execution environment, making developing or debugging code challenging. In addition, stored procedures only support a scalar return value, meaning that any unsupported data type will be returned as a JSON serialized string, even Python lists (since SQL cannot “Pickle” objects). Stored procedures are also limited to a one-hour execution time. This is more time than UDFs, but still very limiting when training on a large quantity of data.</p><h4>Winner: Databricks</h4><p>Look how much shorter the Databricks section is. We are not kidding: it is much easier to do single-node training. During our exploratory analysis, we realized that <a href="https://repo.anaconda.com/pkgs/snowflake/">Snowflake does not support <em>OpenCV</em></a>, a popular computer vision image manipulation library, on their Linux clusters. Nor does Snowflake support Linux packages such as <em>libsndfile </em>for audio file manipulation. Lastly, Snowflake does not support reinforcement learning libraries like KerasRL, OpenAI Baselines, and Stable Baselines. This removes quite a few branches of artificial intelligence from your toolkit. Being limited to single-node training means you will also be limited to the hardware specs of a single instance. If your model requires more memory, compute power, or accelerated computing, your only option is to use a more powerful machine. Keep in mind that UDFs and Stored Procedures also have warehouse-defined memory caps. If you hit this cap, you’ll need to contact Snowflake support to raise that limit for you on a warehouse-by-warehouse basis.</p><h3>Multi-Node/Distributed training</h3><h4>Databricks</h4><p>The Databricks’ Spark ML library natively supports multi-node implementations of many popular machine learning algorithms, including linear/logistic regression, random forests, and gradient-boosted decision trees. Our experience converting single-node deep learning pipelines using Tensorflow into multi-node CPU or GPU was as simple as utilizing Spark’s <a href="https://horovod.readthedocs.io/en/stable/spark_include.html">Horovod</a> implementation. This required us to change less than ten lines of code in our notebook. This is also possible using <a href="https://docs.databricks.com/machine-learning/train-model/pytorch.html">Pytorch</a>, which also Databricks supports.</p><p>Databrick’s UI contains a node-utilization interface, allowing you to monitor resource usage during training. You can set up multi-node training on job clusters, optimized solely for this kind of task if you want to be mindful of compute costs. These job clusters are managed by Databricks on your cloud compute, and Databricks will spin these clusters up and down for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Sk74KReQuUsk9EHA5jYTOA.png" /><figcaption>Going from single to multiple node distributed training for an XGBoost classifier is as simple as increasing a single parameter: num_workers.</figcaption></figure><h4>Snowflake/Snowpark</h4><p>On its face, multi-node/distributed training is impossible within the Snowflake execution environment due to how the environment is set up and the capabilities afforded to the user. It is important to mention that multi-node, in the sense of training multiple independent models at the same time in the same cluster, is entirely possible using stored procedures and UDFs, <a href="https://medium.com/snowflake/distributed-ml-with-snowpark-python-udfs-fd60135c11bd">as this blog post claims</a>. But this is not to be confused with true <strong>distributed training </strong>(such as <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">DDP</a> for Pytorch/Pytorch Lightning or <a href="https://horovod.ai/">Horovod</a> for TensorFlow/Keras/Pytorch), <a href="https://learn.microsoft.com/en-us/azure/machine-learning/concept-distributed-training">defined by Microsoft</a> as a workload to train a single model being split across multiple processors, which work in parallel to speed up the model’s training procedure. Our preference is that the term “distributed ML” should be utilized for such distributed training processes only. There are two main types of distributed training: <a href="https://learn.microsoft.com/en-us/azure/machine-learning/concept-distributed-training#data-parallelism">data parallelism</a> and <a href="https://learn.microsoft.com/en-us/azure/machine-learning/concept-distributed-training#model-parallelism">model parallelism</a>, neither of which Snowflake supports natively.</p><p>To do multi-node distributed training with Snowflake, you must use a separate cluster, such as an open-source Spark cluster or a provider like Azure Synapse or AWS EMR. Models can train distributed within this 3rd party cluster, while communicating with Snowflake as a data store (preferably using the Snowpark API) to get training data as needed.</p><h4>Winner: Databricks, hands down</h4><p>Snowflake must use third-party partners to even do multi-node training. Snowpark acts as a bridge to allow Snowflake to be your data store, but then are you truly doing AI <strong>with</strong> Snowflake or with a Snowflake partner? Meanwhile, Databricks makes this incredibly easy for you.</p><h3>Conclusion?</h3><p>Databricks provides a unified workspace for collaborative model development, with tremendous support for different languages and libraries. Snowflake provides worksheets for SQL analysts or Snowpark in your local environment for data scientists. In terms of single-node functionality, Databricks is straightforward, whereas Snowflake achieves this with user-defined functions (UDFs) and stored procedures. Databricks natively offers multi-node training for machine learning and deep learning workloads with minor modification from any developed Spark application. Multi-node training using Snowflake only as a data source is possible, but there is no native method to perform multi-node training within a Snowflake warehouse.</p><p>Above, we’ve scoped our comparison to just AI training workflows. Snowflake has potential, and we appreciate its engineering quality in terms of its data warehousing solutions, where it shines spectacularly. If you plan to move into AI development in the near future, it makes sense to start with Databricks and Spark. And we did not even break into the model registry, feature store, time travel, experiment tracking, low/no-code solutions like AutoML, or other vital parts of machine learning pipelines, none of which Snowflake supports natively. Our next blog post will focus more on these areas. To put it succinctly, <strong>Databricks beats Snowflake/Snowpark for AI training.</strong></p><p>At <a href="https://global.hitachi-solutions.com/technology/empower/">Empower</a>, we’ve realized that data science, machine learning, and streaming are far more manageable and governed with Databricks Lakehouse and the open Delta Lake format than alternate solutions.</p><p>Have questions about how we can <a href="https://global.hitachi-solutions.com/technology/empower/">accelerate your modern data estate initiatives</a>? Connect with our experts!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f29d3f79fb6d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/empoweranalytics/practitioners-experience-a-comparison-of-training-ml-models-using-snowpark-vs-databricks-f29d3f79fb6d">Practitioner’s Experience: A Comparison of Training ML Models using SnowPark vs Databricks</a> was originally published in <a href="https://medium.com/empoweranalytics">Empower Data Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practitioner’s Perspective: SnowPARK for Data Science & Machine Learning leaves a lot to be desired.]]></title>
            <link>https://medium.com/empoweranalytics/practitioners-perspective-snowpark-f4fad4ab8995?source=rss----1fb0812c90c3---4</link>
            <guid isPermaLink="false">https://medium.com/p/f4fad4ab8995</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[snowpark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Michael Cerny Green]]></dc:creator>
            <pubDate>Tue, 18 Oct 2022 01:46:31 GMT</pubDate>
            <atom:updated>2022-10-18T14:04:58.533Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was co-authored by </em><a href="https://www.linkedin.com/in/michael-cerny-green/"><em>Michael Green</em></a><em>, Senior Data Scientist, and </em><a href="https://www.linkedin.com/in/mike-kirkpatrick-4a6a8885/"><em>Mike Kirkpatrick</em></a><em>, Principal Data Scientist.</em></p><figure><img alt="Practitioner’s Perspective: SnowPARK for Data Science &amp; Machine Learning leaves a lot to be desired" src="https://cdn-images-1.medium.com/max/968/1*_xj5vF7r-jribzqKXLJ-2w.png" /></figure><p>With our work at <a href="https://global.hitachi-solutions.com/technology/empower/">Empower</a>, we help customers unlock their potential across their digital transformation initiatives and have tremendous industry exposure across platforms to make that possible. In today’s world, regardless of industry, the common denominator is data — and how <em>best</em> to make use of it.</p><p>Data teams have familiar gripes with data in the wild: it’s messy, it is fragmented, and often incompatible. One of the principal challenges with building successful models is data preparation — it’s usually a majority of the engineering cost of any advanced analytics project. Any fellow A.I. researcher, data scientist, analyst, or data engineer can attest to this fact. Hence the question becomes — “how can my enterprise not just park my data, but transform it, create value from the data, and ultimately act on the insights the platform can generate?”</p><p>When Databricks outlined the vision of Lakehouse in a <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html">blog</a> in January 2020, followed by several <a href="https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf">technical</a> research <a href="https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf">papers</a> that explained how the world of data science, machine learning, and data warehousing can be unified in a single platform: the world took notice. Across the board companies and product have been working to bring these concepts together [<a href="https://trends.google.com/trends/explore?date=today%205-y&amp;geo=US&amp;q=data%20lakehouse">1</a>,<a href="https://cloud.google.com/blog/products/data-analytics/open-data-lakehouse-on-google-cloud">2</a>,<a href="https://aws.amazon.com/blogs/big-data/build-a-lake-house-architecture-on-aws/">3</a>,<a href="https://www.prnewswire.com/news-releases/starburst-announces-new-product-release-which-extends-flexibility-when-building-data-lakehouse-architecture-301425279.html">4</a>].</p><p>Snowflake has successfully made SaaS warehousing approachable with cloud-based computing–making scaling easy for enterprise customers. While Snowflake disrupts the traditional enterprise data warehousing market with a quality product, SnowPark fails to execute when it comes to building and training powerful machine learning, natural language processing, and deep learning models.</p><p>Customers need the elasticity that Snowflake provides (in its data warehouse), while also allowing simplified data access to act on that data — e<em>.g., avoiding creating just another data technology silo.</em> Efforts by Snowflake to bolt on Spark-inspired capabilities are prone to failure with traditional warehousing solutions. In this article, we share where organizations will struggle with achieving value from Snowflake for advanced analytics, and why we cannot recommend Snowpark for AI/ML.</p><h3>SnowPARK: A 10,000 ft view</h3><p>Snowpark is a library available in Scala and Python that implements an API that’s very similar to Apache Spark. Perhaps this is why the name reads like SPARK, i.e., SnowPARK. The core of that API is the Dataframe API available in Spark, which is a <strong>very</strong> popular data science API that lets you do everything from basic analytics to advanced analytics, such as ML. Using Spark looks and feels very similar to the popular R Dataframe API and the popular Python library known as Pandas.</p><p>Snowpark implements this primarily by producing SQL that it then emits to Snowflake’s Data Warehouse, which then executes this SQL with predicate pushdown. The fact that this works is no surprise, as it’s <a href="https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf">the foundation of Lakehouse architecture</a> that Dataframe’s and SQL can be expressed in terms of each other.</p><p>Data scientists can use Snowpark to write their own code with the full power of Python, and when they need to do analytics on large amounts of data, they call the Snowpark API. This API looks like Spark but translates instructions to SQL under the hood and returns results. With few exceptions, Snowpark does not unify data engineering and data science, it keeps them separated by design.</p><p>We find it essential to note that while Snowpark and its related solutions encompass model production pipelines, none of them assist with model <em>development</em>, a critical piece of a data scientist’s workflow.</p><h3>The problems with Snowpark</h3><p>Our caution with Snowpark fall into two buckets:</p><ol><li>Lack of the straightforward, scalable data science capabilities people use Spark for, and</li><li>Incompatibility with the collaborative and open ecosystem that the Spark community has built</li></ol><p>Let’s unpack both.</p><h3>Lack of Critical Capabilities</h3><p><strong>First, the essential ML capabilities of Spark are not currently supported by Snowpark natively.</strong> Unfortunately, it’s unlikely these can be truly supported without reimplementing much of Spark in Snowflake.</p><ul><li><strong>Structured Streaming</strong>. One of the most popular frameworks in Spark is the streaming engine. With Spark’s streaming approach (and now <a href="https://databricks.com/product/delta-live-tables">Delta Live Tables</a>), Spark seamlessly unifies batch and streaming. This design enables robust streaming use cases that organizations commonly deliver: detecting fraudulent transactions in real-time, performing risk forecasting using vehicle telemetry, or reacting to energy pricing across a region. Unfortunately, Structured Streaming is currently not supported by Snowpark.</li><li><strong>MLFlow and AutoML</strong>. Similar to streaming, the key to making ML more accessible to organizations is to make it easier to use. A big part of that process is enabling data scientists to record and track the results of experiments quickly and easily. Then once your experimentation has produced the best model, you can effortlessly deploy it to a model registry for model serving, lineage tracking, and versioning. ML in production requires governance: tracking to historical performance, transparency, and retraining. For years vendors have been clamoring to provide a complete ecosystem of tools — but only Databricks has managed to gather broad support with the industry-leading, open-source framework <a href="https://mlflow.org/">MLFl</a>ow. With MLFlow, Delta Live Tables, and <a href="https://www.databricks.com/de/product/automl">AutoML</a>, customers can ingest data, transform it, enhance it with ML, and present it to users — all in near real-time. This gets even more attractive with tools like <a href="https://databricks.com/blog/2022/06/29/delta-live-tables-announces-new-capabilities-and-performance-optimizations.html">Project Enzyme</a>, and <a href="https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/machine-learning-at-scale-with-databricks-and-kubernetes/ba-p/3056539">integrations to Kubernetes </a>(and other publishing options). It’s this holistic ecosystem of tools that makes a true Lakehouse incredibly valuable.</li><li><strong>SparkML/MLlib</strong>. Today, Spark is widely used for machine learning. Spark includes outstanding machine learning libraries for recommendation, classification, regression, clustering, and anomaly detection. It’s our position that no other library currently supports such a wide variety of machine learning algorithms available out-of-the-box, as Spark has been continuously improved over the last decade by a highly collaborative community. Implementing and optimizing all these algorithms inside a data warehouse is an enormous and unnecessary task, akin to “reinventing the wheel” of ML libraries. A key difference between the two, Snowpark users build their own integrations with Spark, whereas Databricks users have access to Spark at the starting line.</li><li><strong>Koalas and Pandas</strong>. One of the most powerful features the Spark and Databricks teams brought to customers was adding Koalas to Spark. For the uninitiated, SQL and Python dominate data engineering — and a commonly used package for data manipulation in Python is Pandas. What Koalas allows is a drop-in replacement–i.e., no code changes–for making panda’s scale horizontally on Spark (read: way faster execution times). This feature means that Python users familiar with Pandas do not need to learn new tools to go fast. Unlike Snowpark, Koalas is not a translation layer between Koalas and Pandas but rather a re-implementation of the entire Pandas API for maximal performance. This project was a considerable community effort that took many years and only recently matured to the level that the Spark project incorporated it wholesale into Spark. Unfortunately, Koalas and Pandas are not available in Snowpark.</li><li><strong>Engineering and Machine Learning Pipeline Optimization</strong>. Part of what we believe makes Spark so powerful is that regardless of the language (SQL, Python, Scala, or R) the system translates commands to Dataframe operations which are optimized holistically. This workload aware optimization translates to significant customer cost savings for our customers during pipeline execution. This optimization is impossible to do with Snowpark which executes independent SQL operations against the Data Warehouse, with minimal Python execution against the database — something we will explore in a future post.</li></ul><h3>Why Break from a Winning Ecosystem?</h3><p><strong>Our second concern is with Snowpark’s divergence from established tooling</strong>. With thousands of developers contributing to building Spark over the past 13 years, and businesses increasingly struggling to staff data programs, most companies cannot afford to adopt unknown niche tools — especially for nascent data science programs.</p><ul><li><strong>Connectors, integrations, formats</strong>. Spark supports thousands of connectors to different systems, and dozens of data formats can leverage Spark. Sophisticated tooling like <a href="https://global.hitachi-solutions.com/technology/empower/">Empower Analytics</a> lets customers onboard entire enterprises into a Lakehouse with nothing more than a connection string by utilizing Spark’s powerful integration capabilities. The ability to get data in and out from virtually every type of system provides the extreme flexibility our customers require (even <a href="https://docs.databricks.com/sql/api/queries-dashboards.html">REST APIs</a>).</li><li><strong>Training and Qualified Talent</strong>. Over the past decade, many books have been written about Spark [<a href="https://www.amazon.com/Modern-Engineering-Apache-Spark-Hands/dp/1484274512">1</a>, <a href="https://www.amazon.com/Engineering-Apache-Spark-Delta-Lakehouse/dp/1801077746">2</a>, <a href="https://www.amazon.com/Spark-Action-Jean-Georges-Perrin/dp/1617295523">3</a>, <a href="https://www.amazon.com/Spark-Definitive-Guide-Processing-Simple/dp/1491912219">4</a>, <a href="https://www.amazon.com/High-Performance-Spark-Practices-Optimizing/dp/1491943203">5</a>, <a href="https://www.amazon.com/Stream-Processing-Apache-Spark-Optimizing/dp/1491944242">6</a>, <a href="https://www.amazon.com/Practical-Machine-Learning-Spark-High-Quality/dp/9391392083">7</a>, <a href="https://www.amazon.com/PySpark-Cookbook-implementing-processing-analytics/dp/1788835360">8</a>]. Countless classes, lectures, and <a href="https://www.mooc-list.com/tags/apache-spark">Massive Online Open Courses</a> have been created. With such a huge pool of students and engineers trained and learning how to use Spark, companies leverage this talent to achieve their data goals. This talent availability is at risk when choosing a Spark-inspired API that is <em>not</em> Spark.</li><li><strong>Existing codebases</strong>. Over the years, people and organizations have contributed millions of lines of Spark code. All this code must be rewritten to use the Snowpark API, a costly undertaking.</li></ul><h3>Summary</h3><p>Above, we’ve outlined how Snowpark works: it is a library inspired by Spark, but it doesn’t deliver the most popular ML capabilities that Spark provides and lacks Spark’s ecosystem of enabling tools. The very reasons people pick Spark are not currently present in Snowpark.</p><p>From the popular streaming engine, the vast machine learning libraries, the ubiquitous Pandas/Koalas tooling, or one of the other many tools we referenced in this article, Spark enables enterprise customers to meet their demands. Engineers need a rich ecosystem of connectors and educational resources, which Spark provides. To mimic these capabilities, organizations would have to convince their valuable data engineers and data scientists to convert Spark applications (at a significant cost) to be compatible with Snowpark without getting any conceivable benefit. For this reason, we think Snowpark poses a risk to attracting and retaining top talent.</p><p>At Empower, we’ve invested significantly in helping customers utilize the Lakehouse platform to give end-users richer information faster and at less cost. The value of data lakes for customers is to provide compute options on enterprise data to enable actionable results. SQL warehousing, data science, machine learning, and streaming are far more manageable and governed with Databricks Lakehouse and the open Delta Lake format than compared to alternate solutions, driving vital benefits.</p><p>Have questions? Interested in how we accelerate modern data estate initiatives with our day jobs? Please reach out!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4fad4ab8995" width="1" height="1" alt=""><hr><p><a href="https://medium.com/empoweranalytics/practitioners-perspective-snowpark-f4fad4ab8995">Practitioner’s Perspective: SnowPARK for Data Science &amp; Machine Learning leaves a lot to be desired.</a> was originally published in <a href="https://medium.com/empoweranalytics">Empower Data Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>