<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Sync Computing - Medium]]></title>
        <description><![CDATA[We are an MIT based startup aimed at bridging the gap between cloud infrastructure and business goals. Our product, Gradient, helps companies scale Databricks intelligently. Try free now: https://www.synccomputing.com - Medium]]></description>
        <link>https://medium.com/sync-computing?source=rss----f1db5e70d515---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Sync Computing - Medium</title>
            <link>https://medium.com/sync-computing?source=rss----f1db5e70d515---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:10:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/sync-computing" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From chaos to clarity: How Gradient Insights saves your budget (and sanity)]]></title>
            <link>https://medium.com/sync-computing/from-chaos-to-clarity-how-gradient-insights-saves-your-budget-and-sanity-179707323c8b?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/179707323c8b</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 17:06:11 GMT</pubDate>
            <atom:updated>2025-03-12T17:06:11.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tpf1tcx-M6mTJZ3e.png" /></figure><p>In the world of data engineering, managing Databricks jobs can feel like herding cats while juggling flaming torches. You’re responsible for hundreds or even <a href="https://synccomputing.com/scale-compute/">thousands of jobs</a>, each with its own quirks and performance characteristics. And let’s be honest, most of us don’t have the time, expertise, or frankly, the patience to sift through mountains of raw metrics to figure out what’s really going on.</p><p>That’s why we’re excited to introduce Gradient Insights — a feature designed to transform complex job metrics into actionable intelligence that even your VP can understand.</p><h3>The data engineering reality check</h3><p>Before diving into what Insights can do for you, let’s acknowledge some painful truths about managing Databricks jobs:</p><ul><li><strong>Cluster management is fiendishly complex. </strong>Even experienced data engineers can struggle to interpret performance metrics and make optimal configuration decisions.</li><li><strong>Context gets lost. </strong>The engineer who wrote that critical ETL job three years ago? They’re now working at a startup in Portugal, and they took all their tribal knowledge with them.</li><li><strong>Scale creates blindness.</strong> When you’re responsible for hundreds of jobs, you’re lucky if you know what’s happening with a handful of them, and usually only when something breaks catastrophically.</li><li><strong>Raw metrics aren’t user-friendly.</strong> Gradient has always provided <a href="https://synccomputing.com/compute-monitoring-alert/">comprehensive metrics</a>, but translating those numbers into actions requires expertise that many teams simply don’t have.</li><li><strong>Problems escalate silently.</strong> By the time you notice something’s wrong, it’s often already impacting your business, your budget, or both.</li></ul><h3>Introducing Gradient Insights: your job intelligence partner</h3><p>Gradient Insights analyzes your job metrics and surfaces what matters, giving you clear visibility into potential issues before they become full-blown problems. Some insights also share opportunities for further <a href="https://synccomputing.com/optimize-compute-spend/">cost savings</a>.</p><p>Here’s what Insights detects:</p><h4>High failure rate</h4><p><strong>What this means: </strong>Over 25% of job runs have failed.</p><p><strong>Impact:</strong> Normally you may be notified or alerted when a job fails. However, this insight goes beyond that and looks at how this job has performed historically so you can be proactive instead of reactive. Failed runs hit you twice; they waste time and money. Your team spends valuable hours troubleshooting while you’re still paying for the compute resources that produced zero value.</p><p><strong>Recommended action</strong>: Investigate the job and fix underlying issues to improve the success rate. Gradient highlights these jobs so you can prioritize them for immediate attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VT9X_vjOFZ4ebM0A.png" /></figure><h4>High variability</h4><p>Gradient Insights identifies three types of variability that can signal deeper issues:</p><p><strong>1. Variable costs</strong></p><p><strong>What this means: </strong>Job costs are spread out by more than 25% from their average.</p><p><strong>Impact:</strong> Unpredictable costs wreak havoc on budgeting and forecasting. Your finance team will thank you for addressing these.</p><p><strong>Recommended action:</strong> Verify if this variability is expected or if it indicates an underlying issue with your job configuration or data processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D91RMqeuvkrbQd5x.png" /></figure><p><strong>2. Variable data sizes</strong></p><p><strong>What this means: </strong>The data size processed fluctuates by more than 25% from the average.</p><p><strong>Impact:</strong> <a href="https://synccomputing.com/variable-size-data/">Unexpected changes to data size</a> can indicate upstream problems. One Gradient customer discovered their job had jumped from processing 500GB to over 1TB, which led them to uncover a critical data pipeline error.</p><p><strong>Recommended action:</strong> Investigate the cause of data size variability and determine if it aligns with business expectations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DD2RsMdxYcxbeRlG.png" /></figure><p><strong>3. Variable runtimes</strong></p><p><strong>What this means: </strong>Job completion times are inconsistent and are spread out by more than 25% from the average.</p><p><strong>Impact: </strong>SLA risk increases with runtime variability. When critical downstream processes depend on timely data delivery, variable runtimes can cascade into bigger problems.</p><p><strong>Recommended action:</strong> Examine the job’s configuration and execution patterns to stabilize performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MRezJZUiG6qxDR7C.png" /></figure><h4>Dormant jobs</h4><p><strong>What this means: </strong>No job runs in over two weeks.</p><p><strong>Impact: </strong>While not always urgent, dormant jobs can indicate unintentional disabling or represent organizational clutter. Even worse, upstream processes might still be running, producing data that nobody uses.</p><p><strong>Recommended action: </strong>Confirm whether the dormancy is intentional. If so, consider removing the job and auditing associated upstream and downstream tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eG1G-HTp_SB7eDsM.png" /></figure><h4>Unused jobs</h4><p><strong>What this means: </strong>Almost no change to cost and runtime for this job.</p><p><strong>Impact: </strong>Jobs with suspiciously consistent metrics are prime candidates for review as they often represent automated processes that everyone assumed were necessary but no one actively monitors. Nearly all jobs naturally exhibit some variability; spot pricing fluctuations, data size changes, and code updates typically cause metrics to fluctuate between runs. When a job shows remarkably consistent metrics over time, it often indicates a forgotten process that’s quietly consuming resources without delivering business value. These jobs can silently drain your budget month after month.</p><p><strong>Recommended action: </strong>Verify whether this job is still delivering actual value to your organization. If not, consider stopping the job entirely or migrating it to a more cost-effective solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ymY5AoLgWG68CaWO.png" /></figure><h3>Project details — understand “why”</h3><p>We’ve embedded insights into every project to give you a side-by-side view of the Spark metrics and the corresponding insights extracted from the data. This is extremely helpful when diagnosing issues, optimizing performance, and mastering your Databricks environment.</p><ul><li><strong>Faster root cause analysis:</strong> Quickly identify and resolve issues without shifting thought endless lists of logs. Identify what caused an anomaly and its source in under a minute.</li><li><strong>Effortless anomaly understanding:</strong> Gain the context required to understand why anomalies occurred by connecting Spark metrics, like shuffle and disk spill, to cost and performance information.</li><li><strong>Track historical cluster changes: </strong>Gradient automatically logs every cluster change, along with who made it (Gradient/user). Use these logs to quickly see if a change in the cluster is behind the anomaly you are observing.</li><li><strong>Empower Spark comprehension:</strong> Improve your grasp of Spark operations, by reviewing low level metrics in tandem with their high level impact on cost and performance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y2OKb-zDniIr3YBk.png" /></figure><p>But wait, there’s more! The insights above are just the tip of the iceberg. This initial delivers immediate, tangible value to our customers. Think of it as the appetizer before the main course.</p><h3>Coming soon</h3><p>We’re working on new insights that will take your Databricks job management to the next level:</p><ul><li><strong>Cost and performance anomaly alerts:</strong> Say goodbye to those sneaky budget-busters and performance hiccups. We’ll give you a heads-up before they can ruin your day.</li><li><strong>Data pipeline growth insights: </strong>Watch your data pipelines evolve. We’ll help you understand how they’re growing and what that means for your infrastructure.</li><li><strong>GPU optimization suggestions:</strong> Ever wondered if throwing some GPU power at your jobs is the right move? Gradient’s integration with <a href="https://docs.rapids.ai/index.html">NVIDIA RAPIDS</a> lets you know when to do that, how much it will save you, and how it will impact performance.</li></ul><p>These upcoming features aren’t just bells and whistles — they’re the secret sauce that will help you squeeze every last drop of efficiency out of your Databricks environment. Stay tuned, because the best is yet to come!</p><h3>Conclusion</h3><p>In today’s data-driven environment, the difference between success and failure often comes down to how quickly you can convert raw data into actionable intelligence. Gradient Insights bridges that gap for your Databricks jobs, transforming complex metrics into clear guidance.</p><p>By highlighting what matters most — failures, variability, dormancy, and waste — Insights helps you:</p><ul><li><strong>Proactively address issues</strong> before they impact your business</li><li><strong>Optimize your Databricks spend</strong> by identifying inefficiencies</li><li><strong>Focus your team’s energy </strong>on high-impact improvements</li><li><strong>Maintain reliability</strong> of your critical data pipelines</li></ul><p>The days of being blindsided by job failures or unexpected costs are over. With <a href="https://synccomputing.com/">Gradient</a>, you’re not just monitoring your jobs, you’re mastering them.</p><p>Interested in learning more? <a href="https://landing.synccomputing.com/request-a-demo">Book a time to connect with the team here!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=179707323c8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/from-chaos-to-clarity-how-gradient-insights-saves-your-budget-and-sanity-179707323c8b">From chaos to clarity: How Gradient Insights saves your budget (and sanity)</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the power of Declarative Computing]]></title>
            <link>https://medium.com/sync-computing/unleashing-the-power-of-declarative-computing-a2ceb32b1c30?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/a2ceb32b1c30</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineer]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 18:01:08 GMT</pubDate>
            <atom:updated>2025-03-06T18:01:08.599Z</atom:updated>
            <content:encoded><![CDATA[<h4>Insights from our journey building Gradient, the world’s first AI compute optimization engine.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_57x-3OU9PBrPxUQ.png" /></figure><p>Imagine a world where you could simply tell your data infrastructure what you want it to achieve, rather than meticulously configuring every detail. That’s exactly what Sync Co-founder and CEO, Jeff Chou, and Alation Co-founder and CEO, Satyen Sangani, discuss in this episode of the Data Radicals podcast.</p><blockquote><em>“Instead of having to pick the right resources and set all the configuration settings- just declare the outcomes that you want!” — Jeff Chou, Co-founder &amp; CEO of Sync Computing</em></blockquote><p>Tune into this insightful conversation to learn more about <a href="https://synccomputing.com/product">Gradient</a>, the world’s first AI compute optimization engine, or read on for a summary of the key points.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_wJepqvoBSw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_wJepqvoBSw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_wJepqvoBSw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ac1860265b7f9ce3dac17b2c9859f4a/href">https://medium.com/media/0ac1860265b7f9ce3dac17b2c9859f4a/href</a></iframe><h3>What is Declarative Computing?</h3><p>Declarative Computing flips the script on traditional infrastructure management. Instead of specifying exact resources and cluster configurations, you simply input your desired outcomes, such as cost or runtime, and the appropriate resources are allocated based on your goals and needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xA4v7DmNv-raRPNH.png" /></figure><p>Today, data engineers endure the trial-and-error process of configuring clusters for their jobs. This is manual, error prone, time consuming, and soul crushing work. By shifting the focus from managing resources to achieving business goals, declarative computing enables users to simplify infrastructure management. It also aligns data infrastructure management more closely with business objectives, so that performance improvements translate directly into tangible benefits.</p><p>Learn more about <a href="https://synccomputing.com/what-is-declarative-computing/">Declarative Computing</a></p><h3>The power of Gradient</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*HOu2ZuxA2nC1QJe-.png" /></figure><p>Gradient embodies the essence of Declarative Computing. Want to minimize costs while hitting a specific runtime and latency? Just say the word, and Gradient will figure out the optimal configuration for your jobs using ML-powered optimization.</p><p>It’s not just another optimization tool; it’s a paradigm shift in how we approach Databricks workloads:</p><ul><li><strong>Cost savings by default:</strong> Gradient is set to cut costs to a minimum out of the box.</li><li><strong>Flexible performance targets: </strong>Need to <a href="https://synccomputing.com/control-runtimes-slas/">hit specific runtime SLAs</a>? Gradient ensures you meet your goals at the lowest price point.</li><li><strong>Self-improving models: </strong>Our machine-learning algorithms <a href="https://synccomputing.com/fine-tuning-models/">continuously adapt to changes in your data pipelines</a>, ensuring optimal performance even when facing anomalies.</li></ul><p>Gradient’s advanced machine-learning algorithms are the key to our success. These models continue to improve using a closed-loop feedback system. After each run, performance and cost information are streamed back into Gradient, so the models can tie these outcomes to the configuration changes made. As a result, projections and recommendations improve for jobs every time they run.</p><h3>Looking to the future</h3><p>While Gradient currently focuses on Databricks Jobs, our vision extends far beyond. We’re setting our sights on other data clouds and even <a href="https://synccomputing.com/sync-nvidia-inception-expanding-cpu-gpu-management/">GPUs in the cloud</a>. The concept behind Declarative Computing can be applied to any repeatable workload, including: ETL jobs, open-source Spark,EMR on AWS, serverless functions, Fargate, ECS, Kubernetes, GPUs, and any other system that runs scheduled jobs.</p><p>Our end goal is to build a universal platform that effortlessly manages diverse computing environments. We hope you join us on this journey!</p><h3>Conclusion</h3><p>The shift to Declarative Computing isn’t just about technological advancement; it’s about aligning your data infrastructure directly with your business goals. Instead of guessing what resources you need to run your jobs now and in the future, simply share your goals with <a href="https://synccomputing.com/">Gradient</a> and it will manage your clusters for you.</p><p>As cloud infrastructure evolves, the need for agile and cost-effective solutions becomes increasingly critical. Gradient continuously monitors and adapts to changes in your data pipelines, so your data infrastructure is always optimal. Join us as we build a future where data infrastructure optimization is effortless using AI automation.</p><p>Interested in learning more about Gradient and what it can do for your organization?</p><p><a href="https://landing.synccomputing.com/request-a-demo">Book a personalized demo here!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2ceb32b1c30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/unleashing-the-power-of-declarative-computing-a2ceb32b1c30">Unleashing the power of Declarative Computing</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient Introduces Cloud and Databricks Cost Breakdowns]]></title>
            <link>https://medium.com/sync-computing/gradient-introduces-cloud-and-databricks-cost-breakdowns-c38d6217f0d2?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/c38d6217f0d2</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Wed, 19 Feb 2025 18:28:27 GMT</pubDate>
            <atom:updated>2025-02-19T18:28:27.527Z</atom:updated>
            <content:encoded><![CDATA[<h4>Get the full picture with a single pane of glass for your cloud spend</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*WRk8LjO0wX4rRU-y.jpg" /></figure><p>For data teams, understanding the true cost of operations has always been a complex puzzle. This is because your monthly bills come from multiple sources.</p><p>For example, when using Databricks you have:</p><ol><li>The Databricks bill for DBU consumption</li><li>Your cloud provider’s bill (AWS, Azure, or GCP) for the infrastructure powering your Databricks workloads</li></ol><p>This fragmented view makes it challenging to understand your true total cost of ownership (TCO). It also makes it difficult to identify opportunities for optimization. To combat these issue, we built a Cost Breakdown feature for Gradient. Read on for all the details!</p><h3>The cost visibility challenge</h3><p>Today’s data teams face several challenges when it comes to managing the costs of their Databricks workloads:</p><ol><li><strong>Fragmented billing:</strong> Costs are split between Databricks processing and cloud provider compute bills, making it difficult to get a complete picture</li><li><strong>Limited granularity:</strong> Most solutions only show aggregated costs at the workspace level</li><li><strong>Manual reconciliation:</strong> Teams are forced to spend valuable time combining data from different sources to understand their true spending behavior</li><li><strong>Delayed insights:</strong> Current solutions rely on the billing systems provided by cloud providers and Databricks, which aren’t real-time</li></ol><h3>Introducing Gradient’s cost breakdown feature</h3><p>We’re excited to announce that <a href="https://synccomputing.com/product">Gradient</a> now provides detailed cost breakdowns, showing you exactly how much you’re spending on Databricks versus your cloud infrastructure.</p><p>This new feature adds complete transparency to your data operations costs, by displaying both granular and aggregated costs in a single pane.</p><h4>What you can see</h4><ul><li><strong>Databricks costs:</strong> Clear visibility into your DBU consumption and associated costs</li><li><strong>Cloud infrastructure costs:</strong> Detailed breakdown of spending on AWS, Azure, or GCP resources used to power you workloads</li><li><strong>Workload-level details:</strong> Understand Databricks and cloud compute costs at the individual job level, not just workspace aggregates</li><li><strong>Temporal views:</strong> Track costs over time with daily, weekly, or monthly granularity</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rKNTQ3e5hmjNBdkK.png" /><figcaption>Cost breakdown: Last quarter, aggregated weekly</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8s7QblFNDg-oTY6e.png" /><figcaption>Workload-level cost breakdown</figcaption></figure><h3>Why this matters</h3><p>Understanding your data infrastructure costs is fundamental for data teams. Simply put, you cannot improve what you do not measure.</p><p>The level of cost transparency provided by Gradient enables you to identify opportunities for optimization across Databricks and cloud infrastructure. It also helps you justify spending by demonstrating ROI, proactively manage budgets before costs escalate, and make better informed decisions about resource allocation.</p><p>With Gradient’s cost breakdown feature, you can:</p><ol><li><strong>Track trends:</strong> Monitor how your costs evolve over time and across different types of workloads</li><li><strong>Optimize holistically:</strong> Make decisions that consider both Databricks and cloud infrastructure costs</li><li><strong>Forecast accurately:</strong> Better predict future spending based on comprehensive historical data</li><li><strong>Report confidently:</strong> Share clear, detailed cost breakdowns with stakeholders</li></ol><p>But Gradient doesn’t just show you the numbers, it also helps you optimize them. Our AI-powered optimization engine continuously analyzes your workload patterns and makes intelligent recommendations to <a href="https://synccomputing.com/optimize-compute-spend/">reduce costs</a> while <a href="https://synccomputing.com/control-runtimes-slas/">maintaining performance</a>. By understanding both your Databricks and cloud infrastructure costs, Gradient can suggest optimizations that consider the complete cost picture.</p><h3>Getting started</h3><p>The new cost breakdown feature is available now in your Gradient dashboard. Simply <a href="https://app.synccomputing.com/">log in</a> and onboard your workloads to get visual breakdowns of Databricks vs. cloud costs, aggregated cost trends over time, workload-level insights, and much more.</p><h3>What’s next?</h3><p>This is just another step in our journey to automate data infrastructure management for Databricks users. We’re continuously working on new features to help you understand and optimize your data infrastructure better.</p><p>As you well know, <a href="https://synccomputing.com/cluster-tuning-compute/">cluster management</a> is very complicated. Things don’t always make sense, even when you have the raw metrics, like those that Gradient shows. This is why we’ve built an Insights feature that helps you get a better understanding of the nature of your workloads and distills insights based on lessons learned from managing tens of millions of core hours.</p><p>Below you can see a sneak peak of our upcoming Insights — stay tuned for more!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*O0bZO3URbJ_8F-ri.png" /></figure><h3>Conclusion</h3><p>Gradient’s cost breakdown feature marks a significant advancement in cost management for Databricks users. By providing a unified view of Databricks and cloud infrastructure expenses, data teams can make more informed decisions and optimize their resources effectively. The ability to analyze costs at a granular level, combined with <a href="https://synccomputing.com/">AI-driven optimizations</a> and insights, offers a powerful solution.</p><p>As organizations increasingly rely on data-driven operations, tools like Gradient’s cost breakdown feature will become essential for maintaining financial efficiency and demonstrating ROI in the complex landscape of cloud-based data infrastructure.</p><p>If this sounds interesting to you and you haven’t yet seen Gradient in action, <a href="https://landing.synccomputing.com/request-a-demo">book a time here</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c38d6217f0d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/gradient-introduces-cloud-and-databricks-cost-breakdowns-c38d6217f0d2">Gradient Introduces Cloud and Databricks Cost Breakdowns</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks System Table Workspace Health SQL Toolkit]]></title>
            <link>https://medium.com/@synccomputing/databricks-system-table-workspace-health-sql-toolkit-940330169e9e?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/940330169e9e</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-system-tables]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Wed, 05 Feb 2025 16:08:45 GMT</pubDate>
            <atom:updated>2025-02-05T16:08:45.838Z</atom:updated>
            <content:encoded><![CDATA[<h4>20 System Table queries to understand Databricks usage across the organization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gbc1B_yfd8ZAX56t" /></figure><p>As data engineers, understanding the intricacies of our Databricks environment is important. You can’t optimize performance, budget or ensure efficient resource allocations without it. Thankfully, Databricks gives you a behind-the-scenes look at how your workspace is running in <strong>system tables. </strong>Everything from query performance to job execution and cluster activity is in those tables.</p><p>But raw system data can be tricky to navigate, and sometimes you just need a quick answer to that burning question. That’s why we’ve created a <a href="https://synccomputing.com/databricks-query-toolkit/"><strong>Databricks Health SQL Toolkit</strong></a> — a free set of queries you can copy, paste, and run to get instant insights into your environment. Whether you’re debugging slow queries, tracking costs, or just curious about what’s happening under the hood, these queries should help you get there faster.</p><p>If you find these queries to be useful, you can<a href="https://synccomputing.com/databricks-query-toolkit/#book-call"> download our 1-click dashboard for Databricks</a> to visualize all of them in a sharable dashboard. Wow the team with insights in your Jobs, SQL warehouses, APC clusters, and DLT usage.</p><h3>General Usage</h3><p>Let’s start with the basics. Here are a few queries to help you get a quick overview of how your organization is using Databricks and determine your next steps.</p><h4>Who submits the most amount of queries?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cTm5T-509ZMfN-tM" /></figure><p>Knowing who your most active users by SQL query count helps shed valuable insights into workspace utilization and helps you:</p><ul><li>Identify power users who might benefit from additional training or resources</li><li>Detect unusual activity patterns that could indicate security issues or inefficient practices</li><li>Better understand how your organization uses Databricks</li></ul><p>This will be essential to understand how to optimize your Databricks environment to meet the needs and usage patterns of your team. It will also be useful for cost allocation and user management.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>Who owns the most expensive clusters?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UCUbPTtrDrLfhsAH" /></figure><p>Pinpointing the users that own and run your most expensive Jobs and All-Purpose Compute clusters is the next logical step on the road to cutting costs.</p><p>These insights enable you to:</p><ul><li>Identify the owners of your most expensive clusters</li><li>Attribute costs to users and groups</li><li>Detect anomalies and potential misuse</li></ul><p>Use this query to identify the users to follow up with to understand why costs are so high and devise a plan to reduce them.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>How many clusters are currently active, and how were they created?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jvbNqGiqleeGBS5j" /></figure><p>Understanding your cluster creation activity is crucial for resource management and cost optimization.</p><p>This enables you to:</p><ul><li>Ensure cluster creation practices are inline with policies</li><li>Understand cluster creation trends</li><li>Identify rogue clusters created in alternative ways</li></ul><p>Knowing where cluster creation originates can help optimize your workflows and ensure better governance.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>Which markets (Spot, On-Demand) do my clusters use?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FzjLVRbE0dqk5_67" /></figure><p>Market usage data helps you balance the cost and reliability of your infrastructure. While Spot instances may offer up to 90% lower costs, this is at the expense of reliability. Cloud providers can terminate nodes at any moment, with little to no notice, causing interruptions and even job failures.</p><p>This insight will enable you to:</p><ul><li>Understand spot vs on-demand usage across the organization</li><li>Identify opportunities to leverage Spot</li><li>Ensure efficient use of Spot as fallback</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>How “sticky” is my workspace?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MyMPlcd7jEddUI11" /></figure><p>Databricks stickiness is a critical metric to understand user engagement. It is calculated by dividing daily active users by monthly active users and helps:</p><ul><li>Gauge the extent of adoption across the organization</li><li>Identify trends and changes in usage over time</li><li>Alert on usage anomalies and potential misuse</li></ul><p>A high stickiness ratio means that users are consistently using the workspace, which indicates that you have successfully adopted Databricks within your organization.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h3>Cost Optimization</h3><p>Understanding usage is nice, but cutting costs is the ultimate end goal. Here are a few useful queries to help you pinpoint where to focus your optimization efforts for the most impact.</p><h4>What are the current compute costs associated with my Databricks workspace?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oH3V4A7DYc9tC-eF" /></figure><p>Keeping track of your Databricks compute costs is fundamental for budget management. With some many options (classic, serverless, DBSQL; APC vs on-demand) it’s hard to know what to use when.</p><p>These insights help you:</p><ul><li>Pinpoint high spend targets</li><li>Identify over and under used options</li><li>Detect unexpected or over consumption</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are the product costs associated with my Databricks workspace?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L6Xoguj9IEhmhc60" /></figure><p>Knowing which Databricks products you are spending the most on allows you to implement targeted cost-saving measures to ensure you’re getting the most out of your investment.</p><p>These insights let you:</p><ul><li>Identify high cost targets for optimization</li><li>Get a bird’s eye view of how the org spends on Databricks</li><li>Detect trends and changing patterns</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>Which jobs are most likely over-provisioned?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PwcAoykXIHuBnLPO" /></figure><p>Identifying the workloads with the lowest average CPU usage is a quick way to detect over-provisioned clusters. This information helps you:</p><ul><li>Reduce spend from over-provisioned resources</li><li>Detect misconfigurations</li><li>Better allocate resources</li><li>Understand which jobs are best to use Gradient to auto-optimize</li></ul><p>This query is a great place to start your waste reduction efforts.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>Which jobs are most likely under-provisioned?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w0KbS9Dm99c5fBar" /></figure><p>Identifying the data pipelines with the highest average memory utilization is an easy way to detect potential bottleneck and under-provisioned resources. These insight enables you to:</p><ul><li>Identify jobs with potential bottlenecks</li><li>Improve performance with increased memory allocation</li><li>Detect anomalies and trends over time</li><li>Understand which jobs are best to use Gradient to auto-optimize</li></ul><p>Detecting bottlenecks will help you to improve performance, by identifying the workloads that will benefit from additional resources.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are my most expensive jobs over the past 30 days?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VCQAQ_56FyIomGfY" /></figure><p>Knowing which jobs are costing you the most is crucial for optimizing your data infrastructure. This information helps you:</p><ul><li>Identify the best candidates for optimization</li><li>Implement cost-saving for high-spend workloads (or use Gradient for <a href="https://synccomputing.com/manage-compute-clusters/">automated Jobs cluster tuning</a>)</li><li>Better understand your Databricks workload spend</li></ul><p>Keeping tabs on your most expensive jobs every month is a great way to anticipate spend and detect costly workloads before they incur too many expenses.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What jobs are growing the most in costs week-over-week?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Nb9gIdY8fG2KZ4l" /></figure><p>Workload costs can grow gradually over time. Understanding which of your workloads is growing the most in costs each week enables you to:</p><ul><li>Be the first to identify jobs that are getting increasingly more expensive</li><li>Pinpoint workloads for optimization</li><li>Detect anomalies and potential problems</li></ul><p>By following week over week increases in spend we can identify jobs that are becoming increasingly more expensive, even if spend grows slowly over time.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are the most expensive SQL warehouses for the last 30 days?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UptumBwM2ixue4Pk" /></figure><p>Pinpointing which warehouses are driving up costs over the last 30 days is a great way to attribute costs to users and teams. This information helps you to:</p><ul><li>Tie expensive warehouses to owners</li><li>Simplify chargebacks and cost attribution</li><li>Identify anomalies and trends over time</li></ul><p>This query should be your first step when looking to optimize warehouse costs. It will ensure you focus your efforts on high-impact targets and give you everything you need to plan your next steps.</p><p>Here’s an example output for this query:</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are the most expensive SQL queries over the past 30 days?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nNNJLZVH4w-9oWQZ" /></figure><p>Query costs can grow gradually with every run. Use this query to clear out the noise and focus on your most expensive queries by total cost for the last 30 days. This information enables you to:</p><ul><li>Pinpoint top candidates for optimization</li><li>Detect anomalies and potential issues</li><li>Investigate costs increases before they become substantial</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><p>This query helps you understand which queries to investigate further. The following query add much needed information to help you take your next steps.</p><h4>Why are these queries so expensive, and who owns them?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eGB1LIStbUlGx1gT" /></figure><p>Pinpointing the queries that are costing you the most in a great first step for optimization. But it is just a first step. This query adds granular information about each expensive query. This information helps you:</p><ul><li>Find queries that require optimization</li><li>Understand per run and overall costs</li><li>Better allocate resources</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are the most costly APC clusters over the past 30 days?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*quyCgM9uy2_7wVLf" /></figure><p>Understanding which APC clusters are costing you can help you cut costs by pinpointing candidates for optimization. This information lets you:</p><ul><li>Focus optimization effort on high-value targets</li><li>Identify potential candidates for on-demand clusters</li><li>Better allocate resources</li></ul><p>Use this query to pinpoint the clusters that are prime for further investigation.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/"><strong>Go here for the query</strong></a></p><h4>Which APC clusters are likely under-utilized?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PA0pAG790HnYu_HV" /></figure><p>Identifying your underutilized APC clusters in the past 30 days, along with their relative costs, is a great way to facilitate cost savings and resource optimization. These insights enable you to:</p><ul><li>Create a short list of clusters for downsizing</li><li>Cross reference usage with cost to identify issues</li><li>Detect trends over time</li></ul><p>Use this query to detect under-utilizated APC clusters and connect the dots between those clusters to users that created them.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are the most expensive DLT clusters over the past 30 days?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kc82ttJbAYERQ0LM" /></figure><p>Knowing which delta live tables clusters are costing you the most is the first step to optimization. This information lets you:</p><ul><li>Pinpoint DLT clusters for optimization</li><li>Identify tables for further analysis</li><li>Detect anomalies and trends over time</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h3>Performance Management</h3><p>Two queries to help optimize the performance of your Databricks ecosystem.</p><h4>Which notebooks are consuming the most DBUs over time?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aFiRqMq-friYHuZ-" /></figure><p>Determining which notebooks consume the most DBUs will help target your optimization efforts for maximum effect. This information helps you:</p><ul><li>Identify notebooks for optimization</li><li>Find problems and misuse</li><li>Charge costs based on notebook owner</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>What are my longest running queries?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8JKN404U91Dg-VKY" /></figure><p>Identifying performance bottlenecks is key to keeping your data infrastructure efficient and responsive. This information helps you:</p><ul><li>Pinpoint queries that might be slowing down your system</li><li>Detect which queries need more resources vs code optimization</li><li>Identify patterns in slow queries that could indicate underlying issues</li></ul><p>By optimizing these queries, you can make a big difference in the overall responsiveness and efficiency of your Databricks workspace.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h3>Data and Workflow Management</h3><p>A couple queries to help manage your data and workflows on Databricks</p><h4>What are my most commonly used datasets? Where are they stored?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MUnjGNdfn0eMoUhj" /></figure><p>Knowing which of your datasets is used the most is crucial for optimizing data access and storage. This insight helps you:</p><ul><li>Prioritize optimization efforts by dataset to help lower storage costs</li><li>Apply caching strategies for frequently accessed data</li><li>Make informed decisions about data placement and replication</li></ul><p>Optimizing your most-used datasets may help improve overall performance and reduce data transfer cost across your Databricks environment.</p><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h4>How do I track lineage for data transformations in the workspace?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-7jR_PPLgSVV9Tej" /></figure><p>Visualize all the data lineage in your workspace. By mapping the relationships between source and target tables, you gain clarity into how data flows through the workspace.</p><p>This information helps you understand:</p><ul><li>Dependencies between datasets and transformations</li><li>Where the data used in critical workflows originates from</li><li>Upstream or downstream effects of changes</li></ul><p><a href="https://synccomputing.com/databricks-query-toolkit/">Go here for the query</a></p><h3>Get the queries</h3><p>For the SQL queries to answer these questions and to access a comprehensive 1-click dashboard visualizing all these insights, go directly to the <a href="https://synccomputing.com/databricks-query-toolkit/">Databricks SQL Query Toolkit page</a>.</p><p>This toolkit provides you with ready-to-use queries and a powerful dashboard to help you gain deep insights into your Databricks environment quickly and easily. Paste these queries into your Databricks SQL Editor to see the results, or fill out the form to get a comprehensive dashboard with answers to all these questions.</p><h3>Get the comprehensive dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b8Ed0l7w8bgnYnqr" /></figure><p>If you’d like all of the queries with corresponding plots all in 1 convenient dashboard, request the comprehensive dashboard here.</p><h3>Conclusion</h3><p>By regularly asking and answering questions like these, you’ll be well-equipped to optimize your Databricks workspace, reduce costs, and improve overall performance. Remember, the key to effective data engineering is not just in processing data, but in understanding and optimizing the environment in which that processing occurs.</p><p>We’ll be adding additional queries to the <a href="https://synccomputing.com/databricks-query-toolkit/">Databricks SQL toolkit and 1-click dashboard</a>, so stay tuned!</p><p>Do you think these queries are useful? Are we missing any? We’d love to hear what you think! Feel free to <a href="https://synccomputing.com/contact/">drop us a line here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=940330169e9e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing EC2 costs on Databricks]]></title>
            <link>https://medium.com/sync-computing/optimizing-ec2-costs-on-databricks-fa67a02505ab?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/fa67a02505ab</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[aws-cost-optimization]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 18:23:39 GMT</pubDate>
            <atom:updated>2025-01-28T18:23:39.665Z</atom:updated>
            <content:encoded><![CDATA[<h3>Insights and strategies to optimize EC2 usage on Databricks for cost-effective data processing.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fl-xWekfTBtgaR-1.png" /></figure><p>The global data landscape is experiencing remarkable growth, with unprecedented increases in data generation and substantial investments in analytics and infrastructure. According to data from sources like <a href="https://www.networkworld.com/article/966746/idc-expect-175-zettabytes-of-data-worldwide-by-2025.html">Network World</a> and, G2 the global datasphere is projected to expand from 33 zettabytes in 2018 to an astounding 175 zettabytes by 2025, reflecting a compound annual growth rate (CAGR) of 61%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/0*KtzwltyjeUBO-8oP.png" /></figure><p>As the volume of data continues to grow, so does the need for specialized skills to effectively manage it. This brings us to today’s topic: exploring strategies to manage your organization’s data infrastructure in the most efficient and cost-efficient way possible.</p><h3>Databricks clusters and AWS EC2</h3><p>In today’s landscape, big data, which is data too large to fit into a single node machine, is transformed and managed by clusters. But what are clusters?</p><p>In Databricks, clusters are groups of virtual computers that work together to process and analyze big data. Think of them as a team of workers who divide the tasks among themselves to get the job done faster and more efficiently.</p><p>Read more on <a href="https://synccomputing.com/databricks-cluster-types/">Databricks cluster types</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qsN1qxV2NwBNnIJq.png" /><figcaption>Example Architecture of Spark on EMR</figcaption></figure><p>On a more granular level, an Amazon Elastic Compute Cloud (EC2) instance is a single unit of computing power that is used to address a single virtual machine. Clusters are made up of many EC2 Instances, which consist of driver nodes and worker nodes.</p><ul><li><strong>Driver node (master): </strong>This is the leader of the team. Its main job is to decide what needs to be done, divide the work into smaller tasks, and assign these tasks to the workers. The driver also keeps track of progress and collects the results when the workers finish their tasks.</li><li><strong>Worker nodes: </strong>These are the team members who do the actual work. They take the tasks assigned by the driver, process the data, and send the results back to the driver.</li></ul><p>Databricks clusters contain one driver node and one or more worker nodes. Clusters can scale up or down by adding or removing worker nodes, depending on the workload. For example, when processing a large dataset, you can add more EC2 worker nodes to speed up the task. Once the workload is finished, you can scale down to save costs.</p><p>Clusters offer the flexibility to select different types of EC2 instances tailored to your specific needs. You can choose from a wide range of options based on factors like computing power, memory, storage, and more. These choices play a crucial role in optimizing performance and cost efficiency for your workloads.</p><h4>Clusters in Databricks</h4><p>Databricks offers Job clusters for data pipeline processing and warehouse clusters used for the SQL lakehouse. Job clusters have far more sizing options. This post focuses on optimizing job clusters using EC2 instances for Databricks workloads.</p><p>In Databricks, you can choose between two cluster types to suit your workload needs. The Databricks Classic cluster allows you to manually configure and provision resources, offering full control over the setup. Alternatively, the Databricks Serverless Compute is a cluster type that simplifies this process by automatically configuring and managing resources for you — though it comes at a premium cost.</p><p>Serverless clusters are ideal for lightweight, quick tasks, while Classic clusters are better suited for handling large-scale jobs involving substantial amounts of data. In this post, we focus on the use of Classic clusters for managing and optimizing heavy data workloads effectively.</p><p>Read our comprehensive comparison of <a href="https://synccomputing.com/databricks-compute-comparison-classic-serverless-and-sql-warehouses/">Databricks serverless vs. classic</a>.</p><h3>Understanding EC2 cost structure</h3><p>When using Amazon EC2, understanding the cost structure is crucial to managing and optimizing your cloud expenses effectively. The costs can often be underestimated, but as your data scales, these expenses can grow unexpectedly, potentially reaching surprising levels without you knowing.</p><h4>1. Instance types (e.g., R7g, M6gd, C6i)</h4><p>Amazon EC2 offers a wide variety of instance types tailored to different workloads, each providing unique benefits that influence performance and cost. The instance type you choose can significantly affect your cloud expenses based on your workload’s requirements.</p><p>Here are a few examples of instance types:</p><ul><li>R7g instances, powered by AWS Graviton3 processors, are memory-optimized EC2 instances designed for high-performance, memory-intensive workloads. They offer a high memory-to-CPU ratio, with configurations providing up to 1 Terabyte of memory, making them ideal for in-memory databases, big data analytics, and real-time processing. AWS Graviton3 processors enhance performance by up to 25% compared to their predecessors, with significant energy efficiency gains of up to 60%, reducing costs and supporting sustainability goals.</li><li>M6GD instances are general-purpose EC2 instances equipped with AWS Graviton2 processors and local NVMe-based SSD storage, offering a balanced mix of compute, memory, and storage. These instances are ideal for workloads that require high-speed local storage, such as caching, databases, and containerized applications. In the context of EC2 on Databricks, the inclusion of local NVMe storage is useful for spilling data to disk temporarily when instance memory starts to become exhausted.</li></ul><p>Related: <a href="https://synccomputing.com/do-graviton-instances-lower-costs-for-spark-on-emr-on-aws/">EMR Graviton and whether Graviton instances lower costs for Spark on AWS</a></p><h4>2. Regions and availability zones</h4><p>AWS operates in multiple geographic regions worldwide, and each region has its own pricing based on demand, infrastructure costs, and operational expenses. For example, instances in North America may be less expensive than those in Asia-Pacific regions. Within each region, there are multiple availability zones to ensure high availability. For a full breakdown of EC2 pricing by region and zone, <a href="https://aws.amazon.com/ec2/pricing/on-demand/">go here</a>.</p><p>While we see substantial differences in rates by region, optimizing cost is not as straightforward as choosing the lowest-priced region. Data transfers between regions or zones incur additional costs that can outweigh the cost savings, not to mention the impact on performance. Provisioning EC2 instances in the same region as your data is not only important from a cost perspective, it also reduces access latency and increases transfer speed.</p><p>Understanding regional, zone, and data transfer pricing differences is key to optimizing your expenses, especially for globally distributed applications.</p><h4>3. EC2 storage options</h4><p>Amazon EC2 provides several key storage options tailored to different workload needs:</p><ol><li><strong>Elastic Block Store (EBS):</strong> Persistent, high-performance block storage ideal for boot volumes, databases, and data warehouses. It offers various types, including General Purpose (gp3, gp2) for standard workloads, Provisioned IOPS (io2, io1) for high-performance applications, and throughput-optimized HDD for large sequential workloads. Read more about <a href="https://synccomputing.com/why-your-databricks-cluster-ebs-settings-matter/">EBS optimization</a>.</li><li><strong>Instance Store:</strong> High-speed, ephemeral block storage physically attached to the host server. Suitable for temporary data, such as caching or intermediate processing, with data lost when the instance stops.</li><li><strong>Local NVMe SSDs: </strong>Ultra-low latency, ephemeral storage available with certain instance types (e.g., M6gd, C6gd). Ideal for real-time analytics, high-performance caching, or machine learning, but data does not persist after instance termination.</li><li><strong>Amazon S3:</strong> Highly scalable, durable object storage designed for storing backups, data lakes, logs, and static content. Data is accessed over the network and is persistent, making it ideal for unstructured data storage.</li><li><strong>Amazon EFS (Elastic File System):</strong> Managed, scalable file storage accessible by multiple instances. Designed for shared applications, content management systems, and containerized workloads, with automatic scaling and persistent storage.</li></ol><p>The first three storage options come with the EC2 instance naturally, while the latter options are an external Amazon Service you can integrate with. These options offer flexibility and scalability, allowing you to base your decisions on factors like persistence and latency.</p><h4>4. On-Demand, Spot, and Reserved Instances</h4><p>Amazon EC2 offers three primary pricing models for their instances — On-Demand, Spot and Reserved — each suited to different workloads, budgets, and operational needs.</p><ul><li><strong>On-Demand Instances: </strong>The straightforward way to buy an EC2 instance. You pay what AWS states as their standard pricing and get your instance. It has the highest per-hour rate among the pricing models, and should be used for critical production workloads. Note that you might encounter capacity issues, especially with GPU-powered nodes. For example, if you have a workload that requires 1000 m6g-4xlarge clusters for an hour, you might find you only get some of those from time to time.</li><li><strong>Spot Instances: </strong>A cost-effective alternative for running Amazon EC2 instances, allowing you to access unused compute capacity at significantly reduced rates. However, availability depends on supply and demand, so provisioning is not guaranteed. Keep in mind that AWS can reclaim Spot Instances if capacity is required for other users with higher demand. So make sure to only use Spot instances for workloads that can be interrupted. Despite this, Spot Instances offer substantial cost savings, <a href="https://aws.amazon.com/ec2/spot/pricing/">up to 90% lower</a> than On-Demand pricing.</li><li><strong>Reserved Instances: </strong>A middle ground among the previous two exists, where you reserve a specific EC2 instance type in a particular region for a duration of 1 to 3 years. In exchange for this commitment, <a href="https://aws.amazon.com/aws-cost-management/aws-cost-optimization/reserved-instances/#:~:text=With%20Reserved%20Instances%2C%20you%20can,payment%2C%20the%20greater%20the%20discount.">AWS</a> offers significant discounts of up to 75% compared to on-demand pricing.</li></ul><h4>Comparing EC2 instance pricing models</h4><p><strong>On-Demand Instances:</strong></p><ul><li>Fixed price, highest cost</li><li>Aviliability is guaranteed, if capacity exists</li><li>Best suited for production workloads and other non-interruptible workloads</li></ul><p><strong>Spot Instances:</strong></p><ul><li>Dynamic pricing, up to 90% cheaper.</li><li>Availability is not guaranteed; interruptions are common</li><li>Best suited for fault-tolerant workloads</li></ul><p><strong>Reserved Instances:</strong></p><ul><li>Discounted pricing, up to 75% off.</li><li>1–3 year commitment, with availability guaranteed for the reserved term</li><li>Best suited for extremely predictable workloads that ensure RIs are occupied 24/7</li></ul><h3>Key Strategies for EC2 cost optimization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*GYzeFk5Je_qcT5oK.png" /></figure><p>After exploring the various factors that influence EC2 costs, here are some practical strategies to help you save money.</p><h4>1. Select the right instance type</h4><p>Choosing the right instance type for the right job is like selecting the perfect tool for a task — it can help increase efficiency, optimizes costs, and unlocks the full potential of your workload.</p><p>Start by analyzing the nature of your workloads:</p><ul><li><strong>Compute-Intensive: </strong>For tasks like video encoding, machine learning model training, or high-performance computing (HPC), choose compute-optimized instances (e.g., C6i, C7g).</li><li><strong>Memory-Intensive:</strong> For databases, in-memory caching, or real-time analytics, memory-optimized instances (e.g., R7g, X2idn) are ideal.</li><li><strong>Storage-Intensive: </strong>For workloads requiring large-scale data processing or high-speed storage, look into storage-optimized instances (e.g., I4i, D3en).</li><li><strong>General Purpose:</strong> For balanced computing, memory, and networking needs, use general-purpose instances (e.g., M6i, M7g).</li></ul><p>For example, for Spark jobs without complex transformations we recommend using a high-memory cluster where the entire dataset can fit comfortably within the total memory provisioned. In such scenarios, R7g instances are an excellent choice, offering high memory capacity and optimal performance for memory-intensive tasks. There is no one-size-fits-all solution; you’ll need to evaluate your needs and choose instances accordingly.</p><h4>2. Select the right EC2 pricing model</h4><p>Similarly, choosing the right EC2 instance pricing model can help optimize AWS EC2 for costs. Doing a little bit of groundwork by evaluating your workflow requirements and matching that against a suitable pricing model will always have a positive return.</p><p>Here are a couple insights about each model:</p><ul><li><strong>On-demand Instances for production workloads.</strong> Best suited for jobs that cannot be interrupted or require low spin-up times. A good example can be batch ETL Spark jobs and scheduled automation.</li><li><strong>Spot instances for workloads that can be interrupted. </strong>Spot instances are up to 90% cheaper than on-demand clusters, but this comes at the cost of interruptions and availability issues. Make sure to only use Spot with jobs that can withstand interruptions and potential failures due to node termination.</li><li><strong>Reserved instances for long-term predictable workflows. </strong>These jobs are estimated to run for the next few years, have predictable data sizes, and have been running for some time. Predictability is emphasized here as the number of instances has already been reserved. If there’s an increase in data size or compute, it may cause the jobs to fail due to a lack of sufficient resources. On the other hand, underused instances cause waste, so you want to keep your reserved instances occupied 24/7.</li></ul><p>Depending on the maturity of your data and organization, some companies may require all three pricing models for different workflows. This is to ensure resources are not over or under-utilized.</p><h4>3. Autoscaling policies and elasticity</h4><p>Databricks and AWS EC2 support autoscaling policies, which define rules for automatically increasing or decreasing the number of nodes in a cluster. These policies help control usage costs, set resource limits, and scale up or down based on workload demands. Autoscaling is particularly useful for experienced engineers managing dynamic workloads.</p><p>Example of a scaling policy:</p><ul><li><strong>Scale out:</strong> Add X nodes when total available cluster memory falls below X%. This rule triggers every Y minutes.</li><li><strong>Scale in:</strong> Remove X nodes when no applications are running. This rule also triggers every Y minutes.</li></ul><p>In this example, the policy adds nodes when the cluster’s memory usage exceeds the defined threshold, indicating high resource demand. Conversely, it removes nodes when no workloads are running.</p><p>It should be noted that autoscaling is not one-size-fits-all. Autoscaling might be a perfect fit for your workloads — or it might not. The truth is, the answer often depends on your specific needs. The only way to know for sure is to dive in and experiment.</p><p>Related: Is Databricks autoscaling cost efficient? Find out in this <a href="https://synccomputing.com/is-databricks-autoscaling-cost-efficient/">Databricks autoscaling on the TPC-DS benchmark report.</a></p><h4>4. Storage and region options</h4><p>EC2 comes with a plethora of storage options, and all of them cater to a different use case. Choosing the right storage type for your workload can significantly optimize costs and ensure efficient performance.</p><p>For persistent storage, we recommend S3 as a long term storage solution. It is rich with features revolving around balancing transfer latencies and costs while offering scalability and durability to store large volumes of data. In addition, EC2 images are often stored on S3.</p><p>S3’s tiered storage classes, including Standard, Infrequent Access, and Glacier, allow you to optimize costs based on how frequently data is accessed, making it an excellent choice for backups, archives, and data lakes that require reliable, long-term storage with flexible pricing options.</p><p>For local storage on your machine, EBS storage options typically offer the best value for most use cases. Unless you require ultra-fast transfer speeds, EBS provides sufficient performance at a much lower cost compared to NVMe SSDs</p><p>Only utilize NVMe SSDs for ultra-fast workloads that require ephemeral storage with low latency and high throughput, while directly attached to the instance. Caching is the most common use-case for SSDs as they are one of the most expensive options in this list. The data is also lost when the EC2 instance stops, rendering the usage for persistent data redundant.</p><p>When selecting the right storage, the location of your resources is a critical consideration. Data transfer costs between regions and availability zones can add up, making it essential to plan where your data resides in advance to minimize inter-zone transfer expenses. Additionally, it’s wise to account for the location of your existing resources, whether on-premises or in the cloud, to ensure seamless integration and cost efficiency when deploying EC2 instances.</p><p>Thoughtful planning can significantly reduce unnecessary data transfer costs and improve overall resource management. For reference, the transfer cost between EC2 instance regions is currently about $0.09 per GB.</p><h4>5. Monitor resources and take advantage of credits</h4><p>As the saying goes:<br><em>“Knowledge comes from learning, but wisdom comes from doing.”</em></p><p>You won’t fully understand the exact requirements of your workflows until they’ve been deployed and observed in action. Monitoring helps you identify inefficiencies and optimize resource usage. For instance, it’s not uncommon to find data pipelines using only 5% of the allocated SSD storage for extended periods — wasting valuable resources and increasing costs.</p><p>To effectively manage your resources, use tools like:</p><ul><li><strong>AWS Compute Optimizer: </strong>Analyze your resource usage and receive recommendations for optimizing EC2, EBS, and other services.</li><li><strong>AWS Cost Explorer</strong>: Track and visualize your spending over time.</li><li>A<strong>WS Budgets:</strong> Set custom alerts for when costs exceed predefined limits.</li></ul><p>Additionally, take advantage of free offerings from cloud providers, especially if you’re a small business owner. For example, AWS provides a free EC2 instance (t2.micro) for new accounts as part of its free tier, as well as <a href="https://aws.amazon.com/startups/credits">substantial credits for startups</a>. Use these freebies for experimentation, testing, or ad-hoc workloads, helping you save money while exploring new solutions.</p><p>By continuously monitoring and leveraging available resources wisely, you can significantly reduce unnecessary costs and improve overall efficiency.</p><h4>6. Gradient: AI optimization for data infrastructure</h4><p>We, at <a href="https://synccomputing.com/">Sync</a>, believe that engineers shouldn’t waste time running and testing their workloads on various configurations. They should be spending their time building, while we automate the optimization of resources.</p><p>Our AI agent, <a href="https://synccomputing.com/product/">Gradient</a>, integrates seamlessly with your Databricks workspace, collects logs, and analyzes jobs using advanced machine learning algorithms. It is then able to make changes to your configurations (like node count, instance type) instantly and evaluate the impact of those changes continuously. This closed-loop feedback system is what ensures the models continue to improve with each run.</p><p>By using Gradient, data engineers at enterprise organizations have reported cost savings up to 50% on their compute bill, not to mention hundreds of engineering hours saved. In fact, a senior data engineer with experience at Disney, Meta, and Nielsen used Gradient to optimize his Spark jobs. Gradient was able to save his company over $100k on AWS a year!</p><p>Read this user’s blog post: <a href="https://medium.com/@matt_weingarten/user-perspective-how-sync-computing-can-optimize-cloud-spend-fe55c80c3298">User Perspective — How Sync Computing Can Optimize Cloud Spend</a></p><h3>Conclusion</h3><p>We hope this article has shed some light on optimizing the costs of EC2 in Databricks. Here’s a quick recap:</p><ol><li><strong>Understand cost factors:</strong> EC2 costs depend on various factors, including instance types, pricing models, and storage options. Choosing the right combination is key to efficient cost management.</li><li><strong>Align cluster configurations with business needs: </strong>Develop strategies to select cluster configurations that meet your workload requirements. For instance, use Spot Instances for workloads that can withstand interruptions to reduce costs without compromising performance.</li><li><strong>Leverage automation: </strong>AI Solutions like Gradient can help optimize resource allocations, especially at scale, driving up to 50% cost savings. With Gradient, your data engineers can focus on driving business value and innovation, rather than resource allocation.</li></ol><p>Optimizing resource management is a long-term commitment, and we’re here to support you on this journey.</p><p><a href="https://landing.synccomputing.com/free-consultation">Click here to schedule a free 30-min consultation</a> into your Databricks ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa67a02505ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/optimizing-ec2-costs-on-databricks-fa67a02505ab">Optimizing EC2 costs on Databricks</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding an AI agent to your data infrastructure in 2025]]></title>
            <link>https://medium.com/sync-computing/adding-an-ai-agent-to-your-data-infrastructure-in-2025-f163a48c668d?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/f163a48c668d</guid>
            <category><![CDATA[data-infrastructure]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 19:04:11 GMT</pubDate>
            <atom:updated>2025-01-16T19:04:10.991Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learn how Gradient reimagines compute management through ML-powered optimization.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UOSIGkZHVB9oGADt.png" /></figure><p>Imagine a world where you could simply tell your data infrastructure what you want it to achieve, rather than meticulously configuring every detail. This is precisely what Jeff Chou, Co-founder and CEO of Sync, discussed in the latest daily.dev webinar. This innovative concept is being made real through Gradient, the AI agent for data infrastructure from Sync.</p><p>Tune into Jeff below to learn how Gradient can help you save up to 50% on compute, or read on for key points.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKxfn9IxxAnw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKxfn9IxxAnw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKxfn9IxxAnw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/982367481fac5192185340b845b0746b/href">https://medium.com/media/982367481fac5192185340b845b0746b/href</a></iframe><h3>The resource allocation problem</h3><p>Cloud resource allocation is one of the toughest challenges at the heart of modern computing. These days, when a data engineer wants to submit a job on the cloud they need to specify the compute resources for that job. While that might not sound like a heavy lift, cluster configuration involves determining the instance type, cluster size, number of nodes, memory, and a bunch of other low-level configurations.</p><p>Manual cluster tuning is time-consuming and error-prone. It relies on a trial-and-error process of finding the right configuration to support the job. Once this is achieved, the configuration typically stays that way until the job is flagged for review, e.g. it begins to fail, is one of the more expensive jobs, or runtime/cost anomalies are detected.</p><p>Thankfully, there is a better way. One that also transforms data infrastructure from static to dynamic using <a href="https://synccomputing.com/manage-compute-clusters/">AI automation</a>. We call it “Declarative Computing.”</p><h3>Introducing Declarative Computing</h3><p>Declarative computing is a paradigm that enables data engineers to focus on business goals rather than on resource allocation. Instead of manually tuning clusters, engineers can specify the desired outcome — job runtime or cost, for example — and that will determine what the cluster looks like.</p><p>This approach simplifies the process for users, enabling teams to regain the engineering hours that would have otherwise been spent spinning up and tuning clusters. It also leads to more efficient resource utilization resulting in cost savings and improved efficiency.</p><p><strong>Learn more:</strong> <a href="https://synccomputing.com/what-is-declarative-computing">What </a><a href="https://synccomputing.com/what-is-declarative-computing">is Declarative Computing?</a></p><p>We built <a href="https://synccomputing.com/product">Gradient</a>, an AI agent for data infrastructure, based on the principles of Declarative Compute. It leverages advanced machine learning algorithms to choose the best cluster configurations for your data pipelines, based on your goals.</p><p>Simply declare your objectives and Gradient will automatically find the optimal configuration to <a href="https://synccomputing.com/optimize-compute-spend/">minimize costs</a>, <a href="https://synccomputing.com/control-runtimes-slas/">meet critical runtime SLAs</a>, or achieve target accuracy metrics for AI models. Gradient ensures your configuration is always optimal, by continuously monitoring and adapting to changes in your data pipelines.</p><h3>How Gradient works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*Jab2h0ib5sT1r6bI.png" /></figure><p>Gradient operates on a closed-loop feedback system, which is what enables it to automatically adapt to changes in production workloads. After each job run, cost and performance information are fed back into <a href="https://synccomputing.com/fine-tuning-models/">Gradient’s ML models</a>. Based on this data and cloud configuration information, the model adjusts its projections and recommendations for optimizations, until the optimal configuration is found for each job.</p><p>Gradient starts by making small changes to configuration and assessing their impact on both job costs and performance. Each optimization is based on the impact of the previous ones. Gradient continuously monitors data pipelines so that it can seamlessly adapt to changes or anomalies. As a result, it averages thousands of optimizations per workload every month, far beyond what any data engineer can do by hand. The AI agent enables a single engineer to optimize thousands of jobs.</p><h4>Key features and capabilities:</h4><p>1. <strong>Goal-based optimizations:</strong> By default, Gradient will optimize your jobs for minimal costs. But you can input your desired runtime SLA, and Gradient will configure your jobs to meet them at the lowest price.</p><p>2. <strong>Continuous improvements: </strong>Gradient seamlessly adapts to changing conditions, such as fluctuating data sizes, or sudden spikes in demand, ensuring your data pipelines are always as efficient and cost effective as possible.</p><p>3. <strong>Built for scale: </strong><a href="https://synccomputing.com/scale-compute/">Gradient excels at scale</a>, when you are managing hundreds of jobs in production. Start out with a co-pilot that recommends optimizations that an engineer can review and click to apply, or turn on autopilot mode in a click for optimization at scale.</p><h3>Real-world impact</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hclxbPuHjoxanYXp.png" /></figure><p>Organizations of different industries and backgrounds have achieved considerable cost savings with Gradient. In fact, across the millions of core-hours Gradient has managed for customers, we typically see cost savings in the <strong>30–60%</strong> range for production workloads.</p><p>You can see a couple of interesting data points in the graph above. This graph charts out the distribution of job costs customers have seen before and after implementing Gradient. Note that:</p><ol><li>The average job has seen cost savings</li><li>The customers with the most jobs saw the most significant cost savings. In fact, there is a 87% difference between the costs of the most expensive jobs before and after Gradient.</li></ol><p><strong>Related</strong>: <a href="https://synccomputing.com/databricks-compute-comparison-classic-serverless-and-sql-warehouses/">Databricks compute comparison: Classic Jobs vs Serverless Jobs vs DBSQL</a></p><h3>Addressing the serverless question</h3><p>Databricks launched its serverless compute offering last year and it has been promoting it as a cost efficient, managed solution. However, something to be aware of in general is that serverless offerings tend to come at a premium. It is the cost of convenience. You will no longer spend time configuring your data infrastructure that is managed by the data cloud.</p><p>In our <a href="https://synccomputing.com/databricks-compute-comparison-classic-serverless-and-sql-warehouses/">latest benchmark study</a>, we compared the performance of Gradient optimized classic clusters with those of Databricks serverless options. Some of our findings might surprise you:</p><ul><li>Optimized classic clusters were substantially cheaper than serverless alternatives</li><li>In some cases, cost savings of <strong>up to 5x</strong> were observed with classic clusters</li><li>Overall the cost of Gradient optimized Classic &lt; DBSQL &lt; Jobs serverless</li></ul><p>These findings highlight that while serverless has its place, companies operating at scale and prioritizing cost-efficiency may find significant value in Gradient’s optimization capabilities.</p><p>It’s also worth noting that serverless is still in its early days, so we expect costs to come down over time. That said, we still expect it will be a premium feature, used by companies that are willing to pay for the convenience of a managed infrastructure.</p><h3>Looking ahead: The future of AI in Infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*KfY4BYx7VsE1CcZV.png" /></figure><p>As we start the new year and look beyond, our targets are set on expanding Gradient. Expanding from Databricks to other data platforms like Snowflake, AWS EC2, Lambda, Kubernetes, and even GPUs in the cloud.</p><p>In the years to come, we hope to see more AI agents in infrastructure management. For the most part, infrastructure management is still mostly static. There is no reason to not leverage machine-learning and AI agents to make it dynamic. We hope to see more projects in this field.</p><h3>Conclusion</h3><p>Declarative computing, best embodied by Gradient, represents a massive leap forward in how we think about data infrastructure management. Letting engineers declare business outcomes rather than allocate resources, will unlock the next levels of efficiency and cost-effectiveness in cloud computing.</p><p>As we move into an era where AI agents become increasingly integrated into our infrastructure, Gradient will be integral to helping enterprises navigate the modern data ecosystem. The ability to automatically optimize and adapt to changing conditions will be key to staying competitive in a rapidly evolving technological landscape. Not to mention the ability to optimize at scale.</p><p>Interested in learning more about Gradient? <a href="https://landing.synccomputing.com/request-a-demo?_gl=1*5lcvo3*_gcl_au*MjAxNTM1ODk1Ni4xNzM0NTY2MzIz">Book your personalized demo!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f163a48c668d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/adding-an-ai-agent-to-your-data-infrastructure-in-2025-f163a48c668d">Adding an AI agent to your data infrastructure in 2025</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the Right Databricks Cluster: Spot Instances vs.]]></title>
            <link>https://medium.com/sync-computing/choosing-the-right-databricks-cluster-spot-instances-vs-cae5775cf026?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/cae5775cf026</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-infrastructure]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Tue, 17 Dec 2024 19:23:56 GMT</pubDate>
            <atom:updated>2024-12-17T19:23:56.263Z</atom:updated>
            <content:encoded><![CDATA[<h3>Choosing the Right Databricks Cluster: Spot Instances vs. On-demand Clusters, All-Purpose Compute vs. Jobs Compute</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MB685ppZr-5ALxed.png" /></figure><p>According to <a href="https://wwa.wavestone.com/app/uploads/2023/12/DataAI-ExecutiveLeadershipSurveyFinalAsset.pdf">Wavestone’s 2024 Data and AI Leadership Executive Survey</a>, about 82.2% of data and AI leaders report that their organizations are increasing investments in data and analytics. As companies increasingly rely on big data, the significance of efficient data processing solutions and <a href="https://synccomputing.com/manage-compute-clusters/">optimal configuration of clusters</a> become even more crucial.</p><p>Choosing the correct cluster type for your Databricks Jobs requires considering factors like workload, budget, and performance needs. In this post, we explore the types of clusters in Databricks, ideal use cases for each, and strategies for maximizing their efficiency while lowering the costs of your data infrastructure.</p><h3>What is a cluster?</h3><p>A cluster is a group of virtual machines (VMs), computation resources, and configurations that work together as a joint computing environment. In Databricks, these compute resources are used to run notebooks and data pipelines. Databricks leverages the customer’s existing cloud infrastructure. By integrating Databricks with a cloud provider, such as Azure or AWS, users are able to utilize their current cloud resources while leveraging Databricks’ powerful data processing capabilities.</p><p>Clusters are designed to process large amounts of data in parallel, improving both speed and planning. Each node in a cluster is a machine that handles a portion of the overall data processing task, allowing multiple tasks to run simultaneously so the workload completes faster. Some examples of data workloads these clusters execute include machine-learning jobs, ad-hoc analytics, streaming analytics, and ETL processes.</p><p>In Databricks, clusters can be modified to meet the specific needs of your data workloads. The right configuration, from the size of the cluster to the market (i.e. spot vs on-demand), can significantly affect performance, scalability, and cost efficiency.</p><p>Let’s dive into the various types of clusters, pricing options, and best practices to help you choose the best cluster configuration for your Databricks Jobs.</p><h3>Worker and driver nodes in Databricks</h3><p>In Databricks, clusters consist of two primary types of nodes: workers and a driver.</p><ul><li><strong>Worker nodes:</strong> These nodes are responsible for executing tasks and running Spark executors. They handle the majority of the data processing workload, distributing tasks in parallel to enhance performance. Each worker node runs one executor, which is essential for executing Spark jobs.</li><li><strong>Driver node:</strong> The driver is a special type of worker that coordinates the execution of tasks across the worker nodes. It manages the Spark context, schedules tasks, and collects results from the workers. While it performs similar functions to a worker, its unique role in managing the cluster’s operations sets it apart.</li></ul><p>Choosing the right configuration of worker and driver nodes can be complex due to the vast amount of options available. You also need to take into account the cluster size — more about that later. Lastly, factors such as workload requirements, performance needs, and cost considerations must be taken into account when selecting node types and sizes.</p><p>To ensure you are only paying for the resources you need, <a href="https://synccomputing.com/product/">Gradient</a> offers AI-powered cluster optimization. Gradient assists users in configuring clusters by providing <a href="https://synccomputing.com/compute-monitoring-alert/">insights and recommendations</a> for optimization based on specific workload characteristics. These recommendations can be reviewed and applied in a click, or <a href="https://synccomputing.com/scale-compute/">fully automated for scale</a>.</p><h3>Cluster attributes</h3><p>Each cluster has the following attributes:</p><ul><li><strong>Compute nodes:</strong> These are virtual machine instances that supply resources to the cluster (CPU, memory and storage). Nodes can be resized or added as needed between runs to support scaling.</li><li><strong>Databricks Runtime</strong>: A set of <a href="https://www.databricks.com/glossary/what-is-databricks-runtime#:~:text=Databricks%20Runtime%20is%20the%20set,security%20of%20big%20data%20analytics.">software artifacts</a> that act as an optimized version of Apache Spark, specifically designed for the Databricks platform. Databricks Runtime enhances job performance, scalability, security, and integration capabilities.</li><li><strong>Libraries:</strong> Users can install additional JARs, Python/R packages, Spark packages, and other dependencies for workloads running on the cluster. These are automatically accessible when the cluster starts.</li><li><strong>Init scripts:</strong> Custom scripts that run at the start of a cluster, used to set up dependencies, mount storage, download data, and perform other configuration tasks.</li><li><strong>Notebooks and dashboards:</strong> Data artifacts and visualizations created in Databricks, which can be associated with a cluster for convenient access. The cluster has the necessary permissions to interact with these resources.</li></ul><h3>Cluster market</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CNMsj-iFoZ-HMANb.png" /></figure><p>Generally, clusters can be accessed through two primary pricing models: Spot instances and on-demand clusters. The right choice depends largely on your workload requirements and your budget.</p><p>As a rule of thumb, we do not recommend running crucial production jobs on spot instances, as they might be interrupted or even terminated without notice.</p><h4>Spot instances</h4><p>Spot instances leverage unused capacity from cloud providers, such as AWS and <a href="https://synccomputing.com/everything-you-need-to-know-about-azure-databricks-pricing-2024/">Azure</a>. These instances are typically much cheaper than on-demand clusters,<a href="https://synccomputing.com/the-easy-and-comprehensive-guide-to-understanding-databricks-pricing-how-it-works-and-how-to-reduce-your-cost/"> saving up to 90% of the cost compared to on-demand pricing</a>. The trade-off is that the cloud provider can terminate these instances without notice, or pull away resources (workers) causing the jobs to stall or even fail.</p><p><strong>When to choose Spot instances:</strong></p><ul><li><strong>Fault-tolerant workloads:</strong> If your workload can tolerate interruptions, such as jobs that use checkpointing to restart from the last successful state, spot instances are an excellent, cost-effective option.</li><li><strong>Development and testing environments:</strong> Spot instances are ideal for non-production use cases where job interruption isn’t a critical issue.</li><li><strong>Non-time-sensitive data processing:</strong> If your tasks are not on strict deadlines and can handle delays, spot instances offer a more affordable solution.</li><li><strong>Batch processing:</strong> Tasks that process data in large, scheduled batches (such as ETL or data transformation tasks) can benefit from the lower cost of spot instances, provided they have built-in fault tolerance and are not time sensitive.</li></ul><p><strong>Best practices for Spot instances:</strong></p><ul><li><strong>Implement checkpointing:</strong> Ensure your workloads are designed to resume after an interruption by using checkpointing or other state-saving mechanisms.</li><li><strong>Use automatic retries:</strong> Databricks provides features to retry failed jobs, which is especially important for spot instances that may face termination.</li><li><strong>Monitor cluster health:</strong> Keep track of termination notifications, so you can take action if an instance is about to be reclaimed by the cloud provider.</li><li><strong>Set timeouts:</strong> Configure job timeout settings to prevent jobs from running indefinitely and using up unnecessary resources.</li></ul><h4>On-demand clusters</h4><p>On-demand clusters, on the other hand, provide guaranteed availability of resources and are ideal for mission-critical production workloads that cannot afford interruptions. With on-demand clusters, you pay for the resources you use, but you’re assured that the computing power will be available whenever you need it. These clusters are the most reliable option for production environments and time-sensitive tasks.</p><p><strong>When to choose on-demand clusters:</strong></p><ul><li><strong>Production workloads:</strong> On-demand clusters ensure your production workloads run smoothly with minimal downtime.</li><li><strong>Workloads with strict SLAs:</strong> On-demand clusters are ideal for workloads tied to runtime Service Level Agreements (SLAs). You can easily add resources to <a href="https://synccomputing.com/control-runtimes-slas/">meet your runtime SLAs</a>, the tricky part is to not over-provision your clusters with resources you do not need. Gradient solves this common issue.</li><li><strong>Real-time data processing:</strong> If you require real-time analytics or data processing with low-cold storage requirements, on-demand clusters are the best choice.</li><li><strong>Interactive workloads:</strong> For tasks such as data exploration, machine-learning model development, and collaborative work that require ongoing access to computational resources, on-demand clusters provide the stability and flexibility you need.</li><li><strong>Business-critical applications:</strong> If the data being processed is crucial for decision-making, such as financial transactions, customer analytics, or operations monitoring, on-demand clusters guarantee that the infrastructure is always available.</li></ul><p><strong>Best practices for on-demand clusters:</strong></p><ul><li><strong>Optimize resource utilization: </strong>Keep an eye on resource utilization to ensure you are using the appropriate instance types and cluster sizes for your workloads.</li><li><strong>Establish proper shutdown policies:</strong> Automatically shut down clusters when not in use to avoid incurring unnecessary costs.</li><li><strong>Leverage cluster pools: </strong>Cluster pools help reduce the time it takes to provision clusters by reusing existing resources, especially useful in environments with frequent cluster creation or single minute latency requirements on jobs.</li></ul><p><strong>Related case study:</strong><br>Learn how an <a href="https://synccomputing.com/adtech-saves-300-eng-hours-10k-compute-with-gradient/">AdTech company saved 300 eng hours and $10K with Gradient</a></p><h3>Cluster types</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SK4-eHyZvuz--t17.png" /></figure><p>Databricks cluster node types are designed for specific use cases, with the two main categories being All-Purpose Compute and Job Clusters.</p><h4>All-Purpose Compute clusters</h4><p>All-purpose compute (APC) clusters are designed to support interactive tasks, such as exploratory data analysis, development, and real-time model training. These clusters stay running until manually terminated, providing continuous access to computing resources for tasks that require flexibility and responsiveness. They can also be used as a shared resource, which makes them ideal for research and exploratory processes.</p><p><strong>Key features:</strong></p><ul><li><strong>Interactive development support:</strong> Ideal for data scientists and analysts who need to run notebooks and interact with data on the fly.</li><li><strong>Multi-user access:</strong> Multiple team members can collaborate on the same cluster, making it a great option for team-based work.</li><li><strong>Manual start/stop: </strong>You have full control over when the cluster is running, allowing you to start it for specific tasks and stop it when it is no longer needed.</li><li><strong>Notebook attachment:</strong> You can attach notebooks to an all-purpose cluster to run code interactively.</li></ul><p><strong>Ideal use cases:</strong></p><ul><li><strong>Data exploration: </strong>When you need to query data and visualize results interactively, all-purpose compute clusters provide the flexibility you need.</li><li><strong>Collaborative development: </strong>Teams of data scientists, analysts, and engineers can share a cluster for joint data analysis or model development.</li><li><strong>Prototyping and experimentation: </strong>For testing out new ideas, running experiments, or developing machine learning models, an APC cluster offers continuous access to compute power.</li></ul><h4>Job clusters</h4><p>Jobs compute clusters, unlike all-purpose compute clusters, are provisioned specifically to execute automated, batch workloads. Once the job is complete, the cluster is automatically terminated, making this type of cluster cost-effective for scheduled tasks and periodic data processing jobs. In fact, we found that <a href="https://synccomputing.com/databricks-introduction-guide/">APC clusters can cost up to 50% more than Job clusters</a>, for the same batch workload.</p><p><strong>Key features:</strong></p><ul><li><strong>Ephemeral clusters:</strong> These clusters are created on-demand for a specific job and automatically terminated once the job completes.</li><li><strong>Optimized for batch processing:</strong> Ideal for running tasks that don’t require interactive analysis, such as ETL processes, large-scale data transformations, or machine learning model training.</li><li><strong>Job-specific configurations:</strong> You can configure job clusters with the specific resources needed for each job, making them more efficient than APC clusters for repetitive tasks.</li></ul><p><strong>Ideal use cases:</strong></p><ul><li><strong>ETL pipelines: </strong>For processing and transforming large datasets in a scheduled manner, job compute clusters are an efficient and reliable choice.</li><li><strong>Model training: </strong>Job clusters can run training jobs for machine learning models without requiring continuous monitoring.</li><li><strong>Data processing jobs:</strong> Batch processing tasks that don’t require interactive analysis benefit from job clusters’ flexibility.</li><li><strong>Scheduled report generation: </strong>Automate the generation of reports and dashboards based on predefined schedules.</li></ul><h3>Cluster sizing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*30fYTHOEsCq65QmW.png" /></figure><p>Selecting the right cluster size is one of the most important aspects of configuring your Databricks cluster. Unfortunately, the data cloud doesn’t make it easy. There is a wide variety of cluster size options and combinations available. Some are optimized for storage, others for memory, compute, GPU acceleration, or general purpose.</p><p>The size of the cluster should depend on the complexity of the workload and the amount of data being processed. However, we’ve found that more often than not an engineer will spin up a cluster that is “large enough” to support a workload, prioritizing reliability over cost efficiency. The issue is that once the cluster is configured and the job starts running, the configuration will not likely be reviewed soon, unless the job is one of the expensive ones, or it starts to fail. This situation at scale could cost an organization dearly.</p><p>Getting a better understanding of the options available is the first step to combating the issue.</p><h4>Single-node clusters:</h4><p><strong>Best for: </strong>Testing, learning, and small-scale data processing<br><strong>Advantages: </strong>Cost-effective, easy to manage, and shared resources (allowing for significant increase in speed over distributed computing in cases where latency to shared resources matters). <br><strong>Limitations</strong>:<strong> </strong>Limited processing power, cannot handle large datasets or complex workloads, not designed for shared use.</p><h4>Multi-node clusters:</h4><p><strong>Small clusters (2–8 Nodes)</strong></p><p><strong>Best for: </strong>Development, testing, and small-scale data processing.<br><strong>Advantages:</strong> More affordable, easier to manage.<br><strong>Limitations: </strong>Limited processing power, and possible memory constraints.</p><p><strong>Medium clusters (8–32 Nodes)</strong></p><p><strong>Best for: </strong>Production workloads, and medium-sized datasets.<br><strong>Advantages:</strong> A good balance of performance and cost.<br><strong>Limitations:</strong> Higher operational costs compared to smaller clusters.</p><p><strong>Large clusters (32+ Nodes)</strong></p><p><strong>Best for: </strong>Big data processing and complex analytics.<br><strong>Advantages: </strong>High scalability and performance for large datasets and intensive workloads.<br><strong>Limitations: </strong>Higher cost, requires careful resource management to avoid inefficiencies.</p><h3>Advanced features</h3><p>Databricks offers a couple advanced features built to help users optimize their compute.</p><h4>Photon:</h4><p><a href="https://synccomputing.com/databricks-photon-and-graviton-instances-worth-it/">Photon</a> is Databricks’ next-generation query engine that improves workload performance by utilizing native vectorized execution. With Photon, you can significantly reduce query times and improve overall performance, however, it is more expensive, typically costing <strong>2x a DBU</strong>. Photon will probably be faster than non-photon, but whether or not it’s cheaper will depend on how much faster it is relative to the cost increase.</p><p><strong>Best practices for Photon:</strong></p><ul><li>A/B test your jobs with and without Photon enabled to compare performance and cost differences and make job-specific decisions.</li><li>Monitor and optimize your jobs’ configurations to ensure that you’re getting the most out of Photon.</li></ul><h4>Autoscaling:</h4><p><a href="https://synccomputing.com/is-databricks-autoscaling-cost-efficient/">Autoscaling</a> automatically adjusts the size of your cluster based on workload demand, ensuring that you have the resources to complete your job. But are you overprovisioning? This is something autoscaling doesn’t help with. We compared Gradient optimized job clusters to job clusters running autoscaling and found that Gradient outperformed autoscaling for cost by roughly 40%.</p><p><strong>Best practices for autoscaling:</strong></p><ul><li>A/B test your jobs with and without autoscalling enabled to identify jobs that are prime candidates for autoscaling.</li><li>Set min and max worker limits to prevent excessive scaling and ensure cost control. This typically overlooked recommendation is crucial. Think of it as your guardrails to limit overprovisioning.</li><li>Regularly review cluster performance to fine-tune auto-scaling settings and ensure optimal efficiency.</li></ul><h3>Comparison: APC clusters vs Job clusters</h3><p>All-purpose clusters are designed for flexibility and collaboration, making them ideal for interactive tasks such as data exploration and development. They remain active until manually terminated, allowing multiple users to work simultaneously on the same cluster. This setup is particularly beneficial for teams that require ongoing access to computational resources.</p><p>In contrast, Jobs Compute clusters are optimized for running automated, batch workloads. These clusters are ephemeral, created specifically for a job and terminated once the job is completed. This makes them a cost-effective choice for scheduled tasks like ETL processes and model training, where interactivity is not required.</p><p>By configuring resources according to job specifications, organizations can achieve greater efficiency and save costs. This is precisely what Gradient helps organizations with, automatically ensuring their data infrastructure is as efficient as possible.</p><h3>Conclusion</h3><p>Choosing the right cluster configuration for your data pipelines in Databricks is a critical decision that can impact the performance, cost, and reliability of your data infrastructure. By understanding the different cluster types, pricing models, and configuration options available, you can make informed decisions that line up with your organization’s needs. When configuring a cluster for Databricks, remember to consider your workload characteristics, business requirements, and technical constraints.</p><p>Whether you’re using spot clusters for cost efficiency, on-demand clusters for reliability, or optimizing your setup with advanced features like Photon and autoscaling, Databricks provides your with granular options to manage your data processing workloads. They might even be too granular for most users.</p><p>If you find the sheer volume of options confusing, or simply don’t have the time for the trial and error process of tuning a cluster to your needs, we can help. Gradient is a purpose-built compute optimization system. It uses advanced, self-improving algorithms to offer 100% custom cluster optimizations based on the unique characteristics of your data pipelines and requirements.</p><p>Interested in seeing Gradient in action?</p><p><a href="https://landing.synccomputing.com/request-a-demo">Book a time to learn more and chat about your data needs!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cae5775cf026" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/choosing-the-right-databricks-cluster-spot-instances-vs-cae5775cf026">Choosing the Right Databricks Cluster: Spot Instances vs.</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Gradient with Terraform]]></title>
            <link>https://medium.com/sync-computing/integrating-gradient-with-terraform-9ab08d14ffa5?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/9ab08d14ffa5</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Thu, 12 Dec 2024 22:34:53 GMT</pubDate>
            <atom:updated>2024-12-12T22:34:53.001Z</atom:updated>
            <content:encoded><![CDATA[<h3>Balancing Infrastructure as Code with AI-powered optimization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*twbtRKNB4cxYi9UB.png" /></figure><p>Infrastructure as Code (IaC) has revolutionized how we manage cloud resources, with Terraform emerging as the leading tool for this approach. The ability to define, version, and automate infrastructure through code has transformed operations from manual, error-prone processes into systematic, repeatable workflows. However, cluster configurations defined in Terraform are typically over-provisioned to guarantee performance and stability, leading to unnecessary costs. Optimizing these clusters to dynamically match workload requirements while meeting business constraints around cost and SLAs requires <a href="https://synccomputing.com/cluster-tuning-compute/">continuous, iterative cluster tuning</a>.</p><p>This creates an interesting challenge: how do we maintain the benefits of Infrastructure as Code while allowing for dynamic optimization? The key lies in carefully modifying <a href="https://www.hashicorp.com/blog/detecting-and-managing-drift-with-terraform">Terraform’s drift detection</a> capabilities.</p><h3>The challenge of manual cluster optimization</h3><p>Traditionally, teams face a common anti-pattern when managing Spark clusters: manual intervention in cluster configurations. This typically manifests as:</p><ul><li>Engineers making direct changes to cluster sizes based on performance observations</li><li>Emergency resource adjustments during peak workload periods</li><li>One-off modifications that never make it back into the infrastructure code</li><li>Inconsistent configurations across different environments</li></ul><p>These manual interventions create drift between the actual infrastructure state and what’s defined in code, leading to runaway costs, confusion, inconsistency, and potential failures during future deployments. Thankfully, Terraform’s drift detection can be used to counter this anti-pattern and reset infrastructure back to its original configuration.</p><h3>Enter Gradient: AI for your data infrastructure</h3><p>Gradient transforms manual cluster tuning into automated, code-driven workflows that aligns with Terraform best practices. Instead of relying on human judgment and manual adjustments, <a href="https://synccomputing.com/product/">Gradient</a> provides AI-powered infrastructure recommendations that continuously optimize Spark workload performance to meet cost and runtime requirements.</p><p>Lean how <a href="https://synccomputing.com/how-it-works/">Gradient continuously optimizes Spark jobs for costs and performance.</a></p><h3>Modifying Terraform for Gradient integration</h3><p>Here’s where things get interesting: While Terraform’s drift detection is designed to prevent unwanted infrastructure changes, this default behavior needs a simple tweak to work with Gradient. By default, Terraform will detect Gradient’s optimizations as drift and attempt to revert them to the original configuration — so we’ll use Terraform’s built-in lifecycle configuration to allow Gradient’s AI-powered optimizations to work their magic.</p><p>To avoid this situation, the drift detection needs to be adjusted. Follow these three key steps to integrate Gradient with Terraform:</p><ol><li><strong>Set up Databricks webhook notifications:</strong> Include Workspace and Job Configuration in your Terraform plan</li><li><strong>Tweak Terraform drift detection:</strong> Configure Terraform to ignore recommendation fields when detecting drift</li><li><strong>Automate compute optimization:</strong> Enabling Gradient to auto-apply AI-powered cluster optimizations directly to your Databricks Jobs</li></ol><h4>Setting up Databricks webhook notifications</h4><p>First, set up a webhook notification destination that Gradient will use to monitor job starts:</p><pre>resource &quot;databricks_notification_destination&quot; &quot;sync_webhook&quot; {<br>  display_name = &quot;Notification Destination&quot;<br>  config {<br>    generic_webhook {<br>      url      = &quot;https://example.com/webhook&quot;<br>      username = &quot;username&quot; # Optional<br>      password = &quot;password&quot; # Optional<br>    }<br>  }<br>}</pre><p>Next, configure your Databricks jobs to use this webhook:</p><pre>resource &quot;databricks_job&quot; &quot;example_job&quot; {<br>  name = &quot;example job&quot;<br>  # ... other configurations ...<br>  webhook_notifications {<br>    on_start {<br>      id = databricks_notification_destination.sync_webhook.id<br>    }<br>  }<br>}  </pre><h4>Managing Terraform drift with Gradient</h4><p>There are two approaches to handling drift detection with Gradient:</p><p><strong>Option 1: Ignore entire cluster configuration</strong></p><pre>resource &quot;databricks_cluster&quot; &quot;single_node&quot; {<br>  cluster_name            = &quot;Single Node&quot;<br>  spark_version          = data.databricks_spark_version.latest_lts.id<br>  node_type_id           = data.databricks_node_type.smallest.id<br>  autotermination_minutes = 20<br><br>  spark_conf = {<br>    &quot;spark.databricks.cluster.profile&quot; : &quot;singleNode&quot;<br>    &quot;spark.master&quot; : &quot;local[*]&quot;<br>  }<br><br>  custom_tags = {<br>    &quot;ResourceClass&quot; = &quot;SingleNode&quot;<br>  }<br><br>  lifecycle {<br>    ignore_changes = all<br>  }<br>}</pre><p><strong>Option 2: Ignore specific gradient-managed configurations</strong></p><pre>resource &quot;databricks_cluster&quot; &quot;example_cluster&quot; {<br>  cluster_name  = &quot;example-cluster&quot;<br>  spark_version = &quot;7.3.x-scala2.12&quot;<br>  node_type_id  = &quot;i3.xlarge&quot;<br>  num_workers   = 2<br><br>  custom_tags = {<br>    &quot;sync:project-id&quot; = &quot;&lt;insert-project-id&gt;&quot;<br>  }<br><br>  lifecycle {<br>    ignore_changes = [<br>      num_workers,<br>      node_type_id<br>      # Note: Gradient also modifies EBS volumes<br>    ]<br>  }<br>}</pre><h3>Automating compute optimization</h3><p>Gradient offers two approaches for applying AI-powered recommendations for compute optimization:</p><ol><li><strong>Autopilot/Auto-Apply Feature:</strong> Enable this through the “Edit settings” button in the Gradient project page. When enabled, AI-powered recommendations are automatically applied after each job run. Note that this feature only works with Databricks Workflows and is not available for <em>DatabricksSubmitRunOperator</em>, or the Databricks Jobs API 2.1.</li><li>Copilot/Programmatic Integration: Use the <a href="https://synccomputingcode.github.io/syncsparkpy/">Gradient Python library</a> to retrieve and apply recommendations:</li></ol><pre>sync.api.projects.get_latest_project_config_recommendation(project_id: str)</pre><h3>Best practices for Gradient-Terraform integration</h3><p>a) <strong>Choose the right drift management strategy</strong></p><ul><li>Consider using specific <em>ignore_changes</em> rather than ignoring all changes</li><li>Document which configurations are managed by Gradient’s AI optimization</li><li>Regularly review ignored configurations to ensure they align with Gradient’s current capabilities</li></ul><p>b)<strong> Webhook configuration management</strong></p><ul><li>Store webhook credentials securely</li><li>Monitor webhook health and connectivity</li><li>Implement proper error handling for webhook failures</li></ul><p>c) <strong>Version control</strong></p><ul><li>Track your baseline configurations in version control</li><li>Document Gradient’s optimization boundaries</li><li>Maintain a changelog of any manual overrides</li></ul><p>d) <strong>Monitoring and governance</strong></p><ul><li>Implement monitoring for both Terraform and Gradient changes</li><li>Set up alerts for unexpected configuration changes</li><li>Regularly audit optimization patterns</li></ul><p><strong>Related reading</strong>: <a href="https://synccomputing.com/gradient-decoding-roi-data-driven-dollars/">Gradient’s advanced monitoring and insights</a></p><h3>Conclusion</h3><p>Integrating Gradient with Terraform represents an evolution in infrastructure management — combining the reliability and repeatability of Infrastructure as Code with the intelligence of <a href="https://synccomputing.com/">AI-powered optimization</a>. By configuring Terraform to work alongside Gradient, teams can maintain the benefits of IaC while leveraging automated, AI-driven optimization for their Spark workloads.</p><p>Remember: the goal isn’t to abandon infrastructure governance, but to create a controlled space where AI-powered optimization can occur within well-defined boundaries. This approach ensures that your infrastructure remains both well-managed and optimally configured for performance.</p><p>Interested in learning more?</p><p><a href="https://landing.synccomputing.com/request-a-demo">Book a time to meet with the team to discuss your data needs!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ab08d14ffa5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/integrating-gradient-with-terraform-9ab08d14ffa5">Integrating Gradient with Terraform</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses]]></title>
            <link>https://medium.com/sync-computing/databricks-compute-comparison-classic-jobs-vs-serverless-jobs-vs-sql-warehouses-235f1d7eeac3?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/235f1d7eeac3</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[sqlwarehouse]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Tue, 10 Dec 2024 20:38:33 GMT</pubDate>
            <atom:updated>2024-12-11T15:13:38.063Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this blog we run the TPC-DI benchmark on 3 popular Databricks compute platforms to evaluate the cost and runtime performance.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5ga_WuZKM2gNA9fC" /></figure><p>Databricks is a quickly evolving platform with several compute options available for users, leaving many with a difficult choice. In this blog post, we look at three popular options for scheduled jobs using <a href="https://github.com/shannon-barrow/databricks-tpc-di">Databricks own TPC-DI benchmark</a> suite.</p><p>By the way, kudos to the Databricks team for creating such a fantastic test package. We highly encourage anybody here to use it for their own internal testing. Just to re-iterate, these workloads were not created by us, nor modified in any way. We’re big compute efficiency nerds here at <a href="https://synccomputing.com/">Sync</a>, so we appreciate all contributions in this space.</p><p>The goal of this blog post is to help readers understand the pros, cons, and performance tradeoffs of the various Databricks compute options, so readers can make the best choice for their workloads.</p><h3>The experiment</h3><p>At a high level, the <a href="https://www.tpc.org/tpcdi/">TPC-DI benchmark </a>is an industry standard benchmark for data integration, and mimics many real world workloads that users utilize in their jobs and workflows. Below is a screenshot of the DAG that is created for the entire benchmark:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SSwhkRqOvw3ylMXZ" /><figcaption>DAG of the TPC-DI benchmark</figcaption></figure><p>One of the main knobs to tune for the TPC-DI benchmark is the “Scale Factor” (SF), which changes the size of the data that is processed. The table below translates the SF to actual raw data size for comparison.</p><p>We focused on 3 scale factors:</p><ul><li>Scale Factor: 100 → 9.66 GB</li><li>Scale Factor: 1000 → 97 GB</li><li>Scale Factor: 5000 → 485 GB</li></ul><p>In this study, we’ll compare the running TPC-DI on 3 different Databricks compute products:</p><ol><li><strong>Classic Jobs Compute </strong>— User definable clusters. This is by far the most flexible infrastructure for users, although not all users want to deal with the level of options that are available. The infrastructure still runs inside the customer’s cloud environment.</li><li><strong>Serverless Jobs Compute </strong>— Databricks managed clusters ( also their newest product), where no knobs are available. It’s extremely easy to use, but does have a good number of restrictions at the moment. The infrastructure is run inside Databrick’s cloud environment.</li><li><strong>DBSQL Serverless</strong> — User selectable cluster sizes (e.g. small, medium, large), but that’s about it. Infrastructure runs inside Databrick’s cloud environment.</li></ol><p>For those who want to dive into the details, here’s the full table with all of the details of the experiments and raw numbers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*evpzJPTtWYVjv30B" /><figcaption>Experiment details and raw data</figcaption></figure><p>At a high level, these were some of the choices we made in regards to some of the settings:</p><p><strong>1) Classic Jobs compute </strong>— We only tuned the number of workers here, and used the same recommended instance types. We tuned the optimal cluster size and allowed a max runtime about 40% longer than the other compute platforms, which is consistent with the experience we see with users. In addition, we used all on-demand clusters.</p><p>2) <strong>Serverless Jobs Compute </strong>— We didn’t tune or change anything here, since there’s nothing to change!</p><p><strong>3) DBSQL Serverless</strong> — We used the recommended warehouse sizes from the notebook</p><h3>The results</h3><p>You can see the cost graph across all 9 experiments below. The cost represents the total cost of DBUs and cloud costs (AWS in this case). Several observations can be made:</p><ol><li><strong>Classic compute was cheapest </strong>— Using an optimized classic cluster was by far the cheapest option, which makes sense since it’s the choice with the most options and allows users to fine tune their compute to their needs. One note is we used on-demand clusters, so likely costs can be further improved with Spot nodes.</li><li><strong>DBSQL was much cheaper than Jobs Serverless</strong> — To our surprise DBSQL was consistently almost <strong>2x cheaper</strong> than a Jobs Serverless cluster!</li><li><strong>Jobs Serverless was the most expensive </strong>— The cost of jobs serverless was about <strong>5x more expensive</strong> than classic and about <strong>2x more expensive</strong> than DBSQL</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LY7N32Se9TN27xVU" /></figure><p>For completeness, here are the corresponding runtimes for all 9 experiments. As we mentioned above, we tuned the classic cluster to run slightly longer if it meant a cheaper cost. We tried to keep it to “reasonable” increases in runtime.</p><p>Overall, DBSQL and Jobs Serverless were about the same in terms of runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eBAtydhbowkEszqN" /></figure><h3>Discussions</h3><p><strong>How do I find the optimal Classic cluster?</strong></p><p>The big tradeoff with Classic clusters is you have to know what to pick. This is the dark art of Spark and experienced engineers. Usually people just do some trial and error analysis with a few different configs to get a rough idea of an optimal cluster size. They then typically only revisit the configuration once a quarter at best, while data pipelines change much more rapidly.</p><p>At Sync, we built a tool that automatically finds the best cluster for each of your jobs at scale. We call it <a href="https://synccomputing.com/product">Gradient</a>. If you’re running Databricks jobs at scale and just can’t manually tune all of those clusters to lower costs, feel free to check us out to see if Gradient can help improve your Databricks efficiency automatically.</p><p><strong>Why were DBSQL warehouse costs so much lower than Serverless Jobs costs?</strong></p><p>Their runtimes were the same, so that means that the DBU consumption rate of both are quite different.</p><p>In fact, the <a href="https://docs.databricks.com/en/resources/pricing.html">DBU rate of a large SQL warehouse</a> is 40 DBUs/hr. Since the runtimes were about the same but the end costs were almost 2x, we can roughly estimate that the jobs serverless DBU rate is about 80 DBUs/hr.</p><p>This is a pretty surprising finding (and is not the official DBU rate of Jobs Serverless), and is only based on our loose back of the envelope calculations. However, it does highlight the discrepancy between these different products and how the wrong choice can lead to exorbitant costs.</p><p><strong>How did the query level performance change?</strong></p><p>It’s interesting to break open the timeline view of the queries to see how the different compute platforms behave. Different queries ran at different runtimes with the different platforms.</p><p>One big difference is the “ingest_customermgmt” task was actually a “classic cluster” for all 3 (probably due to some limitations of serverless). This requires a spin up time of the classic cluster which delayed the serverless options. For the classic cluster, this step was very fast, since the whole compute is on classic.</p><p>The other observation is the distribution of the “long tasks” and whether or not there were dependencies. Meaning, you may have a long running task but it actually is in parallel with all other tasks so it’s fine that it’s slow. However, if one task is the core dependency for all other tasks (such as the “ingest_customermgmt” task), then a single long running task in your DAG can really hurt your performance.</p><p><strong>Classic Jobs</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-K2g6JiINFrFofsv" /><figcaption>Timeline graph of the TPC-DI benchmark run on classic compute clusters</figcaption></figure><p><strong>Serverless Jobs</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QmsIw-MxS7YRBgX7" /><figcaption>Timeline graph of the TPC-DI benchmark run on serverless jobs clusters</figcaption></figure><p><strong>DBSQL</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CZBahPYXfTi0AOL3" /><figcaption>Timeline graph of the TPC-DI benchmark run on DB SQL serverless clusters</figcaption></figure><h3>Conclusion</h3><p>Hopefully this detailed study of the TPC-DI benchmark on Classic Jobs vs Serverless Jobs vs SQL Warehouse helps communicate how different your performance will be on Databricks, depending on which compute platform you select. At a high level the main dependencies are:</p><ol><li>The size of your compute resource (e.g. number of workers, warehouse size)</li><li>The DBU rate of your compute resource (DBUs/hr)</li><li>Workload specific features (e.g. DAG dependencies)</li><li>Engineering time (e.g. Salary, opportunity cost)</li><li>Runtime requirements (e.g. SLA goals)</li></ol><p>If you’re at a company where your core backend is scaled on Databricks, then this level of optimization may be a critical step to help lower your total cost of goods sold (COGS).</p><p>However, if you’re at a company just exploring Databricks and costs aren’t a concern, then I recommend just sticking to serverless compute for the convenience.</p><p>If you’re interested in understanding what’s right for your company’s Databricks usage, feel free to <a href="https://landing.synccomputing.com/request-a-demo">book a time that works for you here</a>!</p><p>We’d love to chat.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=235f1d7eeac3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/databricks-compute-comparison-classic-jobs-vs-serverless-jobs-vs-sql-warehouses-235f1d7eeac3">Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sync Partners with Apex Systems]]></title>
            <link>https://medium.com/sync-computing/sync-partners-with-apex-systems-1d30f33daaf5?source=rss----f1db5e70d515---4</link>
            <guid isPermaLink="false">https://medium.com/p/1d30f33daaf5</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[partnerships]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Sync Computing]]></dc:creator>
            <pubDate>Thu, 21 Nov 2024 19:32:52 GMT</pubDate>
            <atom:updated>2024-11-21T19:32:51.919Z</atom:updated>
            <content:encoded><![CDATA[<h3>Powering the future of tech solutions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u9wmlcwiO0GLLjMr.png" /></figure><p>We’re thrilled to announce that Sync has partnered with Apex Systems, a leading global technology services provider with a presence in more than 70 markets across North America, Europe, and India. Apex is the go-to service provider for many modern enterprise solutions.</p><p>This collaboration is poised to boost our ability to develop our advanced AI-backed Databricks compute optimization solution, <a href="https://synccomputing.com/product">Gradient</a>. Combining Apex System’s industry experience with our cutting-edge tech will drive innovation that will propel the industry forward. Read on for all the details.</p><h3>Why Apex Systems?</h3><p><a href="https://www.apexsystems.com/">Apex Systems</a> is a powerhouse in the world of IT services and consulting. They bring to the table a wealth of industry expertise and a proven track record of delivering innovative solutions. Their deep understanding of market trends and vast top-tier technical talent pool perfectly complement Sync’s advanced technology.</p><p>In addition, as a Databricks partner, Apex Systems provides organizations with the guidance required to implement and extract insight from their data. They focus on providing cost-effective cluster design, modular development for reusability, and massively parallel processing to enhance performance for Databricks workloads. And they help customers with everything from migration to optimization to insights.</p><h3>Synergy between Apex Systems and Gradient</h3><p>Apex’s global pool of technical experts makes it easy to build bespoke data solutions to match your initial needs. Once you scale your data operations to hundreds of data pipelines or more, Gradient comes into play. It lets you optimize your workloads at scale, beyond the capabilities of any team of people. With Gradient, a single engineer can optimize thousands of workloads.</p><p>By partnering with Apex Systems, we are supporting high-touch bespoke projects with large scale optimization and management. Both of which are essential to handling the unique needs of today’s enterprise.</p><h3>Innovation that gets us excited</h3><p>Through our partnership, we are set to boost the capabilities of Gradient, in several key areas. These include:</p><p>– Optimization of ETL pipelines: Cut compute costs on your batch Databricks jobs with Gradient’s machine-learning powered compute optimization engine.</p><p>– Optimization of production ML pipelines: Gradient can help you reduce the cost per inference and boost inference speeds.</p><p>– Meeting runtime SLAs: If you are struggling to meet strict runtime SLAs Gradient can help ensure you meet your SLAs consistently, and at the lowest possible price.</p><h3>Looking ahead</h3><p>We, at Sync, remain committed to pushing the boundaries of what’s possible in the world of high velocity computing and data processing. This partnership with Apex Systems is a testament to that commitment.</p><p>In the world of tech, it’s not just about having the best tools; it’s about having the right minds to wield them. With Sync and Apex Systems joining forces, we’re assembling the dream team. We can’t wait to show you what we can build together!</p><h3>Conclusion</h3><p>This partnership isn’t just about combining resources — it’s about merging visions to create a powerhouse of innovation in the tech world. From compute optimization to cutting-edge AI solutions, we’re now better equipped than ever to tackle the complex challenges organizations face today.</p><p>With Sync’s advanced computing solutions and Apex’s industry expertise, we’re ready to push the boundaries of what’s possible in technology. The future of computing is here, and together, we’re poised to shape it.</p><p>If you’re interested in learning more about how we are revolutionizing the world of cloud optimization, <a href="https://landing.synccomputing.com/request-a-demo?_gl=1*hkspgm*_gcl_au*ODI5MDY1MDg0LjE3MjY3ODg0Mzc.">book a time to connect with the team here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d30f33daaf5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sync-computing/sync-partners-with-apex-systems-1d30f33daaf5">Sync Partners with Apex Systems</a> was originally published in <a href="https://medium.com/sync-computing">Sync Computing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>