<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Isi on Medium]]></title>
        <description><![CDATA[Stories by Isi on Medium]]></description>
        <link>https://medium.com/@isaiasgarcialatorre?source=rss-026cb7d33cbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cQhWWLeCMTdrdv3g7eMQAw.jpeg</url>
            <title>Stories by Isi on Medium</title>
            <link>https://medium.com/@isaiasgarcialatorre?source=rss-026cb7d33cbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:43:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@isaiasgarcialatorre/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spark Resource Consumption: A Practical Field Guide (Databricks-friendly)]]></title>
            <link>https://medium.com/@isaiasgarcialatorre/spark-resource-consumption-a-practical-field-guide-databricks-friendly-3460e3139d74?source=rss-026cb7d33cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/3460e3139d74</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[resources]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[guides-and-tutorials]]></category>
            <dc:creator><![CDATA[Isi]]></dc:creator>
            <pubDate>Sun, 26 Oct 2025 19:43:18 GMT</pubDate>
            <atom:updated>2025-10-26T19:43:18.676Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lt7GdmTItS5XZ9KS" /><figcaption>Featured image</figcaption></figure><h3>Understanding Jobs, Stages &amp; Tasks (Quick Primer)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-NLgf0u4ZFFnSu94e577IQ.png" /></figure><p>You can monitor performance hierarchically in:</p><p><strong>Spark UI → Jobs → Stages → Tasks</strong></p><p>(At the end I have left you a good reading recommendation so that you can explore it and learn.)</p><p>Most Spark jobs are limited by <strong>one dominant resource:</strong></p><ul><li><strong>CPU</strong></li><li><strong>Memory</strong></li><li><strong>I/O</strong></li></ul><p>Below is a concise guide to common operations, their dominant resource, why they behave that way, and how to optimize them. Where helpful, I add one-liners from the official Spark docs and Databricks features.</p><p>*<strong>Refresh</strong> <a href="https://spark.apache.org/docs/latest/cluster-overview.html">Spark components</a></p><h3>Quick mental model</h3><p><strong>CPU-bound</strong> → heavy compute (hashing, sorting, aggregations, serialization). Scale with cores and vectorized execution; avoid expensive UDFs.</p><p><strong>Memory-bound</strong> → operations that must keep large working sets (wide groupBy, big joins, caching). If memory doesn’t fit, Spark spills to disk.</p><p><strong>I/O-bound</strong> → time lost reading/writing storage or shuffling across the network (Parquet/Delta I/O, wide shuffles).</p><p><strong><em>Official context:</em></strong><em> </em><a href="https://spark.apache.org/docs/latest/tuning.html"><em>Spark’s tuning guide</em></a><em> emphasizes </em><strong><em>data serialization</em></strong><em> and </em><strong><em>memory tuning</em></strong><em> as primary levers; if data fits in memory, </em><strong><em>network/I/O</em></strong><em> often becomes the bottleneck.</em></p><h3>Reads &amp; Writes (Columnar Formats)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*75IUaziPigV8wH_kRkoipQ.png" /><figcaption>read,write</figcaption></figure><p>Read:</p><ul><li><a href="https://docs.databricks.com/aws/en/delta/tune-file-size#optimized-writes-for-delta-lake-on-databricks">Optimized writes for Delta Lake on Databricks</a></li><li><a href="https://docs.databricks.com/aws/en/delta/tune-file-size#autotune-file-size-based-on-table-size">Autotune file size based on table size</a></li><li><a href="https://docs.databricks.com/aws/en/optimizations/predictive-optimization#what-operations-does-predictive-optimization-run">Operations that helps on read</a></li></ul><h3>Partitioning &amp; Shuffles</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsrHV1yid0LYwIoue21hKg.png" /><figcaption>repartition,coalesce</figcaption></figure><p>Read:</p><ul><li><a href="https://docs.databricks.com/aws/en/tables/partitions#gsc.tab=0">Partitions</a></li></ul><h3>Column Transforms</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nczx_okggkX7opkGjWM-Rw.png" /><figcaption>select,withColumn,filter,where,drop,rename,distinct,dropDuplicates,explode,flatten</figcaption></figure><h3>Joins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aldDo_SjkOuK81teqBUOhA.png" /><figcaption>join</figcaption></figure><p>More info:</p><ul><li><a href="https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-join">Join Type</a></li><li><a href="https://oindrila-chakraborty88.medium.com/different-types-of-join-strategies-in-apache-spark-5c0066999d0d">Join Strategies</a></li><li><a href="https://docs.databricks.com/aws/en/optimizations/aqe#gsc.tab=0">Adaptative Query Execution (AQE)</a></li></ul><h3>Aggregations &amp; Group Operations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UrcGptb9CwgA6zxuQzNhLQ.png" /><figcaption>groupBy,pivot,count,sum,avg</figcaption></figure><h3>Sorting &amp; Layout</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gn09G3nRT7s0am0fYNM7wA.png" /><figcaption>orderBy,sort,partitionBy</figcaption></figure><h3>Actions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DYFe9LvsSJLReGd-nMhYrA.png" /><figcaption>count,collect,show,foreach,map</figcaption></figure><p><a href="https://dailydatabricks.tips/tips/Performance/NOOP.html#benefits-of-using-spark-write-noop">Explore write noop</a> to trigger job</p><h3>Caching, Persistence &amp; Spilling</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OifL4LKz_ntRtFB9kF9fsA.png" /></figure><p><em>Official note:</em> Spark recommends serialized storage (MEMORY_ONLY_SER) and <strong>Kryo</strong> serialization to reduce object overhead — often the biggest performance win.</p><h3>Diagnosing Bottlenecks (Spark UI)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*26gOEgfip8u-UAlFDVy8cA.png" /></figure><p>I recommend <a href="https://docs.databricks.com/aws/en/compute/troubleshooting/debugging-spark-ui">this full guide</a> in Databricks Docs about Spark UI</p><h3>Databricks-Specific Notes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*294JrUqMG369_egK48eJiw.png" /></figure><p>Read:</p><ul><li><a href="https://www.databricks.com/product/photon">Photon</a> <strong>(I will soon publish a detailed analysis )</strong></li><li><a href="https://docs.databricks.com/aws/en/delta/clustering">Liquid Clustering</a></li><li><a href="https://docs.databricks.com/aws/en/tables/managed#why-use-unity-catalog-managed-tables">Explore Managed tables</a></li></ul><h3>Final Note for Databricks Readers</h3><p>Focus on <strong>data layout</strong></p><p>Let <strong>AQE</strong> handle most runtime optimizations, and when something feels off, open <strong>Spark UI → SQL / Stages / Tasks</strong> to map the bottleneck to CPU, memory, or I/O.</p><p>Hope this helps, :)</p><p>Isi</p><p>You can also find me:</p><p><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/145555">Databricks Community</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3460e3139d74" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks: Serverless vs. Classic — Who Wins the Cost Sprint?]]></title>
            <link>https://medium.com/@isaiasgarcialatorre/databricks-serverless-vs-classic-who-wins-the-cost-sprint-dc2503cced53?source=rss-026cb7d33cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc2503cced53</guid>
            <category><![CDATA[warehouse]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[Isi]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 23:27:13 GMT</pubDate>
            <atom:updated>2025-06-30T23:30:00.741Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Databricks Cost Sprint: Serverless vs Classic — Who Wins the SQL Warehouse Race?</strong></h3><h3>Calculating the Cost of a SQL Warehouse</h3><p>Disclaimer: This is an example in AWS but could be applied to other clouds provider.</p><p>The total hourly cost of a SQL Warehouse in Databricks is based on three key factors:</p><p><strong><em>(Warehouse size)×(price per warehouse type)×(num of concurrent instances)</em></strong></p><h3>1. Warehouse Size and Type</h3><p>In <strong>Classic</strong> and <strong>Pro</strong> warehouse types, the number of <strong>DBUs (Databricks Units)</strong> assigned to each size is fixed and well-known. For example, a <strong>Size L</strong> always uses <strong>40 DBUs</strong>.</p><ul><li>The Databricks cost is calculated as:</li><li><strong>Number of DBUs × Price per DBU</strong>, depending on the warehouse type.</li><li>Additionally, the <strong>AWS cost is static</strong>, since Databricks always launches the same EC2 machines associated with that warehouse size.</li><li>For instance, <strong>Size L = 16 x i3.2xlarge + 1 x i3.8xlarge</strong>, regardless of the actual workload. (We will see later)</li></ul><p>In the case of <strong>Serverless</strong>, the number of DBUs assigned is the same as in Classic or Pro, but the <strong>price per DBU is different</strong>, because Databricks manages the infrastructure entirely.</p><p>There are <strong>no direct AWS costs</strong>, as the compute runs in Databricks’ own cloud account.</p><h3>2. Number of Concurrent Instances</h3><p>This value depends entirely on <strong>horizontal scaling</strong> — that is, the number of active clusters running in parallel to serve multiple concurrent queries.</p><ul><li>Although each instance always uses the same size and configuration, the <strong>total cost multiplies by the number of active instances</strong>.</li><li>Therefore, the total cost of a SQL Warehouse is the result of:</li><li><strong>(Cost per instance based on size and type) × (Number of parallel active instances)</strong></li></ul><h3>Summary Table — Clusters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C_97THPR0ERe3mZCEctZhg.png" /></figure><h3>What EC2 Instances Does Databricks Launch in AWS?</h3><p>Databricks provides a reference table that outlines the <strong>EC2 instance types</strong> it uses depending on the <strong>requested warehouse size</strong>. This applies specifically to <strong>Classic</strong> and <strong>Pro</strong> warehouses.</p><p>For these types, the compute resources are <strong>launched in the customer’s own AWS account</strong>, and the instance types remain <strong>constant regardless of workload</strong>. This allows for predictable performance and cost estimation.</p><blockquote><em>⚠️ In the case of </em><strong><em>Serverless</em></strong><em>, the underlying infrastructure is fully managed by Databricks, and the instance types may </em><strong><em>vary dynamically</em></strong><em> depending on </em><strong><em>availability</em></strong><em>, </em><strong><em>pricing</em></strong><em>, or </em><strong><em>internal optimization strategies</em></strong><em>. As a result, Databricks does not expose fixed instance mappings for Serverless.</em></blockquote><p><a href="https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior">SQL warehouse sizing, scaling, and queuing behavior | Databricks Documentation</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMpLCSX_WzDcwiq-Y6au9g.png" /><figcaption><strong>Warehouse Size L -&gt; 1 x i3.8xlarge + 16 x i3.2xlarge</strong></figcaption></figure><h3>Pricing Calculator</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZaMMNEYk6ypJtkmSWpq3Og.png" /></figure><h3>Cost Breakdown</h3><p><strong>Final price = Cluster size L (40 DBUs) × Number of hours (1 hour) × Price per DBU</strong></p><p>For <strong>Classic</strong> and <strong>Pro</strong> warehouses, you also need to <strong>add the cost of the AWS infrastructure</strong>, since the compute resources are launched in your own AWS account.</p><h4>Example — Cluster size L:</h4><ul><li><strong>Driver node</strong>: 1 × i3.8xlarge</li><li><strong>Worker nodes</strong>: 16 × i3.2xlarge</li></ul><p>So the total cost will include:</p><ul><li><strong>Databricks cost</strong>: Based on DBUs used × DBU rate (varies by warehouse type)</li><li><strong>AWS EC2 cost</strong>: Based on EC2 pricing for the instance types used</li></ul><h3>AWS Cost Breakdown</h3><p>In SQL Warehouse clusters, there are two types of <strong>Spot Instance Policies</strong>: <strong>cost optimized</strong> and <strong>reliability optimized</strong>. This determines whether the <strong>worker nodes</strong> run as <strong>spot instances</strong> (cost optimized) or <strong>on-demand instances</strong> (reliability optimized).</p><p>The <strong>driver node is always on-demand</strong>, regardless of the policy.</p><p><strong>In this case, we’ll analyze the cost optimized option (cheapest option)— but the same structure applies to both.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-FwinPM7MdKMs6-_QXHFhA.png" /><figcaption>Driver always on-demand</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*utogdEzFi0R0abDHPqyfAg.png" /><figcaption>workers spot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EvJZR2Gyi-qVVCEt3FNAw.png" /></figure><p>Now that we’ve established the infrastructure costs, let’s move on to the next key factor:</p><p><em>(Warehouse size)×(price per warehouse type)×</em><strong><em>(num of concurrent instances)</em></strong></p><h3><strong>How do we calculate the number of instances?</strong></h3><p><strong>Why is this variable so important?</strong></p><p>The issue is that, by simply looking at the cost comparison table above, someone might conclude that even if <strong>Serverless is more expensive than Classic </strong>is the most appealing option — and not without reason. It offers:</p><ul><li><strong>Very fast startup time</strong> (4–5 seconds)</li><li><strong>Support for Predictive I/O</strong>, improving read/write efficiency</li><li><strong>Intelligent Workload Management (IWM)</strong> support</li><li><strong>1-minute auto-termination(via API, default min 5 minutes)</strong> for idle clusters</li></ul><h3>How Autoscaling and Queuing Work in SQL Warehouses</h3><p>The way Databricks handles <strong>autoscaling and query queuing</strong> differs depending on whether you’re using <strong>Pro/Classic</strong> or <strong>Serverless</strong> SQL warehouses.</p><h4>Pro &amp; Classic Warehouses</h4><p>For Pro and Classic types, autoscaling is <strong>reactive</strong> and based on:</p><ul><li>Current running queries</li><li>Queries in queue</li><li>Expected incoming queries over the next 2 minutes</li></ul><p>Databricks estimates the total compute time needed. If the estimated time is:</p><ul><li><strong>&lt; 2 minutes</strong> → no scaling</li><li><strong>2–6 minutes</strong> → +1 cluster</li><li><strong>6–12 minutes</strong> → +2 clusters</li><li><strong>12–22 minutes</strong> → +3 clusters</li><li><strong>&gt; 22 minutes</strong> → +3 clusters +1 for every extra 15 mins of load</li></ul><p>Additionally, any query <strong>waiting more than 5 minutes</strong> in queue forces an upscale.</p><p>Warehouses <strong>downscale</strong> if load stays low for 15 minutes, keeping just enough clusters to support the <strong>peak concurrency</strong> seen in that time window.</p><h4>Serverless Warehouses with IWM</h4><p>Serverless warehouses use <strong>Intelligent Workload Management (IWM)</strong> to make autoscaling more responsive and cost-efficient.</p><ul><li>IWM uses <strong>machine learning</strong> to predict the compute cost of each incoming query and compares it with current available capacity.</li><li>If compute is available → the query runs immediately.</li><li>If not → the query is queued and triggers <strong>rapid autoscaling</strong> if the queue doesn’t drain quickly.</li></ul><p>Thanks to this <strong>predictive and real-time system</strong>, Serverless can:</p><ul><li>Scale up faster to reduce latency</li><li>Admit queries closer to hardware limits</li><li>Scale down quickly when demand drops</li></ul><p>In all cases, Databricks enforces a maximum queue size of <strong>1,000 queries</strong> across all warehouse types.</p><p><a href="https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior#queueing-and-autoscaling-for-pro-and-classic-sql-warehouses">https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior#queueing-and-autoscaling-for-pro-and-classic-sql-warehouses</a></p><h3>IWM (Intelligent Workload Management)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9VltQqG_1Sr_62jmDIXhQ.png" /></figure><p>Autoscaling in <strong>Serverless SQL Warehouses</strong> is highly efficient in terms of performance, but it can lead to <strong>less transparent cost behavior</strong>. If a warehouse is saturated with a high number of simultaneous queries, Databricks may scale out aggressively, launching multiple instances in parallel.</p><p>This can result in a <strong>significant increase in hourly costs</strong> without the user realizing it, since there is no explicit or immediate control over how many instances are active at any given moment.</p><h4>SO HOW WE CAN COMPARE THEM?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-SKaQVNEIUuo4r_9mG34Fw.png" /></figure><h4>Understanding “USD per Useful Minute”</h4><p>While baseline prices can give a rough estimate of hourly cost, they <strong>don’t reflect the real-world efficiency</strong> of your warehouse — especially for short or bursty workloads. That’s where <strong>USD per Useful Minute</strong> becomes a more meaningful metric.</p><p>This value considers the <strong>actual billed time</strong>, including overhead introduced by factors such as:</p><ul><li>Auto-termination delays (common in Classic/Pro)</li><li>Startup time</li><li>Idle time not reclaimed efficiently</li></ul><p><strong>USD per Useful Minute</strong> = (Hourly Price ÷ 60) × (1 work minute + overhead) ÷ 1 work minute</p><p>Where:</p><ul><li><strong>Hourly Price</strong> includes both Databricks DBU costs and AWS compute (if applicable)</li><li><strong>Overhead</strong> represents extra minutes charged after the job completes</li><li>The default case shown uses <strong>1 minute of actual work</strong>, ideal for testing short-job efficiency</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZ90NMGW2QBwKciG1tkoWw.png" /></figure><p>Below is a curve showing how <strong>USD per Useful Minute decreases</strong> as actual work time increases. This allows you to visually compare when it makes sense to use Serverless vs. Classic or Pro.</p><ul><li><strong>Red line</strong>: Classic</li><li><strong>Green line</strong>: Serverless</li><li><strong>Blue line</strong>: Pro</li><li>Dashed vertical line shows the <strong>equilibrium point</strong>, where Serverless and Classic cost the same per useful minute.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7AF_ytXsoUK6kRhSI3Pw1Q.png" /></figure><p>This metric is especially helpful when:</p><ul><li>Benchmarking cost/performance in DAGs, notebooks or scheduled jobs</li><li>Deciding when to switch to Serverless (e.g. for jobs &lt; 10 min)</li><li>Justifying cost optimization with data, not assumptions</li></ul><h3>How overhead fades with time — and when Serverless must be faster</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*5xGmBjXNkUdizazyEc-2eQ.png" /></figure><p>When a job is <strong>very short</strong>, the 10-minute auto-termination charge on Classic/Pro dominates the bill.</p><p>As the job lengthens, that fixed overhead is “diluted” over more useful minutes, so Classic/Pro get progressively cheaper <strong>per real minute</strong>.</p><p>Serverless adds only one extra minute, so its curve starts low and flattens quickly.</p><h4>Break-even speed table</h4><p><em>If a Classic job really lasts </em><strong><em>T minutes</em></strong><em>, the table shows how long Serverless (SL) could take to cost the </em><strong><em>same</em></strong><em>.</em></p><p><em>• If “Margin vs Classic” is </em><strong><em>positive</em></strong><em>, SL can even run </em><strong><em>slower</em></strong><em> and still be cheaper.</em></p><p><em>• If it’s </em><strong><em>negative</em></strong><em>, SL must finish </em><strong><em>that much faster</em></strong><em> to break even.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*TVXhLzjvRYI_LDRNZq5jMQ.png" /></figure><h3>Take-aways</h3><ul><li><strong>Sub-10-minute workloads</strong> → Serverless is cheaper even if it runs a bit slower.</li><li><strong>~10 minutes</strong> → Classic and Serverless cost the same; choose by performance or features.</li><li><strong>Longer jobs</strong> → Classic overtakes unless Serverless can deliver a <strong>30–60 % speed-up</strong>.</li><li><strong>Pro</strong> is always the most expensive per useful minute unless its extra performance is required.</li></ul><p><strong>In this use case </strong>this quick rule could be applied:</p><p><strong>“If the query runs under 10 minutes, default to Serverless; over 10 minutes, measure real runtime — Serverless must be roughly one-third faster to justify the cost.”</strong></p><p><strong>Now it’s up to you to calculate the cost and see if it’s worth it.</strong></p><p>Hope this helps you to decide, :)</p><p>Isi</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc2503cced53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks SQL Warehouse: una guía sobre costos, rendimiento y mejores prácticas]]></title>
            <link>https://medium.com/@isaiasgarcialatorre/databricks-sql-warehouse-una-gu%C3%ADa-sobre-costos-rendimiento-y-mejores-pr%C3%A1cticas-94e122ba1c84?source=rss-026cb7d33cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/94e122ba1c84</guid>
            <category><![CDATA[rendimiento]]></category>
            <category><![CDATA[databricks-sql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[guia]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Isi]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 12:08:21 GMT</pubDate>
            <atom:updated>2025-04-18T12:08:21.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*w6w0K2L18neqhAa2AeAmZQ.jpeg" /><figcaption><em>Thanos le enseña a su hija el valor de no gastar por encima del contrato de Databricks</em></figcaption></figure><p><strong>Los clusters SQL de Databricks</strong> ofrecen una plataforma escalable y optimizada para ejecutar consultas analíticas y gestionar grandes volúmenes de datos. Diseñados para combinar rendimiento, flexibilidad y facilidad de uso, estos clusters permiten a los equipos de datos trabajar de forma eficiente, ajustándose a las necesidades específicas de cada carga de trabajo, desde consultas ad hoc hasta pipelines complejas de análisis.</p><h3>Tipos de SQL Warehouses</h3><p>Databricks ofrece tres tipos principales de SQL Warehouses: <strong>Serverless</strong>, <strong>Pro</strong> y <strong>Classic</strong>, cada uno diseñado para diferentes necesidades de rendimiento, costos y características de escalabilidad. A continuación, se describen sus principales características y casos de uso.</p><h3>1. Serverless SQL Warehouses</h3><p>Los <strong>Serverless SQL Warehouses</strong> aprovechan la arquitectura sin servidor de Databricks para ofrecer el máximo rendimiento y escalabilidad con costos optimizados.</p><h4>Características Principales</h4><ul><li><strong>Tiempo de inicio rápido:</strong> Entre <strong>2 y 6 segundos</strong> típicamente.</li><li><strong>Escalado rápido hacia arriba:</strong> Adquiere más recursos de cómputo cuando es necesario para mantener baja latencia.</li><li><strong>Escalado rápido hacia abajo:</strong> Minimiza costos durante períodos de baja demanda.</li><li><strong>Admisión eficiente de consultas:</strong> Más cercana a las limitaciones del hardware que a las de la máquina virtual.</li><li>Compatible con <strong>Predictive I/O</strong>, lo que mejora la eficiencia de lectura y escritura.</li><li><strong>Soporta Intelligent Workload Management (IWM)</strong>.</li></ul><h4>Casos de Uso</h4><ul><li>Procesos <strong>ETL</strong>.</li><li>Inteligencia de negocios (<strong>BI</strong>).</li><li>Análisis exploratorio, especialmente con cargas variables o picos.</li></ul><h3>2. Pro SQL Warehouses</h3><p>Los <strong>Pro SQL Warehouses</strong> ofrecen capacidades avanzadas de rendimiento, pero con menos flexibilidad y velocidad de escalado en comparación con Serverless.</p><h4>Características Principales</h4><ul><li>Compatible con <strong>Predictive I/O</strong>, pero <strong>no soporta Intelligent Workload Management (IWM)</strong>.</li><li>La capa de cómputo reside en tu cuenta de AWS, no en la cuenta de Databricks.</li><li>Tiempo de inicio más lento: <strong>aproximadamente 4 minutos</strong>.</li><li>Escalado menos reactivo que Serverless.</li></ul><h4>Casos de Uso</h4><ul><li>Regiones donde los <strong>Serverless SQL Warehouses</strong> no están disponibles.</li></ul><p>Arquitecturas híbridas o federadas, como:</p><ul><li>Bases de datos conectadas directamente a tu red (cloud u on-premises).</li><li>Integración con servicios adicionales en tu red, como buses de eventos o sistemas on-premises.</li></ul><h3>3. Classic SQL Warehouses</h3><p>Los <strong>Classic SQL Warehouses</strong> son la opción básica para ejecutar consultas con características estándar de Databricks SQL.</p><h4>Características Principales</h4><ul><li><strong>No soporta Predictive I/O ni IWM</strong>.</li><li>La capa de cómputo reside en tu cuenta de AWS.</li><li>Ofrece un rendimiento de nivel de entrada en comparación con Serverless o Pro.</li><li>Tiempo de inicio lento: <strong>aproximadamente 4 minutos</strong>.</li><li>Escalado menos reactivo que Serverless.</li></ul><h4>Casos de Uso</h4><ul><li>Consultas interactivas para exploración de datos con requisitos de rendimiento básicos.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*G-f1ErPNR6aaQTvltia6QA.png" /></figure><h4>Resumen tipo de cluster</h4><p>La elección del tipo de SQL Warehouse depende de tus necesidades específicas:</p><ul><li><strong>Serverless:</strong> Mejor opción para escalabilidad rápida, cargas variables y bajo tiempo de respuesta.</li><li><strong>Pro:</strong> Útil en entornos híbridos o cuando Serverless no está disponible.</li><li><strong>Classic:</strong> Alternativa económica para análisis exploratorios y cargas predecibles.</li></ul><h3>Costes en SQL Warehouses de Databricks</h3><p>El costo de los SQL Warehouses en Databricks depende de múltiples factores: el <strong>tipo de warehouse</strong>, el <strong>tiempo de ejecución</strong> y las <strong>características habilitadas</strong>. Una correcta configuración permite equilibrar rendimiento y costos, maximizando la eficiencia.</p><h3>Factores que Impactan en el Costo</h3><h3>1. Tipo de SQL Warehouse</h3><h4><strong>Serverless SQL Warehouse:</strong></h4><ul><li><strong>Costo más alto por minuto</strong> debido a su rendimiento premium y capacidad de autoscaling rápido.</li><li>Ideal para cargas variables, consultas ad hoc y tareas críticas con alta concurrencia.</li></ul><h4><strong>Pro SQL Warehouse:</strong></h4><ul><li><strong>Costo intermedio</strong>, con buen rendimiento gracias Predictive I/O.</li><li>Adecuado para workloads moderadas y predecibles, o para arquitecturas híbridas.</li></ul><h4><strong>Classic SQL Warehouse:</strong></h4><ul><li><strong>Costo más bajo</strong>, pero limitado en rendimiento.</li><li>Recomendado para análisis exploratorios y cargas predecibles simples.</li></ul><h3>2. Tiempo de Ejecución</h3><p>Los warehouses facturan por tiempo activo. Configurar <strong>auto-terminación</strong> es clave:</p><ul><li><strong>Serverless:</strong> Configura tiempos cortos de inactividad (1–5 minutos).</li><li><strong>Pro/Classic:</strong> Configura tiempos moderados de auto-terminación (10 minutos).</li></ul><h3>3. Escalado y Concurrencia</h3><p><strong>Serverless:</strong></p><ul><li>Escala rápidamente según la demanda, lo que lo hace eficiente para cargas variables.</li><li>Sin embargo, puede ser costoso si las consultas son largas o el uso es constante.</li></ul><p><strong>Pro/Classic:</strong></p><ul><li>Escalan más lentamente, pero son más económicos para cargas constantes y predecibles.</li></ul><h3>4. Tamaño del Warehouse</h3><ul><li><strong>Escalado Vertical:</strong> Aumenta la capacidad de cada worker para consultas pesadas.</li><li><strong>Escalado Horizontal:</strong> Incrementa el número de workers para mejorar la concurrencia.</li></ul><p><strong>Recomendación:</strong></p><ul><li>Configura workers medianos (XXS XS S)para cargas generales.</li><li>Ajusta el número de workers según la concurrencia necesaria, evitando sobredimensionar.</li><li>Que tu primera opción sea siempre Classic es cierto que su tiempo de encendido es algo mayor, pero si los tiempos no son un problema su coste-rendimiento es siempre el mejor.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*Hy7Tf45JD1ROdq67GWaZ8g.png" /></figure><h3>Recomendaciones para Optimizar Costos</h3><h4><strong>Configura Auto-terminación:</strong></h4><p>Establece tiempos de auto-terminación bajos para minimizar costos innecesarios:</p><ul><li><strong>Serverless:</strong> 1–5 minutos.</li><li><strong>Pro/Classic:</strong> 10 minutos.</li></ul><h4><strong>Documentación y recursos:</strong></h4><p>Revisa la documentación disponible y los recursos de calculo de costes:</p><ul><li><a href="https://docs.databricks.com/en/compute/sql-warehouse/warehouse-behavior.html"><strong>Databricks SQL warehouse docs</strong></a></li><li><a href="https://www.databricks.com/product/pricing/product-pricing/instance-types"><strong>Cost calculator</strong></a></li></ul><h4><strong>Optimiza Consultas:</strong></h4><p>Diseña consultas eficientes para reducir tiempos de ejecución:</p><ul><li>Usa particiones para limitar el procesamiento de datos.</li><li>Evita operaciones costosas o innecesarias.</li></ul><h4><strong>Monitorea el Uso:</strong></h4><p>Utiliza métricas de Databricks para ajustar:</p><ul><li>El tamaño de los workers (escalado vertical).</li><li>El número de workers (escalado horizontal).</li></ul><h4><strong>Ajusta el Escalado según el Tipo de Warehouse:</strong></h4><ul><li><strong>Serverless:</strong> Aprovecha el autoscaling para cargas variables y mantén límites de workers adecuados.</li><li><strong>Pro/Classic:</strong> Ajusta manualmente los límites de escalado horizontal (por ejemplo, 2–6) según la concurrencia.</li></ul><p>¡Espero que esta publicación te haya sido útil! Guárdala en tus favoritos para futuras consultas y no te pierdas mis próximas publicaciones, tanto en inglés como en español. :)</p><p>Isi</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94e122ba1c84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks SQL Warehouses: A Complete Guide to Cost, Performance, and Best Practices]]></title>
            <link>https://medium.com/@isaiasgarcialatorre/databricks-sql-warehouses-a-complete-guide-to-cost-performance-and-best-practices-76feffd60306?source=rss-026cb7d33cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/76feffd60306</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-sql]]></category>
            <category><![CDATA[gui̇de]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[performance]]></category>
            <dc:creator><![CDATA[Isi]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 11:46:43 GMT</pubDate>
            <atom:updated>2025-04-18T12:09:25.058Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*w6w0K2L18neqhAa2AeAmZQ.jpeg" /><figcaption>Thanos teaches his daughter the value of not spending more than the Databricks contract</figcaption></figure><p>Databricks SQL clusters offer a scalable and optimized platform for executing analytical queries and managing large data volumes. Designed to combine performance, flexibility, and ease of use, these clusters allow data teams to efficiently adapt to specific workload requirements, from ad-hoc queries to complex analytical pipelines.</p><h3><strong>Types of SQL Warehouses</strong></h3><p>Databricks provides three main SQL Warehouse types: <strong>Serverless, Pro, and Classic</strong>, each designed to meet different performance, cost, and scalability needs.</p><h4><strong>1. Serverless SQL Warehouses</strong></h4><p>Serverless SQL Warehouses leverage Databricks’ serverless architecture for maximum performance and optimized costs.</p><p><strong>Key Features:</strong></p><ul><li>Fast startup: Typically 2–6 seconds.</li><li>Rapid autoscaling (up and down): Minimizes latency and cost during variable workloads.</li><li>Efficient query handling: Hardware-level limitations rather than VM constraints.</li><li>Predictive I/O supported: Enhanced read/write efficiency.</li><li>Supports Intelligent Workload Management</li></ul><p><strong>Use Cases:</strong></p><ul><li>ETL processes</li><li>Business Intelligence (BI)</li><li>Exploratory analysis with variable or peak loads</li></ul><h4><strong>2. Pro SQL Warehouses</strong></h4><p>Pro SQL Warehouses offer advanced performance capabilities with less scalability and responsiveness compared to Serverless.</p><p><strong>Key Features:</strong></p><ul><li>Supports Predictive I/O but lacks Intelligent Workload Management (IWM).</li><li>Compute layer resides in your AWS account.</li><li>Slower startup: Approximately 4 minutes.</li><li>Less reactive scaling compared to Serverless.</li></ul><p><strong>Use Cases:</strong></p><ul><li>Regions without Serverless availability.</li><li>Hybrid or federated architectures (databases directly connected to your network, event buses, on-premises systems).</li></ul><h4><strong>3. Classic SQL Warehouses</strong></h4><p>Classic SQL Warehouses are a basic solution with standard Databricks SQL features.</p><p><strong>Key Features:</strong></p><ul><li>Supports Photon but lacks Predictive I/O and IWM.</li><li>Compute layer resides in your AWS account.</li><li>Entry-level performance.</li><li>Slow startup: Approximately 4 minutes.</li><li>Moderate scaling.</li></ul><p><strong>Use Cases:</strong></p><ul><li>Interactive data exploration with basic performance requirements.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*WGx5TuTDm6SLOSeaJXYQQQ.png" /></figure><h3>Cost Considerations</h3><p>Cost depends on warehouse type, runtime, and enabled features. Proper configuration balances performance and cost effectively.</p><h3>Factors Affecting Cost</h3><h4><strong>1. SQL Warehouse Type</strong></h4><p><strong>Serverless SQL Warehouse:</strong></p><ul><li>Highest per-minute cost due to premium performance and rapid autoscaling capabilities.</li><li>Ideal for variable workloads, ad-hoc queries, and critical tasks with high concurrency.</li></ul><p><strong>Pro SQL Warehouse:</strong></p><ul><li>Intermediate cost, providing good performance thanks to Photon and Predictive I/O.</li><li>Suitable for moderate and predictable workloads or hybrid architectures.</li></ul><p><strong>Classic SQL Warehouse:</strong></p><ul><li>Lowest cost but limited performance capabilities.</li><li>Recommended for exploratory analysis and simple, predictable workloads.</li></ul><h4><strong>2. Execution Time</strong></h4><p>Warehouses are billed based on active runtime. Setting proper auto-termination times is crucial:</p><ul><li><strong>Serverless:</strong> Set short idle times (1–5 minutes).</li><li><strong>Pro/Classic:</strong> Set moderate auto-termination times (~10 minutes).</li></ul><h4><strong>3. Scaling and Concurrency</strong></h4><p><strong>Serverless:</strong></p><ul><li>Rapidly scales according to demand, making it efficient for variable workloads.</li><li>Can become expensive if queries are long-running or if usage is continuous.</li></ul><p><strong>Pro/Classic:</strong></p><ul><li>Slower scaling, but more cost-effective for constant and predictable workloads.</li></ul><h4>4. Warehouse Size</h4><ul><li><strong>Vertical Scaling:</strong> Increases worker capacity for heavy queries.</li><li><strong>Horizontal Scaling:</strong> Increases the number of workers to handle higher concurrency.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5HCdAoUTMsV8MGEgKJP7A.png" /></figure><h3><strong>My recommendation:</strong></h3><ul><li>Configure medium-sized (XXS XS S) for general workloads.</li><li>Adjust the number of workers according to required concurrency, avoiding unnecessary over-provisioning.</li><li><strong>Always prefer using Classic first before any other option. Even though it has a longer startup time, if execution time isn’t critical, it will always be the most cost-effective choice</strong></li></ul><h3>Recommendations for Cost Optimization</h3><h4>Configure Auto-Termination</h4><p>Set short auto-termination intervals to minimize unnecessary costs:</p><ul><li><strong>Serverless:</strong> 1–5 minutes.</li><li><strong>Pro/Classic:</strong> ~10 minutes.</li></ul><h4>Documentation and Resources</h4><p>Review available documentation and cost calculation resources.</p><ul><li><a href="https://docs.databricks.com/en/compute/sql-warehouse/warehouse-behavior.html">Databricks SQL Warehouse Documentation</a></li><li><a href="https://www.databricks.com/product/pricing/product-pricing/instance-types">Databricks Cost Calculator</a></li></ul><h4>Optimize Queries</h4><p>Design efficient queries to reduce execution times:</p><ul><li>Use partitions to limit data processing.</li><li>Avoid costly or unnecessary operations.</li></ul><h4>Monitor Usage</h4><p>Use Databricks metrics to adjust:</p><ul><li>Worker sizes (vertical scaling).</li><li>Number of workers (horizontal scaling).</li></ul><h4>Adjust Scaling According to Warehouse Type</h4><ul><li><strong>Serverless:</strong> Leverage autoscaling for variable workloads, setting appropriate worker limits.</li><li><strong>Pro/Classic:</strong> Manually adjust horizontal scaling limits (e.g., 2–6) according to concurrency requirements.</li></ul><p>I hope you found this post helpful! Feel free to bookmark it for future reference, and don’t miss my upcoming posts, written both in English and Spanish. :)</p><p>Isi</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76feffd60306" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>