<![CDATA[Stories by Rui Carvalho on Medium]]> https://medium.com/@ruimigueltcarvalho?source=rss-b9b2cde34283------2 https://cdn-images-1.medium.com/fit/c/150/150/1*r5_J03IXjmfxVIMU7oBASA.png Stories by Rui Carvalho on Medium https://medium.com/@ruimigueltcarvalho?source=rss-b9b2cde34283------2 Medium Sun, 15 Nov 2026 16:28:35 GMT <![CDATA[Running SQL on an External Database From Databricks. What Are Your Options?]]>

A practical guide to running DDL on Azure SQL from a Databricks notebook.

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/running-sql-on-an-external-database-from-databricks-what-are-your-options-2ef5808416a0?source=rss-b9b2cde34283------2 https://medium.com/p/2ef5808416a0 Fri, 13 Mar 2026 17:47:36 GMT 2026-03-13T17:47:53.792Z <![CDATA[Built an Automated Expiration Monitor for Databricks Tokens and Azure Key Vault]]>

How to make sure a pipeline never fails due to token and secrets expiration alerts via Microsoft Teams

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/built-an-automated-expiration-monitor-for-databricks-tokens-and-azure-key-vault-8feca7a81c91?source=rss-b9b2cde34283------2 https://medium.com/p/8feca7a81c91 Fri, 06 Mar 2026 09:39:20 GMT 2026-03-06T09:41:11.061Z <![CDATA[I Work a 9–5, Build Content and Study. Here Is How I Stay Balanced]]>

A simple system built over the years that helps me do more without paying the price.

Continue reading on Medium »

]]> https://medium.com/@ruimigueltcarvalho/i-work-a-9-5-build-content-and-study-here-is-how-i-stay-balanced-5f1894d9fbef?source=rss-b9b2cde34283------2 https://medium.com/p/5f1894d9fbef Fri, 27 Feb 2026 16:31:00 GMT 2026-02-27T17:53:19.152Z <![CDATA[How Autoloader Solved My File Ingestion Problem in Databricks]]>

How Databricks Autoloader handles incremental file ingestion from cloud storage to Delta Lake without custom tracking code

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/how-autoloader-solved-my-file-ingestion-problem-in-databricks-b57a8197c4dd?source=rss-b9b2cde34283------2 https://medium.com/p/b57a8197c4dd Thu, 19 Feb 2026 11:04:35 GMT 2026-02-19T11:06:47.063Z <![CDATA[How I Replaced 180 Lines of Schema Definition With a Single VARIANT Column]]>

Using Databricks VARIANT to handle unpredictable JSON from external APIs without constant schema maintenance

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/how-i-replaced-180-lines-of-schema-definition-with-a-single-variant-column-cfd4d6f78b89?source=rss-b9b2cde34283------2 https://medium.com/p/cfd4d6f78b89 Wed, 11 Feb 2026 10:20:38 GMT 2026-02-11T10:20:53.675Z <![CDATA[How Databricks Fixed the Materialized View Cost Problem]]>

New REFRESH POLICY and EXPLAIN commands give you control over incremental refresh behavior before surprise full table scans drain your…

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/how-databricks-fixed-the-materialized-view-cost-problem-e625e71280aa?source=rss-b9b2cde34283------2 https://medium.com/p/e625e71280aa Mon, 02 Feb 2026 20:51:10 GMT 2026-02-02T20:51:31.735Z <![CDATA[Databricks Materialized Views and why the Enzyme engine matters]]>

A deep technical guide to Databricks SQL materialized views, Enzyme refresh logic, incremental vs full recompute, performance trade offs…

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/databricks-materialized-views-and-why-the-enzyme-engine-matters-7c427c41cb16?source=rss-b9b2cde34283------2 https://medium.com/p/7c427c41cb16 Mon, 05 Jan 2026 10:31:01 GMT 2026-01-05T10:31:14.775Z <![CDATA[Spark Join Types Explained]]>

Learn Broadcast Hash Join, Sort Merge Join and Shuffle Hash Join with a simple mental model and real explain patterns to debug slow Spark…

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/spark-join-types-explained-dbb28dfe2db8?source=rss-b9b2cde34283------2 https://medium.com/p/dbb28dfe2db8 Fri, 26 Dec 2025 11:54:41 GMT 2025-12-26T11:55:06.582Z <![CDATA[Fixing Databricks Deployments When Python Modules Get Turned Into Notebooks]]>

Refactoring a Databricks deployment pipeline when Python modules started to break

Continue reading on The Data Therapy »

]]> https://medium.com/the-data-therapy/fixing-databricks-deployments-when-python-modules-get-turned-into-notebooks-6af58144cc5e?source=rss-b9b2cde34283------2 https://medium.com/p/6af58144cc5e Tue, 23 Dec 2025 13:52:27 GMT 2025-12-26T11:53:36.954Z <![CDATA[Databricks SQL INSERT REPLACE Explained—REPLACE USING and REPLACE ON for Delta Tables]]>

A cleaner way to refresh partitions and fix keys without writing long MERGE statements

Continue reading on Learning SQL »

]]> https://medium.com/learning-sql/databricks-sql-insert-replace-explained-replace-using-and-replace-on-for-delta-tables-49c5a2cb58f0?source=rss-b9b2cde34283------2 https://medium.com/p/49c5a2cb58f0 Tue, 16 Dec 2025 13:11:54 GMT 2025-12-16T13:37:14.930Z