InThe Data TherapybyRui Carvalho·Mar 13Running SQL on an External Database From Databricks. What Are Your Options?A practical guide to running DDL on Azure SQL from a Databricks notebook.A response icon1A response icon1
InThe Data TherapybyRui Carvalho·Mar 6Built an Automated Expiration Monitor for Databricks Tokens and Azure Key VaultHow to make sure a pipeline never fails due to token and secrets expiration alerts via Microsoft TeamsA response icon1A response icon1
Rui Carvalho·Feb 27I Work a 9–5, Build Content and Study. Here Is How I Stay BalancedA simple system built over the years that helps me do more without paying the price.
InThe Data TherapybyRui Carvalho·Feb 19How Autoloader Solved My File Ingestion Problem in DatabricksHow Databricks Autoloader handles incremental file ingestion from cloud storage to Delta Lake without custom tracking code
InThe Data TherapybyRui Carvalho·Feb 11How I Replaced 180 Lines of Schema Definition With a Single VARIANT ColumnUsing Databricks VARIANT to handle unpredictable JSON from external APIs without constant schema maintenance
InThe Data TherapybyRui Carvalho·Feb 2How Databricks Fixed the Materialized View Cost ProblemNew REFRESH POLICY and EXPLAIN commands give you control over incremental refresh behavior before surprise full table scans drain your…
InThe Data TherapybyRui Carvalho·Jan 5Databricks Materialized Views and why the Enzyme engine mattersA deep technical guide to Databricks SQL materialized views, Enzyme refresh logic, incremental vs full recompute, performance trade offs…
InThe Data TherapybyRui Carvalho·Dec 26, 2025Spark Join Types ExplainedLearn Broadcast Hash Join, Sort Merge Join and Shuffle Hash Join with a simple mental model and real explain patterns to debug slow Spark…
InThe Data TherapybyRui Carvalho·Dec 23, 2025Fixing Databricks Deployments When Python Modules Get Turned Into NotebooksRefactoring a Databricks deployment pipeline when Python modules started to break
InLearning SQLbyRui Carvalho·Dec 16, 2025Databricks SQL INSERT REPLACE Explained—REPLACE USING and REPLACE ON for Delta TablesA cleaner way to refresh partitions and fix keys without writing long MERGE statements
