<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fernando De Nitto on Medium]]></title>
        <description><![CDATA[Stories by Fernando De Nitto on Medium]]></description>
        <link>https://medium.com/@fernandodenitto?source=rss-70dd2ca04d3e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ntEpGwHu3EhgcfHMG4HANQ.png</url>
            <title>Stories by Fernando De Nitto on Medium</title>
            <link>https://medium.com/@fernandodenitto?source=rss-70dd2ca04d3e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 07:16:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fernandodenitto/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaining Lakeflow Jobs Across Teams Without Guessing at Timing]]></title>
            <link>https://fernandodenitto.medium.com/chaining-lakeflow-jobs-across-teams-without-guessing-at-timing-3ec7f9f167c3?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ec7f9f167c3</guid>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 18:32:02 GMT</pubDate>
            <atom:updated>2026-11-11T18:32:02.385Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cron buffers between two teams’ jobs fire too early or sit idle for nothing; data-driven triggers replace the guess with an actual signal.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uwFv1T7m9BdY6Ucc.png" /></figure><p>Two teams, one nightly chain. Team A’s job lands bronze and builds silver; team B’s job aggregates silver into gold and refreshes the dashboard finance opens at 8 a.m. Nobody on team B can see team A’s code, and nobody on team A is on call for team B’s dashboard. The two jobs are connected by exactly one thing: a cron schedule on team B’s job, set to a time someone once decided was “late enough.” That connection is weaker than it looks, and it is the default shape almost every multi-team pipeline on Databricks starts with.</p><h3>Why a schedule-with-a-buffer design breaks</h3><p>A [[jobs-triggers]] schedule answers “run at this time,” not “run once the data is actually there.” When team A’s job runs late — a slow vendor file, a retried task, a backfill — team B’s job fires on schedule anyway and aggregates a silver table that is still mid-write or simply stale. When team A’s job finishes early, team B’s buffer was wasted compute and a report that was ready sooner than anyone saw it. Padding the buffer further doesn’t fix either case; it only moves the two failure modes around.</p><p>The second thing this design hides is ownership. If team B’s job depends on team A’s job only by the clock, there is no object in the platform that says so. A [[jobs-task-dependencies]] graph shows task order inside one job, but it has no concept of another job entirely, let alone one in another workspace. The dependency lives in a wiki page, a Slack thread, or someone’s memory — which is exactly the kind of dependency that survives a reorg badly.</p><h3>Replacing the guess with a signal</h3><p>Lakeflow Jobs has had a way out of pure scheduling for a while: a <strong>table update trigger</strong>. Instead of starting on a cron tick, the downstream job watches one or more Unity Catalog tables and starts when they receive a commit — insert, merge, or delete. The watched objects can be managed Delta or Iceberg tables, external Delta tables, streaming tables, materialized views, or views and metric views built on top of supported tables; OpenSharing objects and system tables can be watched too. With more than one table you choose whether <em>any</em> updated table is enough to fire, or whether <em>all</em> of them have to have changed since the last run, which is exactly the “gold needs both bronze and silver” case.</p><pre>resources:<br>  jobs:<br>    gold_refresh:<br>      name: gold_refresh<br>      trigger:<br>        pause_status: UNPAUSED<br>        table_update:<br>          table_names:<br>            - main.silver.orders_clean<br>            - main.silver.customers_clean<br>          condition: ALL_UPDATED<br>          wait_after_last_change_seconds: 180<br>      tasks:<br>        - task_key: aggregate<br>          sql_task:<br>            warehouse_id: ${var.warehouse_id}<br>            file: { path: ./sql/gold_aggregate.sql }<br>        - task_key: refresh_dashboard<br>          depends_on: [{ task_key: aggregate }]<br>          dashboard_task:<br>            dashboard_id: ${var.dashboard_id}</pre><p>wait_after_last_change_seconds matters here precisely because team A&#39;s job is not one write, it&#39;s several: each table update resets the countdown, so team B&#39;s job starts once the whole batch has actually settled, not on the first partial commit. A separate min_time_between_triggers_seconds can also hold back a run even if the tables keep changing, if what you need is a floor on how often gold recomputes rather than a ceiling on latency.</p><p>As of November 2026, there’s a second option for the cross-team case specifically: a <strong>job completion trigger</strong>, currently in Beta, that starts a downstream job when one or more upstream jobs finish, including upstream jobs that live in a different workspace, as long as both workspaces are attached to the same Unity Catalog metastore. The dependency is configured on the downstream job, so team B still doesn’t need write access to team A’s job — only knowledge that it exists and the right to depend on its completion. Where the table update trigger reacts to a change in the data, the job completion trigger reacts to the upstream job’s own run outcome, which is the better fit when what you actually care about is “did team A’s job succeed,” not “did some table change” — a job can finish and change nothing, or a table can be touched by a process that never counted as “done.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lwdz1xYC7RQPibem.png" /></figure><p>Both of these sit alongside scheduled, file arrival, model update (Beta), and continuous triggers under the same mechanism, and Lakeflow Jobs now lets a single job carry up to five triggers at once during a Multiple Triggers Beta — a continuous trigger still can’t be combined with anything else, but a table update trigger and a scheduled backup trigger can run side by side on the same job, each paused and resumed independently. There’s also a Beta option to attach a SQL condition to a scheduled, table update, file arrival, model update, or job completion trigger, so the trigger firing doesn’t automatically mean the run proceeds if the condition isn’t met.</p><h3>What still goes wrong</h3><p>Swapping cron for a data-driven trigger removes one failure mode and introduces others that are easy to miss because they fail quietly.</p><p>The table update trigger reads each watched object’s metadata and change history under the job’s <strong>Run as</strong> identity, which needs SELECT on every table (plus USE CATALOG and USE SCHEMA on its catalog and schema), or SELECT on a view&#39;s underlying tables for a watched view. If that principal is missing a grant, the trigger simply fails to evaluate — there&#39;s no failed run to notice, because no run starts at all. The same applies to a watched view that depends on a table outside Unity Catalog, a federated table, or one that calls read_files directly: Databricks lets you create that trigger, but it will never fire, and the only warning is the absence of runs you expected.</p><p>Scale has a sharper edge too: a table update trigger can watch up to 25 tables, but only if file events are enabled on every one of them, including the underlying tables of any watched view — a view built on 26 tables simply can’t be used in a trigger, and two views that each depend on 14 tables already count as 28. Without file events enabled everywhere, the ceiling drops to 10, counted per physical table, not per object you selected in the UI.</p><p>The job completion trigger has its own version of a familiar trap. It’s easy to reach for it expecting the same control you get from a task’s run_if condition — ALL_SUCCESS, AT_LEAST_ONE_SUCCESS, ALL_DONE, and the rest, which decide whether one task inside a job starts based on its own upstream tasks. A job completion trigger operates one level up: it decides whether the downstream job&#39;s run starts at all, matched against the upstream jobs&#39; results, and it has nothing to say about which of the downstream job&#39;s own tasks then run or get retried — that&#39;s still run_if, inside the job it triggered. And where most triggers simply skip a run that arrives while the job is already at its concurrency limit, a job completion trigger instead holds onto that ready completion until a slot opens up — useful so a fast upstream run isn&#39;t dropped, but worth knowing before you assume &quot;skipped&quot; behavior applies everywhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GJjQuQVcdLUlG3Ri.png" /></figure><h3>Operating the handoff</h3><p>None of this removes the need to watch the downstream job once it’s wired to someone else’s job instead of a clock. A run’s detail view still has a graph, a timeline, and a list tab; when a chain breaks, the fastest read is the first task colored as failed in the graph — every gray task after it marked UPSTREAM_FAILED or UPSTREAM_CANCELED is a consequence of that one failure, not a separate problem to debug. The run history matrix for a job keeps 60 days of that picture, with duration bars that make a run quietly getting slower visible before it turns into a timeout — which is exactly what a duration-warning notification is for: an alert fired when a run exceeds an expected threshold, even on a run that eventually finishes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v2DL8BLmat9ZQSLm.png" /></figure><p>Sixty days isn’t enough for a quarterly view of how a cross-team chain behaves, so that’s where system.lakeflow.job_run_timeline and system.lakeflow.jobs earn their keep — a join across the account&#39;s full run history, not just one workspace&#39;s UI:</p><pre>SELECT<br>  j.name,<br>  COUNT(*)                                             AS runs_30d,<br>  ROUND(100.0 * AVG(t.result_state = &#39;FAILED&#39;), 1)     AS failure_rate_pct,<br>  ROUND(percentile(t.duration_minutes, 0.5), 1)        AS median_minutes<br>FROM (<br>  SELECT job_id, workspace_id, result_state,<br>         timestampdiff(SECOND, period_start_time, period_end_time) / 60.0 AS duration_minutes<br>  FROM system.lakeflow.job_run_timeline<br>  WHERE period_start_time &gt;= current_date() - INTERVAL 30 DAYS<br>    AND result_state IS NOT NULL<br>) t<br>JOIN system.lakeflow.jobs j<br>  ON j.job_id = t.job_id AND j.workspace_id = t.workspace_id<br>GROUP BY j.name<br>ORDER BY failure_rate_pct DESC;</pre><p>Run that across both teams’ jobs and the handoff stops being invisible: a rising failure rate or a creeping median duration on team A’s job shows up before team B’s dashboard is late, instead of after.</p><h3>For Monday</h3><p>Find the Lakeflow job in your own chain that is connected to another job purely by a cron schedule and a shared assumption about timing. Check whether what it actually needs is a [[jobs-triggers]] table update trigger on the tables it reads, or — if the dependency is really “did that other job succeed,” possibly in another workspace — the new job-completion trigger, keeping in mind it’s still Beta. Either way, grant the job’s Run as principal exactly the SELECT it needs on the watched objects before you flip the trigger on, and set up a failure notification on the trigger itself, because a trigger that silently stops evaluating looks identical to a quiet night.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ec7f9f167c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keeping up with Databricks had become a second job, so I built a changelog for it]]></title>
            <link>https://fernandodenitto.medium.com/keeping-up-with-databricks-had-become-a-second-job-so-i-built-a-changelog-for-it-3f6f273f751c?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f6f273f751c</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 13:12:38 GMT</pubDate>
            <atom:updated>2026-09-20T13:12:38.172Z</atom:updated>
            <content:encoded><![CDATA[<h4>Previews, end-of-support dates, renames and the parts worth learning properly — in one place, for people who would rather spend the day engineering.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rDfHpLhZifQKS8gl9VhbWg.png" /></figure><p>I am a data engineer. I work on Databricks every day, and for most of the last two years the hardest part of that job had nothing to do with engineering.</p><p>It was keeping up.</p><p>Release notes land every week. Services change name. Runtimes reach end of support on dates nobody announces twice. A preview ships that would remove a workaround I have been maintaining for a year, and I find out about it six months later, from someone’s post in my feed, by which point it has been generally available for a quarter.</p><h3>The product is not the problem. The pace is.</h3><p>Every hour spent reading announcements is an hour not spent on the thing I am paid for: designing pipelines that hold up, keeping costs sane, making data people can trust. But not reading them has its own price. You fall behind quietly. You keep solving problems the platform solved for you last spring, and you plan migrations three weeks before the date instead of six months before it.</p><p>I did not want to pick one. The information exists — it is in release notes, docs, exam guides, a dozen blogs — it is just scattered, written in marketing tense, and impossible to scan for the only question that matters on a Monday morning: does this affect me, and do I need to do something about it?</p><p>So I built the thing I was missing. It is called Lakenaut. It is open, it is free, and it exists because I needed it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBFTUgyXhNNNZVI0MESevQ.png" /></figure><h3>Five things, one place</h3><p><strong>The preview radar.</strong> What is coming and how far along it is: private preview, public preview, beta. This is the part I use most, and the one that took the longest to explain to people. It is not a list of announcements. It is a list of things to think about now. Every team carries two or three problems with no good answer today; often the answer already exists and is one stage away from being supported. Knowing that changes what you choose to rewrite this quarter, and which workaround finally gets a visible end date.</p><p><strong>End of support.</strong> Every date on one page, with the supported replacement next to it. The upgrade is almost never the hard part. The hard part is finding out three weeks before the date, when the work collides with everything else that quarter — and then discovering that the job nobody owns is still pinned to the old runtime. Six months of notice turns that from a project into an afternoon of planning.</p><p><strong>Renames.</strong> Old name, new name, the date it changed. The cost of a rename is never in the product; it is in everything written before it. Your runbooks, your onboarding docs, the internal wiki, the Stack Overflow answer that actually solved your problem, the course you paid for last year. When you hit a name that no longer exists, you should not have to reconstruct a timeline to know what to search for.</p><p><strong>News.</strong> What changed, who it affects, and whether you need to act. Often the answer is no, and it says so. A changelog that only ever tells you something is exciting is a changelog you stop reading.</p><p><strong>Learning paths.</strong> Because the platform alone does not do the job — how you use it is what makes the difference, and that part is not in the release notes. There are eight paths: data engineering, SQL analytics, machine learning, generative AI, governance and security, platform administration, and the foundations underneath them. Each one is broken into stages — data engineering goes ingest, transform, declare, orchestrate, operate, ship — and each stage is a handful of short concepts rather than a course. Where a path lines up with a certification, it says which one.</p><p>There are over 170 concepts written so far, across 28 areas of the platform. No account, nothing to buy. If you would rather read offline, the whole thing downloads as an Obsidian vault.</p><h3>How it stays current without becoming a second job of its own</h3><p>A site like this is worthless the moment it goes stale, and I was not going to hand-maintain it. So the tedious half is automated and the judgement half is not.</p><p>An agent watches the release notes, the documentation behind each concept and the official exam guides. It may only read sources that are registered and switched on, and a duplicate ledger stops the same story being written twice. What it produces is a proposal: a draft news item, a flag that a page is now out of date, a suggested edit sitting in a review queue. A person approves it. Two jobs keep the slow-moving parts honest — one re-checks preview stages, one watches the end-of-support dates go past — and one rewrites old product names in the prose when a name moves.</p><p>The machine does the reading. The decisions stay human. That is the whole arrangement, and it is why I trust the dates on the page enough to plan my own quarter around them.</p><h3>What it is not</h3><p>It is not vendor announcements rewritten as enthusiasm. It is not a course funnel. It is not affiliated with Databricks in any way — the product names belong to their owners, and this is one engineer’s attempt to keep a record of a platform that moves faster than the people using it.</p><p>It is also not finished. The radar and the lifecycle dates are the parts I lean on daily; the paths are filling in as I write them.</p><h3>If you work with Databricks</h3><p>Start with the radar and the end-of-support page. Those two answer the Monday question in about a minute, which was the entire point. There is a weekly newsletter if you would rather it came to you than went looking for it.</p><p>It is at <strong>lakenaut.dev</strong>. If something on it is wrong, or if there is a date I am not tracking, tell me — the report link is on every page, and that is the part I most want to hear about.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f6f273f751c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How a get Five AWS Certifications in Eight Weeks — and the Slow Eight Months That Made It Possible…]]></title>
            <link>https://fernandodenitto.medium.com/how-a-get-five-aws-certifications-in-eight-weeks-and-the-slow-eight-months-that-made-it-possible-221c6d9ef008?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/221c6d9ef008</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Wed, 02 Sep 2026 17:29:40 GMT</pubDate>
            <atom:updated>2026-09-02T17:29:40.735Z</atom:updated>
            <content:encoded><![CDATA[<h3>How a get Five AWS Certifications in Eight Weeks — and the Slow Eight Months That Made It Possible (plus a gift for you!)</h3><h4>The first one I passed was retired days later. The entry-level one turned out to be the hardest. By December I was sitting three exams a month, and nothing about me had changed except how I read a question and how much I no longer had to study.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HPHbrqo1DH12Vka1OkWmPw.png" /></figure><p>The first AWS certification I ever passed was retired days after I passed it.</p><p>AWS Certified Data Analytics — Specialty went off the shelf on April 9, 2024. My badge is dated April 2024. I got in through a closing door, and not because of any career strategy: the consultancy I was working for at the time was an AWS partner, partner tiers run on certified headcount, and someone had to sit that exam. I was already building on those services every day, so it made sense for it to be me.</p><p>That is the entire strategic depth of my first certification: an internal requirement and a bit of convenience.</p><p>Nine months later I had mew seven AWS certifications. Five of them were passed in November and December alone, three of those in December. Same person, same job, same two hours a night after work.</p><p>The interesting part of this story is not the number. It’s the shape of the curve — why the first exams were slow and expensive and the last ones came weeks apart. That gap is where all the useful information lives, and it has almost nothing to do with working harder.</p><p>And if you’re only here for practice material: the exams I built are further down and they’re free. But the section on how these questions are actually decided is the part that gets you the pass.</p><h3>The actual timeline</h3><p>Here it is, with dates, because the dates are the argument.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbFHnBYHgpUN9Mku9HXnsw.png" /><figcaption>My AWS Certifications Timeline in 2024</figcaption></figure><p>One exam in April. One in September. Five in the last eight weeks of the year.</p><p>If I’d been getting progressively smarter, the curve would be smooth. It isn’t smooth — it’s a hockey stick, and the elbow sits somewhere around November. Everything below is an attempt to explain that elbow.</p><h3>I walked the staircase backwards, and it turned out to be the right accident</h3><p>The recommended path is a ladder: Cloud Practitioner, then Associate, then Professional or Specialty. I started at the top and worked down.</p><p>Specialty first, in April. Then, in September, the Data Engineer — Associate — which is, with a certain irony, the exam AWS created to replace the one I’d passed five months earlier. Only in November did I go back and sit the foundational exam, the Cloud Practitioner, expecting a formality.</p><p>It wasn’t. In terms of study effort and method, I found it harder to prepare for than the specialty.</p><p>The reason is structural, and it’s worth understanding because it predicts which exams will hurt <em>you</em>.</p><p>A specialty exam is deep and narrow. Data Analytics covered a defined list of services — Kinesis, Glue, Athena, EMR, Redshift, QuickSight, Lake Formation — and went deep into each. There’s a fixed perimeter. You can walk it, service by service, until nothing inside it is unfamiliar.</p><p>The foundational exam is the opposite: shallow and enormous. It touches the entire catalogue, plus billing models, support plans, the shared responsibility model, pricing mechanics, the Well-Architected pillars, and a long tail of services you will genuinely never use. Nothing in it is difficult. Everything is in it.</p><p>My study method is meticulous. I don’t move on from a service until I understand what it does, when you’d choose it, and what it costs you. That method scales beautifully with depth and terribly with breadth. The specialty rewarded exactly how I work; the practitioner punished it, because “understand everything properly” stops being a viable strategy when the perimeter is the whole of AWS.</p><p>The lesson generalises well past AWS: the difficulty of a certification is not a property of the exam. It’s the interaction between the exam’s shape and your study method. The exam everyone calls easy may be the one that costs you the most evenings.</p><h3>Theory builds the map. It doesn’t pass the exam.</h3><p>Two exams from opposite ends of the difficulty scale were enough to stop improvising.</p><p>The first thing that became clear is what theory is actually for. If you use AWS at work, you use the fifteen services your projects need. You get very good at those and structurally blind to everything else. You don’t know what you don’t know, which is how people end up writing a Lambda to do something a managed feature has done natively for two years.</p><p>Studying for a certification forces the other hundred and fifty services in front of you. For someone already working in the field, that’s the real return — not the badge, the removal of blind spots. I started using features I didn’t know existed purely because an exam put them in front of me.</p><p>But knowing the services and answering the questions are two different skills, and that distinction is what costs people a retake. Theory gets you the map. It does not get you the pass. Most people who fail an AWS exam know the material. They lose on how the questions are built.</p><h3>The resources I actually used</h3><p>No padding. This is the short list.</p><p><strong>Video courses.</strong> I studied mainly with Stephane Maarek’s courses on Udemy. They’re structured, kept current, and they don’t spend forty minutes on “what is cloud computing” before getting to the point — which matters enormously when your study window is a couple of hours after dinner.</p><ul><li><a href="https://trk.udemy.com/cb-aws-certified-cloud-practitioner">Cloud Practitioner</a></li><li><a href="https://trk.udemy.com/cb-aws-certified-solutions-architect-associate-saa-c03-2">Solutions Architect — Associate</a></li><li><a href="https://trk.udemy.com/cb-aws-certified-data-engineer-associate-dea-c01">Data Engineer — Associate</a></li><li><a href="https://trk.udemy.com/X4vyqo">Machine Learning Engineer</a></li><li><a href="http://trk.udemy.com/aN9RkW">AI Practitioner</a></li></ul><p><strong>Practice questions.</strong> I did a very large volume of them on <a href="https://certbase.pro/">certbase.pro</a>. <br>This is where preparation actually turns into a pass, for reasons I’ll get to.</p><p><strong>The official exam guide and the AWS documentation.</strong> Whenever a course or a practice question left me with a doubt, the tiebreaker was the guide and the docs — not a blog post, not an answer from 2021. Exam questions are written against the product’s current behaviour, and AWS moves faster than third-party material.</p><p><strong>Daily hands-on.</strong> I had an advantage here and I won’t pretend otherwise: I was building on AWS every day. If you aren’t, replace those missing hours with a free-tier account and small things you break on purpose. Reading about S3 lifecycle policies and watching an object slide into Glacier while you can no longer query it are different memories.</p><h3>The mindset switch, which is where these exams are actually won</h3><p>Here’s the thing I’d tell anyone preparing for an AWS certification, and the reason this article exists.</p><p>AWS questions are not knowledge questions. They’re decision questions. You get a scenario, four options, and usually two or three of them are technically valid — they’d all produce the required outcome. Only one is right <em>for that scenario</em>.</p><p>So the most useful habit I built is this: <strong>never start by looking for the right answer.</strong></p><h3>First pass — eliminate what cannot be right</h3><p>Read the options hunting for what’s clearly wrong. There is almost always an option built on a service that doesn’t do what’s being asked at all: the wrong category of tool, something that solves an adjacent problem, a name that only looks plausible if you half-recognise it. Kill those immediately, before thinking about the scenario in any detail.</p><p>That usually takes four options down to two. The exam is always decided between those two, and the difference is almost never conceptual — it’s operational.</p><h3>Second pass — go back to the question and find the constraints</h3><p>This is where AWS hands you half the answer, and where most people lose points by reading too fast.</p><p>Every scenario contains two or three words that decide the outcome. They’re rarely at the end, where you’d look for them; they’re buried mid-paragraph. Some of the ones I learned to mark before even reading the options:</p><ul><li><strong>“with the least operational overhead”</strong> — the answer is managed or serverless. Anything you’d have to patch, scale or babysit is out, even if it’s the more powerful design.</li><li><strong>“with minimal development effort”</strong> — the answer is a built-in feature, not code you write. If one option is a native capability and another is a Lambda you’d have to build and maintain, the question has already told you which one it wants.</li><li><strong>“without changing the existing architecture”</strong> / <strong>“the team must keep using X”</strong> — the answer bolts on, it doesn’t replace. Any option starting with “re-platform to…” is dead on arrival, however much better it would be.</li><li><strong>“most cost-effective”</strong> — usually the option that avoids provisioning capacity for a peak you hit twice a day.</li><li><strong>“near real time”</strong> versus <strong>“the report is read every morning”</strong> — the difference between a streaming answer and a batch answer, stated in one clause.</li><li><strong>“the analysts only know SQL”</strong>, <strong>“the team has no ML expertise”</strong>, <strong>“data can arrive up to three days late”</strong> — each of these eliminates an entire family of answers on its own.</li></ul><p>When two constraints show up in the same scenario, they’re there to knock out the two survivors of your first pass. Cheapest <em>and</em> lowest latency is a trade-off, and the question is telling you which side of it to land on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xoK-__f-d1Vu0qbqXPb3YQ.png" /><figcaption>The question is already driving you on the correct answer. This is true also in the real work!</figcaption></figure><h3>Third pass — distrust the most powerful answer</h3><p>AWS rewards the appropriate solution, not the most sophisticated one. The instinct — especially if you’re an engineer who enjoys this stuff — is to pick the design you’d be proud of. That instinct is wrong in an exam room and, frankly, mostly wrong in production. If the scenario describes a nightly batch, the answer is not streaming. If a managed service covers 90% of the requirement, the answer is not the custom pipeline that covers 100% and needs an on-call rota.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FMDx_hbnVK1QwYI5X69PJw.png" /><figcaption>The process I used to go really fast</figcaption></figure><p>You don’t get this from studying. You get it from doing questions until the reasoning is automatic — and above all from reviewing every wrong answer until you can say out loud why the other three failed. The learning is in the eliminations. If you only check your score, you’re measuring yourself, not improving.</p><h3>The practice exams I built — free for readers of this article</h3><p>Which brings me to the one thing I couldn’t find while preparing.</p><p>Most practice material out there is aligned to exam guides that have moved on: retired services, superseded terminology, domains that no longer carry the weight they used to. That’s worse than useless, because it hands you confidence you haven’t earned and you only find out on exam day.</p><p>So I rebuilt my own sets against the current guides — same domains, same weights, same scenario style, same difficulty — with a written explanation for every single option: why the correct one is correct and, more importantly, why the other three fail. That second half is the part that trains everything I’ve described in this section, and it’s the part almost nobody bothers writing.</p><p>They’re on Udemy, and they’re free for anyone reading this. No email, no funnel, no expiring countdown — consider it the fee for your attention:</p><p>🎯 <a href="https://www.udemy.com/course/aws-saa-c03-practice-exams-195-questions-explanations/?couponCode=7A0DDEDC399BE8DB369D"><strong>AWS Certified Solutions Architect — Associate (SAA-C03) — Practice Exams</strong></a></p><p>🎯 <a href="https://www.udemy.com/course/aws-cloud-practitioner-clf-c02-practice-tests-answers/?couponCode=FB48F8E7B37C56317546"><strong>AWS Certified Cloud Practitioner (CLF-C02) — Practice Exams</strong></a></p><p>🎯 <a href="https://www.udemy.com/course/aws-dea-c01-practice-exams-195-questions-explanations-k/?couponCode=ED0F52F3A6A615A15D8B"><strong>AWS Certified Data Engineer — Associate (DEA-C01) — Practice Exams</strong></a></p><blockquote>The courses are free for the first 100 readers who claim them, so hurry up! (I will try to renew the coupon every month if you text me).</blockquote><p>If you’re heading down the ML track instead, those aren’t built yet. Leave a comment saying which exam you’re sitting and I’ll know what to write next.</p><h3>Studying by difference, or why the fifth exam costs a fraction of the first</h3><p>Now back to the elbow in the curve.</p><p>AWS certifications overlap enormously. The data engineering track and the ML track are not separate worlds — the same cast keeps walking on stage under different exam names. S3, EC2, Lambda, IAM, KMS, VPC, Glue, DynamoDB, Step Functions, CloudWatch, the cost models. By the autumn I had a solid working base on all of them, and re-watching those lessons would have been the most expensive thing I could do with my evenings.</p><p>So from that point on, preparation looked like this: open the official exam guide, go through it domain by domain, and mark only the topics where that domain asks something I’d never been asked before. Everything else got skipped without guilt — including large chunks of courses I’d paid for.</p><p>The key is that what changes between one certification and the next usually isn’t the service. It’s the angle the domain takes on it. DynamoDB is the same database in every exam; what moves is the question. In a data engineering context you’re asked about it as a sink, about Streams feeding a downstream pipeline, about capacity modes under a bursty write pattern. In an ML context you’re asked about it as a low-latency lookup at inference time. Glue is the same service, but one exam wants the catalog and the crawler and another wants job bookmarks and the cost of a badly sized DPU allocation. You don’t relearn the service. You learn the new angle, and that’s an hour, not a week.</p><p>This is what December looks like from the inside. Machine Learning — Specialty and Machine Learning Engineer — Associate are two exams over largely the same territory; sitting them weeks apart, the second one cost me a fraction of the first. The Solutions Architect — Associate, arriving after everything else, was mostly a matter of assembling components I’d already met one by one in other exams.</p><p>A course is built for someone starting from zero. From the third certification onward you are not that person, and sitting through the fundamentals again out of diligence isn’t rigour — it’s the most expensive form of procrastination available to a certified person.</p><p>But this only works under one condition, and it’s the whole point: <strong>the first pass has to be done properly.</strong></p><p>If you learned DynamoDB as “the NoSQL one” and Lambda as “the serverless one”, you’ve built nothing. You’ll relearn both every time and pay full price for every exam. If you learned them once with the access patterns, the limits, the failure behaviour and the cost model attached, you’ll be living off that for years. Depth up front is what buys the speed later.</p><p>Which is, in retrospect, why the Cloud Practitioner was worth its disproportionate pain. It was the exam that forced me to complete the base map — wide, shallow, and with no holes left in it — and it sits in November, immediately before the four certifications that came in the following weeks. The exam that cost the most per hour is the one that returned the most, almost immediately.</p><p>So the practical scheduling advice is: don’t space certifications out, cluster them while the material is still warm. But the version that actually matters is: build the foundations once, build them properly, and then stop paying for them.</p><h3>Two of those badges are for exams nobody can sit anymore</h3><p>Data Analytics — Specialty was retired in April 2024, days after I passed it, and replaced by Data Engineer — Associate. Machine Learning — Specialty — which I passed in December 2024 — was retired on March 31, 2026, with AWS splitting its territory across three newer credentials: Machine Learning Engineer — Associate, AI Practitioner, and the new Generative AI Developer — Professional.</p><p>Two out of seven, obsolete as exams. I passed one of them in its final week and the other roughly a year before it was announced dead.</p><p>This bothers me less than it probably should, because it clarifies what a certification is actually worth. The badge is a receipt. What stayed is the map — the shape of the AWS analytics and ML stack, which services exist, what each is for, where the trade-offs sit — plus the reasoning habit, which is completely portable. Nothing I learned preparing for DAS-C01 stopped being true when the exam was pulled.</p><p>If you’re starting today, the equivalent path is: <strong>Data Engineer — Associate</strong> if your work is pipelines, <strong>Machine Learning Engineer — Associate</strong> if it’s models in production, <strong>AI Practitioner</strong> if you want the vocabulary quickly, and <strong>Generative AI Developer — Professional</strong> if you’re actually building on foundation models. Check the current exam guides before buying any course — if the material still references retired exam codes, bin it.</p><h3>Four mistakes worth avoiding</h3><p><strong>Studying breadth the way you study depth.</strong> Foundational exams punish perfectionism. Decide upfront which exams deserve service-by-service rigour and which need coverage and pace.</p><p><strong>Studying every course from lesson one.</strong> The mirror image of the previous mistake. Once you have a real base, your study plan is the exam guide, not the course index. Skip what you already own and spend the time on the angles the new domain adds.</p><p><strong>Practising without dissecting.</strong> A practice score tells you nothing. The value is in writing down, for every miss, which constraint in the scenario you failed to notice. After a few dozen of those, you start seeing constraints on the first read.</p><p><strong>Waiting until you feel ready.</strong> That day doesn’t arrive. Book the date first and work backwards. It’s the only reason five of these landed in eight weeks instead of drifting across two years — and the same reason two Databricks certifications later took me two months rather than six.</p><p>The AWS run is what made the Databricks preparation fast, because by then I wasn’t learning how to study anymore, only what to study. I wrote that method up in detail here: 👉 <a href="https://medium.com/databrickscommunity/how-i-passed-both-databricks-data-engineer-certifications-in-two-months-studying-after-work-843ef2390868"><em>How I Passed Both Databricks Data Engineer Certifications in Two Months</em></a>.</p><p>If you’re preparing for any of these and you’re stuck on something, leave a comment. I answer all of them.</p><p>Good luck for your exams!</p><p><em>Disclaimer: I’m not affiliated with AWS. Some of the links above are affiliate links — they cost you nothing extra, and I only link to material I actually used.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=221c6d9ef008" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering Interviews: SQL Alone Is Not Enough]]></title>
            <link>https://fernandodenitto.medium.com/data-engineering-interviews-sql-alone-is-not-enough-2d83768ddba8?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d83768ddba8</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[interview-questions]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 17:11:01 GMT</pubDate>
            <atom:updated>2026-08-19T17:11:01.616Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How to prepare for a Data Engineering interview in 2026: projects, coding, data modeling, system design, and the human side of the job.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Op42YU53QYuouHulTuE8AA.png" /></figure><p>Data engineering is no longer a quiet niche sitting behind dashboards and reports. It is the infrastructure that makes reliable analytics, data-driven products, and AI applications possible.</p><p>That is why Big Data Specialists are included among the fastest-growing technology roles projected through 2030 by the World Economic Forum. But increased demand does not make interviews easier. Companies are looking for people who can do more than operate a platform: they want engineers who can make sound technical decisions in real-world conditions.</p><p>A Data Engineering interview is not simply a test of whether you know every Spark function or SQL syntax rule. It is an opportunity to show that you can turn imperfect data into reliable, understandable, and useful systems.</p><p>Here is how to prepare.</p><h3>Your resume is already the first interview</h3><p>A strong resume is not a list of technologies. It is a set of professional stories that you should be ready to explain in depth.</p><p>Writing “built pipelines using Kafka, Spark, and Snowflake” says very little. An interviewer wants to understand the problem you solved, why you chose those technologies, and what measurable result you delivered.</p><p>A useful structure is:</p><blockquote><em>context → problem → decision → trade-off → outcome</em></blockquote><p>For every meaningful project, be ready to answer questions such as:</p><ul><li>What business or data problem were you solving?</li><li>Who used the final output?</li><li>Which teams were involved, and how did you work with them?</li><li>Why did you choose that approach instead of an alternative?</li><li>What constraints did you face: cost, latency, volume, compliance, or team skills?</li><li>How did you measure success?</li><li>What went wrong, and what did you change?</li></ul><p>Numbers matter, as long as they are accurate and meaningful. “Built a high-scale pipeline” is vague. “Reduced dashboard refresh time from six hours to twenty minutes for 40 operational dashboards” is a credible and useful story.</p><p>You do not need millions of events per second to demonstrate impact. Automating a reconciliation process, reducing data incidents, or making a critical dataset traceable can be just as valuable. The important thing is understanding why the work mattered.</p><h3>Coding: fewer tricks, more data awareness</h3><p>Coding interviews for Data Engineers are often different from general software engineering interviews. Algorithms still matter, but the focus is usually more practical: transforming data, handling edge cases, choosing appropriate data structures, and writing readable code under pressure.</p><p>You may be asked to work with a dataset of events and:</p><ul><li>normalize dirty or missing fields;</li><li>deduplicate records;</li><li>aggregate data over time windows;</li><li>handle timestamps, time zones, and out-of-order events;</li><li>implement data-quality checks;</li><li>write transformations in Python, Pandas, or PySpark.</li></ul><p>The hardest part is rarely just making the code run. It is showing your reasoning.</p><p>Before coding, clarify the important assumptions:</p><ul><li>How large is the input?</li><li>Does the dataset fit in memory?</li><li>Can records be duplicated?</li><li>How should missing or malformed values be handled?</li><li>Does the result need to be real-time, or is batch processing acceptable?</li><li>Which failures should stop the pipeline, and which should be quarantined?</li></ul><p>These questions distinguish someone solving an exercise from someone designing a data system.</p><p>Practicing on platforms such as LeetCode or HackerRank can help, but do not turn preparation into an endless puzzle marathon. Focus on lists, dictionaries, sets, sorting, time complexity, and edge cases. If the job description mentions Spark, dbt, Airflow, or Pandas, be prepared to use them in practice.</p><h3>SQL: the common language of Data Engineering</h3><p>SQL is not a secondary skill. It is one of the most common and revealing parts of a Data Engineering interview.</p><p>Knowing how to write a SELECT statement with a JOIN is not enough. You need to translate a business question into a query that is correct, efficient, and maintainable.</p><p>Practice in particular:</p><ul><li>joins and their effect on cardinality;</li><li>CTEs and subqueries;</li><li>window functions;</li><li>conditional aggregations;</li><li>dates, timestamps, and time zones;</li><li>deduplication;</li><li>funnels and event sequences;</li><li>Slowly Changing Dimensions;</li><li>incremental queries;</li><li>execution plans, partitioning, and reducing data scanned.</li></ul><p>A strong SQL interview is not only about the final query. It is also about how you validate it.</p><p>Did you check whether a join multiplies rows? Did you define what happens when a user has no events? Did you make sure the denominator of a business metric is correct?</p><p>This attention matters more than ever. In dbt Labs’ 2025 State of Analytics Engineering report, poor data quality remains one of the most persistent challenges for data teams. Writing an elegant query is useful; producing data people can trust is the real goal.</p><h3>System design: explain decisions, not just tools</h3><p>System design is often the most demanding part of the process.</p><p>The initial question may be intentionally broad: “Design an analytics platform for a marketplace,” “Build a real-time event pipeline,” or “Design a data warehouse for a SaaS product.”</p><p>The most common mistake is immediately listing technologies:</p><p>Kafka. Spark. Data lake. Warehouse. dbt. Airflow.</p><p>That is not system design. It is an inventory.</p><p>A better approach starts with requirements:</p><ol><li>Who will consume this data, and what decisions will they make with it?</li><li>What data sources exist?</li><li>What volume, velocity, and variety of data must be handled?</li><li>How fresh does the data need to be?</li><li>What privacy, security, and retention requirements apply?</li><li>Which metrics need to be correct and reconcilable?</li><li>How will you handle errors, retries, backfills, and observability?</li></ol><p>Only then should you discuss the architecture.</p><p>A complete answer should cover the entire data lifecycle:</p><p>Sources → Ingestion → Storage → Transformation → Quality → Serving → Monitoring</p><p>For each component, explain the alternatives and trade-offs. Batch or streaming? Data lake or warehouse? Denormalized tables for fast reads, or a more normalized model to reduce duplication? Exactly-once semantics, or lower latency with accepted limitations?</p><p>There is not always one correct answer. There is, however, an answer that is justified by the requirements.</p><p>For example, Spark Structured Streaming remains relevant because it lets teams express many real-time workloads with a model similar to batch queries. But choosing Spark is not automatically a good decision. For modest volumes and simple operational requirements, it may introduce unnecessary complexity.</p><p>Maturity shows when you can say: “This solution would work, but for this use case I would choose something simpler.”</p><h3>Data modeling: start with the questions users will ask</h3><p>Data modeling is not the final step you take once the data is available. It determines whether people can actually use that data.</p><p>When you are asked to model a domain — orders, bookings, subscriptions, campaigns — do not begin with tables and columns. Begin with questions:</p><ul><li>Which metrics do users need?</li><li>At what level of granularity?</li><li>Which dimensions are needed for segmentation?</li><li>How should historical changes be handled?</li><li>What is the official definition of a conversion, active customer, or completed order?</li></ul><p>Then you can justify a choice between a dimensional model, a normalized model, data vault, or a hybrid approach.</p><p>Interviewers are not looking for someone who repeats “star schema” as a magic formula. They want someone who can adapt a model to the way the data will be consumed.</p><h3>AI has made the role more important, not less human</h3><p>AI has raised the bar for Data Engineering.</p><p>Generative models, retrieval systems, and intelligent products all depend on data that is accessible, governed, current, and verifiable. Without quality checks, lineage, access policies, and observability, AI does not solve data chaos. It amplifies it faster.</p><p>That is why it is worth discussing these areas in an interview:</p><ul><li>data quality and automated testing;</li><li>cataloging and lineage;</li><li>access control and sensitive data;</li><li>pipeline and cost monitoring;</li><li>trusted datasets for analytics and AI;</li><li>contracts between data producers and consumers.</li></ul><p>You do not need to claim expertise in every tool. You need to show that you understand a pipeline is not finished when it loads data into a table. It is finished when the team can trust that data.</p><h3>An interview is not an oral exam</h3><p>Technical ability matters. But a good Data Engineer usually works at the intersection of engineering, analytics, product, and business.</p><p>That is why interviewers also pay attention to how you communicate.</p><p>Ask questions. State your assumptions. If you do not know an answer, do not invent one: explain how you would find it. If a decision depends on requirements, say so clearly.</p><p>A successful interview is not about having an immediate answer to every question. It is about sharing high-quality reasoning.</p><p>And remember: the process goes both ways. Ask how the company handles data ownership and quality, how much time is invested in technical debt, how priorities are set, and how the data team works with product teams.</p><p>You can be an excellent candidate and still find that the team is not the right place for you.</p><h3>A practical preparation plan</h3><p>If I had to reduce everything to a simple plan, it would be this:</p><ul><li>Prepare three project stories with context, trade-offs, numbers, and impact.</li><li>Practice writing SQL without autocomplete, then explain its correctness and performance.</li><li>Solve coding exercises focused on data transformations and edge cases.</li><li>Simulate at least two end-to-end system-design interviews.</li><li>Review data modeling, quality, observability, and security concepts.</li><li>Do a mock interview with someone who will interrupt you, challenge your choices, and ask for clarification.</li></ul><p>You do not need to know everything. Nobody does.</p><p>You need to show that you can approach an ambiguous problem, ask the right questions, choose a proportionate solution, and build data systems that other people can trust.</p><p>That, much more than a single perfect SQL query, is the job of a Data Engineer.</p><p><em>Sources: </em><a href="https://www.weforum.org/publications/the-future-of-jobs-report-2025/in-full/2-jobs-outlook/"><em>World Economic Forum, Future of Jobs Report 2025</em></a><em>, </em><a href="https://www.getdbt.com/resources/state-of-analytics-engineering-2025"><em>dbt Labs, State of Analytics Engineering 2025</em></a><em>, and </em><a href="https://spark.apache.org/docs/latest/streaming/getting-started.html"><em>Apache Spark Structured Streaming documentation</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d83768ddba8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Passed Both Databricks Data Engineer Certifications in Two Months — Studying After Work]]></title>
            <link>https://fernandodenitto.medium.com/how-i-passed-both-databricks-data-engineer-certifications-in-two-months-studying-after-work-843ef2390868?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/843ef2390868</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Sat, 15 Aug 2026 10:18:21 GMT</pubDate>
            <atom:updated>2026-08-28T13:22:42.366Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsQWW3k-aJu0-AmcV-O4gQ.png" /></figure><h3>The four-phase method I brought over from eight AWS certifications, and the one thing that actually separates people who pass from people who don’t</h3><p>In 2024 I did something slightly unreasonable: <strong>eight AWS certifications in a single year</strong>, all studied in the evenings after a full working day.</p><p>Not because I had free time. Because at some point I understood something that applies to every certification, on every platform: <strong>the bottleneck is never how much you study. It’s how you study.</strong></p><p>When I moved to Databricks, I ran the same playbook.</p><p>Result: <strong>Data Engineer Associate and Data Engineer Professional, both passed in two months</strong>, on a couple of hours a day.</p><p>This is the full method, the exact resources I used, and the mistakes I’d tell you to avoid. No shortcuts, no magic — but no wasted time either.</p><h3>First: know what you’re actually walking into</h3><p>Most people fail because they prepare for an exam that no longer exists. Databricks refreshes its exam guides regularly, and a lot of the material floating around is one or two versions behind.</p><p><strong>Data Engineer Associate</strong> — current guide: <strong>May 2026</strong>.</p><p>Five domains:</p><ul><li>Databricks Intelligence Platform 10%</li><li>Development and Ingestion 30%</li><li>Data Processing &amp; Transformations 31%</li><li>Productionizing Data Pipelines 18%</li><li>Data Governance &amp; Quality 11%</li></ul><p>45 questions, 90 minutes, USD 200 plus local tax, proctored. Recertification every two years.</p><p>Two changes that trip people up: the terminology moved from <strong>Workflows</strong> to <strong>Lakeflow Jobs</strong>, and <strong>CI/CD with Databricks Asset Bundles</strong> is now explicitly in scope. If your study material still says “Workflows,” it’s outdated — bin it.</p><p><strong>Data Engineer Professional</strong> — current guide: <strong>November 30, 2025</strong>. <strong>59 scored multiple-choice questions</strong>, restructured into <strong>ten domains with explicit weights</strong>. Two of them are new as first-class domains: <strong>Data Sharing and Federation</strong> (Delta Sharing, Lakehouse Federation) and <strong>Data Modelling</strong>. Scope covers Delta Lake, Unity Catalog, Auto Loader, Lakeflow Spark Declarative Pipelines, Databricks compute including serverless, Lakeflow Jobs and the medallion architecture — plus designing secure, reliable, cost-effective ETL pipelines, schema management, observability, governance, performance optimization, streaming workloads, orchestration, and CI/CD with the Databricks CLI, REST API and Asset Bundles. Code is in Python.</p><p>So before you open a single video, download and read the official exam guides:</p><p>👉 <a href="https://www.databricks.com/learn/certification/data-engineer-associate">Data Engineer Associate — official page</a></p><p>👉 <a href="https://www.databricks.com/learn/certification/data-engineer-professional">Data Engineer Professional — official page</a></p><p>It takes twenty minutes and it will save you twenty hours.</p><h3>The difference between the two exams (this is the important part)</h3><p>Let me be blunt, because this is the single most useful thing in this article.</p><p><strong>The Professional is not “the Associate but harder.” It’s a different kind of exam.</strong></p><p>The Associate asks: <em>do you know how this works?</em></p><p>The Professional asks: <em>have you already dealt with this in production, at 2am, when it broke?</em></p><p>Professional questions are long scenarios with multiple constraints, where every option is technically valid and only one survives contact with reality. You get a pipeline that’s misbehaving and you have to work out <em>why</em>. If you walk in with an Associate-style preparation, you can genuinely fail while knowing the material.</p><p>Which is why the method below stays the same for both, but the weight of each phase shifts.</p><h3>The four-phase method</h3><h3>Phase 1 — Understand with your hands, not your eyes</h3><p>The most common mistake is studying Databricks by watching videos passively. It doesn’t work, because the exam asks <em>what happens if</em>, not <em>what does this mean</em>.</p><p>For every topic, I opened the workspace and rebuilt it myself. You don’t understand Auto Loader because you know it does incremental ingestion — you understand it once you’ve fed it files with a changing schema and watched what schema evolution actually does. You don’t understand Delta Lake until you’ve run DESCRIBE HISTORY, time-travelled to an earlier version, and seen with your own eyes what VACUUM takes away.</p><p>Ten minutes of hands-on beats an hour of video. For the Professional this isn’t a preference, it’s the whole game. The things I built by hand that showed up almost directly in exam questions:</p><ul><li>a full <strong>bronze → silver → gold</strong> pipeline, not a toy one</li><li><strong>deduplication</strong> on a stream with a watermark — and understanding what the watermark does when late data arrives</li><li>a <strong>SCD Type 2</strong> implemented with MERGE, with all its traps</li><li><strong>Change Data Feed</strong> used to propagate changes downstream</li><li>a <strong>stream-stream join</strong>, to see what happens to state</li><li>a job that <strong>fails and restarts</strong>, to really understand checkpoints and delivery semantics</li><li>the same project deployed to two targets with <strong>Asset Bundles</strong></li></ul><p>Every time I broke something, I learned an exam question.</p><h3>Phase 2 — Review to connect, not to memorize</h3><p>After hands-on I did a focused review, but with a different goal: not remembering syntax, but <strong>being clear about what I was doing and why</strong>.</p><p>My test is simple: if I can’t say out loud why I’d pick this tool over that one, I haven’t understood it. Why medallion instead of a single layer? Why Auto Loader instead of COPY INTO? Why a streaming table instead of a materialized view?</p><p>For the Professional I pushed this further and built what turned out to be my most valuable study document: not a summary of topics, but <strong>a list of forks with the rule for choosing</strong>.</p><ul><li>liquid clustering or partitioning?</li><li>MERGE or foreachBatch?</li><li>streaming table or materialized view?</li><li>Delta Sharing or Lakehouse Federation?</li><li>serverless or classic compute?</li><li>automatic schema evolution or strict enforcement?</li></ul><p>Both exams are full of questions that are exactly this: a scenario, four technically possible answers, one sensible one.</p><h3>Phase 3 — Official documentation as the source of truth</h3><p>This is the phase almost everyone skips, and it’s the one that earned me the most points.</p><p>Every time a course or a practice question left me with a doubt, I went to the official docs. Not Stack Overflow. Not a blog post from 2023. The docs.</p><p>👉 <a href="https://docs.databricks.com/">docs.databricks.com</a></p><p>The reason is practical: exam questions are written against the product’s real, current behaviour. If a feature changed in the last six months, the documentation knows and your course probably doesn’t. For the Professional I read the Structured Streaming, performance optimization, Unity Catalog and Asset Bundles sections end to end — including the notes on limits and failure behaviour, which everybody skims and which are exactly where the nastiest questions come from.</p><h3>Phase 4 — The mindset switch</h3><p>Here’s where it’s won or lost.</p><p><strong>Theory is one thing. The mental setup you bring into the exam room is another.</strong> In the final week I stopped learning new topics entirely and started training myself to think the way the question writers think. Concretely:</p><ul><li><strong>Read the scenario hunting for constraints, not for the answer.</strong> Every question hides two or three words that eliminate half the options: <em>“without duplicates,” “with the lowest latency,” “minimizing cost,” “without reprocessing everything,” “data can arrive up to three days late,” “the downstream team reads every 15 minutes.”</em> In the Professional these are buried mid-paragraph on purpose. I mark them before I even read the options.</li><li><strong>Eliminate, don’t select.</strong> Of four options, two are usually clearly wrong and two are plausible. The exam is always decided between those two, and the difference is almost always an operational detail rather than a concept.</li><li><strong>Distrust the most powerful answer.</strong> Databricks rewards the <em>appropriate</em> solution, not the most sophisticated one. If the scenario describes a daily batch, the answer is not streaming. In production, an elegant design that dies on the first dirty record is a wrong design — and the exam agrees.</li><li><strong>Manage time and fatigue.</strong> 45 questions in 90 minutes is two minutes each; stuck questions get flagged and revisited. And 59 scenario-based questions is a real cognitive load — from question 40 onward your sharpness drops and you start making stupid mistakes. Always simulate full, timed exams, never in fragments. You’re training stamina, not just knowledge.</li></ul><p>You don’t get this switch from studying. You get it from doing questions until the reasoning becomes automatic.</p><h3>The resources I actually used</h3><p>No padding here — this is the exact list.</p><h3>The courses</h3><ul><li>Associate: 👉 <a href="https://trk.udemy.com/QYxm7P"><strong>Databricks Certified Data Engineer Associate</strong></a></li><li>Professional: 👉 <a href="https://trk.udemy.com/5kB9oj"><strong>Databricks Certified Data Engineer Professional</strong></a></li></ul><p>Why these two: <strong>they get straight to the point.</strong> No hours of generic “what is big data” introductions, no filler. They cover what’s actually on the exams, and they cover it with hands-on, practical examples — which is exactly what Phase 1 of the method needs. When you only have a couple of hours a night, wasted time is your number one enemy, and neither of these wastes any. For the Professional especially, a purely theoretical course is useless: you need to see the code run and the pipeline break.</p><h3>Question practice</h3><p>For training the reasoning I used <a href="https://certbase.pro/"><strong>certbase.pro</strong></a>, which has questions very close to the real exam in style and difficulty. That mattered most in the final phase — on the Professional, the <em>style</em> of the questions is half the difficulty.</p><h3>My own practice exams</h3><p>While preparing, I kept hitting the same problem: most of the practice material out there is <strong>aligned to old exam guides</strong>. Deprecated features, outdated terminology, topics that are no longer in scope — and for the Professional, almost nothing covering Data Sharing, Federation and Data Modelling, which are now full domains. That’s worse than useless, because it gives you false confidence.</p><p>So I built my own sets, rebuilt against the current exam guides and <strong>updated as of August 2026</strong>: same domains, same weights, same scenario style and same difficulty level as the real thing, with a detailed explanation for every option — why the right one is right and, more importantly, why the other three aren’t. That last part is where the learning actually happens.</p><p>🎯 <a href="https://www.udemy.com/course/databricks-data-engineer-associate-practice-exams-2026-z/?referralCode=BBB61CB705DC300BBE86"><strong>Databricks Data Engineer Associate — Practice Exams 2026</strong></a></p><h3><strong>Free for the first 100 readers of this article</strong> with coupon clicking here</h3><p>🎯 <a href="https://www.udemy.com/course/databricks-data-engineer-professional-practice-exams-2026/?referralCode=F6CD063344B42AB06969"><strong>Databricks Data Engineer Professional — Practice Exams 2026</strong></a></p><p><strong>Discount for the first 100 readers of this article</strong> with coupon <a href="https://thenitto.s.gy/lZ16al">clicking here</a></p><h3>The plan I’d follow if I started over</h3><p><strong>Associate — four weeks, one to two hours a day</strong></p><ul><li><strong>Week 1</strong> — Read the exam guide. Set up the workspace. Platform, architecture, Delta Lake. Hands-on everything.</li><li><strong>Week 2</strong> — Ingestion and transformations. This is 61% of the exam, so it gets double the time: Auto Loader, streaming tables, PySpark and Spark SQL.</li><li><strong>Week 3</strong> — Lakeflow Jobs, productionizing, CI/CD with Asset Bundles, Unity Catalog and governance.</li><li><strong>Week 4</strong> — Questions only. Analyse every mistake, go back to the docs for every doubt, build your personal one-page sheet of recurring decisions.</li></ul><p><strong>Professional — four more weeks</strong></p><ul><li><strong>Week 5</strong> — Advanced Structured Streaming: watermarks, state, stream-stream joins, delivery semantics.</li><li><strong>Week 6</strong> — Data modelling and quality: medallion done properly, SCD Type 1 and 2, CDC and Change Data Feed, deduplication.</li><li><strong>Week 7</strong> — Performance and cost optimization, monitoring, security and governance, Delta Sharing and Lakehouse Federation.</li><li><strong>Week 8</strong> — CI/CD, testing, deployment with CLI, REST API and Asset Bundles — then full timed simulations.</li></ul><p>Book the exam when you’re consistently above 85% on practice tests, and not before. But don’t wait forever either: past a certain point you’re not learning any more, you’re just postponing.</p><h3>Do you need the Associate before the Professional?</h3><p>Formally, no — it isn’t a prerequisite. Practically: <strong>yes, unless you have at least a year of real Databricks work behind you.</strong> Taking them back to back has a concrete advantage: the fundamentals are still warm, so you spend all your time on depth instead of basics. If you’ve been living in Databricks daily for a couple of years, go straight for the Professional.</p><h3>Three mistakes I see constantly</h3><ol><li><strong>Studying outdated material.</strong> Guides, names and scope have all changed. If your material says Workflows instead of Lakeflow Jobs, you’re studying 2024.</li><li><strong>Only practising the happy path.</strong> The questions live in the failure cases: late data, duplicates, schema drift, jobs dying halfway. If you’ve never broken anything, you’re not ready.</li><li><strong>Booking the exam “when I feel ready.”</strong> That day never arrives. Pick a date, book it, and work backwards. That’s the only reason I got two certifications done in two months on a few hours a day.</li></ol><h3>One last thing</h3><p>A certification doesn’t make you a data engineer. But it forces you to systematically cover parts of the platform you’d never touch in day-to-day work, and that’s the real value. I started using features I didn’t know existed purely because an exam put them in front of me.</p><p>In two years I went from zero certifications to eight AWS plus two Databricks, without quitting my job and without burning my weekends. Not because I’m especially clever — because I stopped studying <em>a lot</em> and started studying <strong>with a method that always ends in training the reasoning</strong>, not stacking up facts.</p><p>If you’re preparing for either of these and you’re stuck on something, leave a comment. I answer all of them.</p><p><strong>Good luck. Let me know when you pass 👏</strong></p><p><em>Disclaimer: I’m not affiliated with Databricks. The materials referenced here are independent preparation resources and do not contain real exam questions.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=843ef2390868" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Agent Shouldn’t Have an Approve Button]]></title>
            <link>https://fernandodenitto.medium.com/your-ai-agent-shouldnt-have-an-approve-button-755b40c0ea6a?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/755b40c0ea6a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 14:29:46 GMT</pubDate>
            <atom:updated>2026-08-07T14:29:46.553Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I spent months building Ontologiq, an open-source layer where agents can propose actions on your data — but the channel to execute them doesn’t exist.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6unTxFg_9uS7G0Hvdhx9A.png" /><figcaption>Preview of ontologiq workbench UI</figcaption></figure><p>The industry standard for AI agent safety is one line of code:</p><pre>if (userConfirmed) execute()</pre><p>The model asks, a human clicks yes, the thing happens. It feels safe. It isn’t — for two reasons that took me a while to articulate, and then months to build my way out of.</p><p>The first: the confirmation lives <em>inside the same session as the model</em>. The agent that proposed the action is the agent that reports whether you confirmed it. Any safety property that depends on the model behaving is a prompt, not a guarantee.</p><p>The second is quieter and worse: <strong>time passes</strong>. An agent proposes cancelling an order at 9am. The approval arrives at 2pm, after lunch, after a meeting. Between those two moments the order shipped. The 2pm approval was true at 9am — and almost every human-in-the-loop implementation I’ve seen executes it anyway, because nothing ever looks at the data again.</p><p>I built Ontologiq around one sentence: <strong>an AI agent cannot complete a governed action on its own.</strong> Everything else exists to make that sentence expressible.</p><h3>What it actually is</h3><p>Ontologiq is an open-source compiler (Apache 2.0, Python, on PyPI). You declare your business objects in YAML — identity, a state computed from live warehouse data, relations, and the actions that may be performed on them:</p><pre>object: order<br>source:<br>  table: orders<br>identity: [order_id]</pre><pre>state:<br>  open: shipped_at is null<br>  shipped: shipped_at is not null</pre><pre>actions:<br>  - name: cancel<br>    requires: state == &#39;open&#39;<br>    policy:<br>      roles: [support]<br>    approval: required<br>    effect:<br>      type: webhook<br>      url: &quot;{{ env.OPS_API }}/orders/{order_id}/cancel&quot;</pre><p>From that, it compiles SQL views for your warehouse, a catalog, documentation, and — the part I care about — <strong>MCP tools for AI agents</strong>. The agent gets a propose_order_cancel tool. Note the verb. There is no execute_order_cancel. There is no approve tool. Not hidden, not permission-gated: <em>not on the surface, on any transport</em>. The distinction matters — a permission can be misconfigured; an endpoint that doesn&#39;t exist cannot.</p><p>When the agent calls it, the tool checks the caller’s role, evaluates state == &#39;open&#39; against the live warehouse, and creates a proposal:</p><pre>{<br>  &quot;outcome&quot;: &quot;pending_approval&quot;,<br>  &quot;proposal_id&quot;: &quot;92e207a4-…&quot;,<br>  &quot;note&quot;: &quot;A human must approve this proposal before anything happens.<br>           You cannot approve it yourself; report the proposal id to the user.&quot;<br>}</pre><p>Approval happens somewhere the model cannot reach — a CLI, or a local web workbench:</p><pre>ontologiq approvals list<br>ontologiq approvals approve 92e207a4-…<br>ontologiq audit</pre><p>The agent knows its own proposal_id. So here&#39;s the design question that shaped everything: if the workbench had an approve endpoint, what would stop the agent from sending one HTTP request and closing the loop without a human? Nothing would. <strong>That&#39;s why the endpoint doesn&#39;t exist.</strong> A POST to an approve-shaped URL returns 405, and a test named test_there_is_no_write_route pins it there. When identity is needed over the network, approval requires an OIDC browser login — a session cookie and a CSRF token an agent holding a bearer token never has.</p><h3>The approval is re-checked at the moment it matters</h3><p>Back to the 9am/2pm problem. In Ontologiq, the precondition is evaluated <strong>twice</strong>: once when the agent proposes, and again at the moment of execution. If the order shipped while the proposal sat in the queue, the approval doesn’t execute — it refuses, and tells you why. The test for this has my favourite name in the codebase: test_approval_reevaluates_the_precondition_and_refuses_stale_proposals.</p><p>A few more decisions in the same spirit, because governance is mostly the sum of unglamorous details:</p><ul><li><strong>The arguments a human approved are pinned by hash</strong> and re-verified before the effect fires. What was approved is what runs — not what the proposal mutated into.</li><li><strong>Exactly one execution can ever win.</strong> Claiming a proposal is a single conditional UPDATE; two racing approvers can’t fire the webhook twice.</li><li><strong>A lost effect response is recorded as </strong><strong>unknown, never </strong><strong>failed.</strong> This one is personal: recording a possibly-applied effect as failed is exactly how a refund gets issued twice. An honest &quot;I don&#39;t know&quot; beats a confident lie.</li><li><strong>Everything lands in an append-only audit log</strong> — who proposed, who approved, which version of the ontology governed the decision.</li></ul><h3>It never writes to your database</h3><p>Ontologiq doesn’t cancel the order. It tells <em>your</em> system — via a webhook or your own Python handler — that someone authorized decided to, with the full chain of who proposed, who signed, and what was true at the time. Serving processes connect to the warehouse read-only.</p><p>This sounds like a limitation. It’s the opposite: it’s what makes the thing adoptable. It needs write access to nothing, which means trying it doesn’t require a meeting with your security team.</p><h3>You already did the hard part</h3><p>Here’s the framing that took me longest to see. The enterprise ontology industry sells you a modelling <em>project</em>: workshops, taxonomies, consultants, months. But if you have a warehouse with dbt models on it, the hard part — identity resolution, cross-system joins — is already done and paid for. dim_customer isn&#39;t yet &quot;a customer with identity, state, and governed actions&quot;, but the missing piece is thin. The YAML format is deliberately small and boring, because it&#39;s configuration for a runtime, not a modelling language competing on expressiveness.</p><p>There’s even an ontologiq import dbt that scaffolds objects from your existing models. Semantic layers tell you <em>how much</em>. This is the layer of <em>who</em>, <em>in what state</em>, and <em>what may be done</em>.</p><h3>Where it stands</h3><p>Honest status: <strong>alpha, one person, built in the open from the first commit.</strong> DuckDB is proven end to end — the quickstart needs no credentials and no server. Postgres and MySQL run the entire loop in CI against live containers on every push: build, propose, the record changing state, <strong>the approval that refuses</strong>, the webhook. Databricks passed the same smoke against a live serverless warehouse. The docs say exactly what is enforced versus merely declared, and the format will change between 0.x releases.</p><p>What I’d most like right now: people trying to <strong>break the governance claims</strong> — the security model is documented precisely so it can be attacked. And if you’re putting agents in front of production systems today, I genuinely want to know what your approval flow looks like, because I suspect for most teams the honest answer is still that one line of code at the top of this post.</p><p><em>Ontologiq is open source (Apache 2.0): </em><a href="https://github.com/ontologiq/ontologiq"><em>github.com/ontologiq/ontologiq</em></a><em> · docs at </em><a href="https://docs.ontologiq.org/"><em>docs.ontologiq.org</em></a></p><pre>pip install &quot;ontologiq[serve]&quot; ontologiq-duckdb<br>ontologiq init shop &amp;&amp; cd shop<br>ontologiq build</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=755b40c0ea6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing PII in Databricks: From Manual Masking to AI-Driven Governance]]></title>
            <link>https://fernandodenitto.medium.com/managing-pii-in-databricks-from-manual-masking-to-ai-driven-governance-52875de105f6?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/52875de105f6</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pii]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 19:59:09 GMT</pubDate>
            <atom:updated>2026-06-28T19:59:09.869Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How Unity Catalog, Data Classification, and ABAC turned sensitive-data protection from a scripting chore into an automated, auditable system — and what that means for your compliance posture.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pAlJmDoj6nghpRJZ" /><figcaption>Image from <a href="https://www.databricks.com/blog/find-sensitive-data-scale-data-classification-unity-catalog">Databricks Blog</a></figcaption></figure><p>If you run a data platform of any meaningful size, sensitive data is already hiding in it. New tables land every day. A “comments” column quietly accumulates email addresses. A staging table copies production customer records and nobody remembers to lock it down. Meanwhile the regulatory stakes keep climbing — GDPR fines alone exceeded €5.6 billion in 2025, up more than €1 billion year over year.</p><p>The good news: over the past year, Databricks has shifted PII management from a manual, script-heavy exercise into something native, AI-driven, and continuous. This article walks through the full picture — the foundational governance layers you should already be using, and the recent developments that change the recommended pattern entirely.</p><h3>The mental model: it’s not one problem, it’s several</h3><p>The single biggest mistake teams make with PII is treating it as one task. It isn’t. Protecting personal data in a lakehouse breaks down into at least six distinct problems:</p><ol><li><strong>Discovery</strong> — where does PII actually live?</li><li><strong>Access control</strong> — who can see it, and under what conditions?</li><li><strong>Pseudonymization</strong> — can we analyze it without exposing identity?</li><li><strong>Encryption</strong> — is it protected at rest and in transit?</li><li><strong>Audit</strong> — can we prove who accessed what?</li><li><strong>Erasure</strong> — can we honor a “right to be forgotten” request?</li></ol><p>Databricks addresses each of these through <strong>Unity Catalog</strong>, its unified governance layer. Let’s go through them.</p><h3>1. Discovery: stop guessing where your PII is</h3><p>You can’t protect what you can’t find. This is where the most important recent change lives.</p><h3>Databricks Data Classification (the headline feature)</h3><p>Databricks Data Classification uses an <strong>agentic AI system</strong> to automatically discover and tag sensitive data across all your catalogs — no manual tagging, no custom regex scripts. It provides continuous visibility into where personally identifiable information resides.</p><p>A few things make it genuinely different from the old approach:</p><ul><li><strong>Accuracy</strong>: it combines pattern recognition, metadata, and large language models, reaching up to 60% higher accuracy than regex-only tooling.</li><li><strong>Privacy by design</strong>: your data never leaves your environment; classification runs under Databricks’ AI security controls. The feature is HIPAA compliant and follows Databricks’ AI trust-and-safety standards.</li><li><strong>Incremental, cost-aware scanning</strong>: it scans the full catalog once, then only rescans new or changed tables and columns, using Unity Catalog lineage to prioritize. New tables and columns are typically classified within 24 hours of creation. Since the original Beta, Databricks reports improved detection speed and scanning costs cut by up to 75% — a system battle-tested to hold up as your platform grows.</li><li><strong>Storage cost</strong>: results are written to default storage and you aren’t billed for that storage. It does require serverless compute, which is enabled by default in workspaces with Unity Catalog.</li><li><strong>Easy to turn on</strong>: you enable it directly from any catalog’s <em>Details</em> tab — no pipeline to build.</li></ul><h3>Review, validation, and the results table</h3><p>Detection isn’t a black box. A focused <strong>review UI</strong> surfaces high-confidence detections together with sample data, so a steward can validate and <strong>bulk-apply tags</strong> rather than confirming one column at a time. You get full visibility into which columns contain PII and who currently has access to them.</p><p>Full results land in a system table — system.data_classification.results — which holds classification results across the entire metastore, including sample values, for custom reporting or programmatic tagging. Treat access to this table carefully: anyone with SELECT on it can see the sample values behind each detection, so restrict it to users cleared to view metastore-wide results.</p><h3>Regulatory framework coverage</h3><p>The classification tags aren’t generic. They map to real compliance regimes, organized into global tags, regional tags, and framework-specific tags covering:</p><ul><li><strong>PII</strong> — data that identifies an individual directly or in combination</li><li><strong>GDPR</strong> — personal data under the EU regulation</li><li><strong>HIPAA</strong> — identifiers under the Safe Harbor de-identification method</li><li><strong>PCI DSS</strong> — payment card data</li><li><strong>GLBA</strong> — nonpublic personal financial information</li><li><strong>DPDPA</strong> — personal data under India’s Digital Personal Data Protection Act</li></ul><p>Note that regional tags only activate in workspaces deployed in the corresponding region (U.S. tags are the exception — available everywhere).</p><blockquote><strong><em>Business case (FanDuel):</em></strong><em> automated, comprehensive PII detection across a fast-growing data environment, ensuring sensitive information is consistently identified — and freeing the governance team to focus on higher-value work instead of manual tagging.</em></blockquote><h3>Lineage closes the loop</h3><p>Unity Catalog <strong>lineage</strong> lets you trace where PII flows downstream — into which notebooks, dashboards, and models. For a GDPR audit, this is the difference between “we think this data is used here” and a defensible map of the entire flow. At the 2026 Data + AI Summit, <strong>external lineage</strong> went generally available, extending the graph to upstream source systems and downstream BI reports so a single lineage view can span your full data flow end to end.</p><h3>2. Access control: classify once, enforce everywhere with ABAC</h3><p>Here is the real architectural shift. The old pattern was to write a masking function for every sensitive column, by hand. The new pattern is <strong>attribute-based access control (ABAC)</strong>: you classify data once, and policies apply automatically wherever that classification appears.</p><p>At the 2026 Data + AI Summit, Databricks announced the <strong>General Availability of ABAC policies for row filtering and column masking</strong>, alongside Governed Tags and Data Classification. The workflow is now tightly integrated: from the Data Classification results page, you click <em>Review</em> on a tag, open the <em>User Access</em> tab, and the policy form arrives pre-filled to mask columns carrying that tag — you just point it at a registered masking function and save.</p><h3>The building blocks still matter</h3><p>Under the hood, the enforcement primitives are worth knowing:</p><ul><li><strong>Column masking</strong> and <strong>row filters</strong> applied through Unity Catalog</li><li><strong>Dynamic views</strong> that obscure columns based on the caller’s group membership, via is_account_group_member()</li></ul><p>A classic dynamic-masking function looks like this:</p><pre>CREATE FUNCTION mask_email(email STRING)<br>RETURN CASE<br>  WHEN is_account_group_member(&#39;pii_readers&#39;) THEN email<br>  ELSE regexp_replace(email, &#39;(.)(.*)(@.*)&#39;, &#39;$1***$3&#39;)<br>END;<br><br>ALTER TABLE customers ALTER COLUMN email SET MASK mask_email;</pre><p>The point of ABAC is that you no longer wire this up table by table — the tag drives the policy at scale.</p><h3>A concrete ABAC scenario</h3><p>Picture a customer transactions table. It mixes sensitive columns (customer_name, email, phone) with perfectly shareable ones (transaction_id, customer_id). The old way of protecting it meant either blocking the whole table or maintaining a tangle of view logic.</p><p>With ABAC, you write the policy once: <em>mask every column tagged </em><em>class.name, </em><em>class.email_address, and </em><em>class.phone_number for everyone except the security team.</em> From then on, the policy applies automatically to any data carrying those tags — masking only the sensitive columns while leaving transaction_id and customer_id fully open. No table-wide lockdown, no per-column functions, no complex views to keep in sync. That&#39;s the difference between protection that you maintain and protection that scales with your business.</p><blockquote><strong><em>Business case (Oportun):</em></strong><em> classification results feed an authorization workflow built on Databricks Apps to enable </em><strong><em>Just-In-Time access</em></strong><em>. Sensitive data stays accessible only when it’s actually needed, with detection and protection automated across the whole estate.</em></blockquote><h3>Automating sensitivity tiers</h3><p>Beyond masking, classification unlocks scalable access workflows. You can categorize tables automatically into sensitivity levels — <em>confidential, restricted, internal, public</em> — and route access requests through approval flows keyed to those levels. The user gets approved based on the dataset’s sensitivity, not a manual judgment call every time.</p><h3>3. Pseudonymization and tokenization</h3><p>For analytics that need to correlate behavior without exposing identity, pseudonymization sits between “raw PII” and “deleted.” Native options include:</p><ul><li><strong>Hashing</strong> with sha2() for irreversible pseudonymization of identifiers</li><li><strong>Encryption</strong> with Spark SQL’s native aes_encrypt() / aes_decrypt()</li></ul><p>For reversible, enterprise-grade tokenization, teams typically integrate an external vault or KMS (Protegrity, Privacera, or a cloud KMS).</p><blockquote><strong><em>Business case:</em></strong><em> an analytics pipeline correlates user behavior across sessions using a hashed identifier, so data scientists can build cohorts and funnels while the real identity never reaches the analytics layer.</em></blockquote><h3>4. Encryption and isolation</h3><p>The baseline: cloud-provider-managed encryption at rest and in transit, upgraded with <strong>customer-managed keys (CMK)</strong> when you need to control the key lifecycle yourself. For the most sensitive workloads, isolate data into dedicated workspaces or catalogs so that segregation is structural, not just policy-based.</p><h3>5. Audit and monitoring: proving it, not just doing it</h3><p>Compliance is as much about <em>demonstrating</em> control as exercising it. Continuous classification turns three traditionally painful tasks into routine ones:</p><ul><li><strong>Audit-readiness</strong> — pull complete logs showing exactly where PII resides and which users and groups can access it. system.access.audit records who accessed what, forming the backbone of a GDPR/CCPA audit trail.</li><li><strong>Full lineage</strong> — trace where PII flows downstream so you don’t miss the spots where it was accidentally copied into a derived dataset.</li><li><strong>Data-deletion requests</strong> — locate every instance of a user’s data across all your tables before you act on an erasure request (more on that next).</li></ul><p>On top of these, the new <strong>Governance Hub</strong> (Private Preview as of mid-2026) gives data stewards and admins a centralized command center to monitor posture, surface risky access changes and policy drift, and prioritize remediation across data, AI, cost, and performance.</p><p>As one Databricks engineering leader frames it, a data team’s real currency is trust — <em>consistency over time</em> — and automated classification plus remediation is what makes that trust defensible.</p><h3>6. The right to be forgotten</h3><p>GDPR Article 17 requests have to reach the physical files, not just the logical table. <strong>Delta Lake</strong> handles granular DELETE and MERGE, and VACUUM removes the data from historical files:</p><pre>DELETE FROM customers WHERE customer_id = &#39;xyz&#39;;<br>VACUUM customers RETAIN 168 HOURS;</pre><p>Mind the VACUUM retention window — it governs how long deleted data lingers in old files before physical removal, which is exactly what an erasure request is concerned with.</p><h3>Bonus: governing PII in the age of AI agents</h3><p>If you’re building GenAI applications, PII protection now extends to runtime. Announced at the 2026 Summit, <strong>Contextual Service Policies</strong> (Beta) move governance from “who can access a model or tool” to “what it can do in a given interaction.” Built-in guardrails protect against PII exposure, prompt injection, and unsafe content — and agent activity is captured in a unified telemetry layer that can be analyzed in Lakewatch, Databricks’ lakehouse-native SIEM.</p><p>This matters because an agent with broad data access is a new kind of PII exposure surface. Classifying and masking the underlying tables is necessary but no longer sufficient; you also want to constrain what the model can <em>do</em> with what it can reach.</p><h3>On the roadmap</h3><p>Worth knowing if you’re planning ahead, since a few of these reshape how you’d automate the pattern. Databricks has signalled the following directions for Data Classification:</p><ul><li><strong>API and Terraform support</strong> — to manage classification as infrastructure-as-code, coming to Public Preview.</li><li><strong>Built-in regional and domain-specific classifiers</strong> — such as PHI and PCI, coming to Public Preview.</li><li><strong>Custom classification rules</strong> — agentic AI that learns patterns specific to <em>your</em> company’s data, in Private Preview.</li></ul><p>Treat these as direction-of-travel rather than commitments, and confirm current status before you design around them.</p><h3>The recommended pattern, in one line</h3><p>The takeaway is a clean pipeline that replaces the old hand-rolled approach:</p><blockquote><strong><em>Data Classification (AI) → Governed Tags → automatic ABAC policies → audit via system tables</em></strong><em>, plus runtime PII guardrails for any GenAI workloads.</em></blockquote><p>You stop writing masking functions one table at a time. You let an AI agent find the sensitive data continuously, attach governed tags, and drive masking and row-level policies automatically from those tags — with a full audit trail and a centralized governance view on top.</p><h3>A few caveats worth stating</h3><ul><li><strong>Feature maturity varies by cloud.</strong> GA vs. Preview status for Data Classification and some ABAC capabilities differs across AWS, Azure, and GCP. Some Azure documentation still lists Data Classification as a workspace-level preview. Always confirm the exact status for <em>your</em> deployment before you design around it.</li><li><strong>Classification is a starting point, not a guarantee.</strong> The 60% accuracy lift over regex is real, but you’ll still want domain-specific review for edge cases, non-English data, and false-positive-sensitive contexts.</li><li><strong>Access to the results table is itself a PII risk.</strong> Sample values live there. Lock it down.</li></ul><p>The direction of travel is unmistakable, though: sensitive-data governance in Databricks is becoming something you <em>configure and supervise</em> rather than something you <em>script and maintain</em>. For a function that used to eat a meaningful slice of every data engineering team’s time, that’s a quietly significant shift.</p><p><em>Building on Databricks? Start with the Unity Catalog Data Classification documentation for your cloud (AWS, Azure, or GCP) and verify current GA/Preview status before you commit to a design.</em></p><h3>References &amp; Further Reading</h3><p><strong>Official documentation</strong></p><ul><li><a href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-classification">Data Classification in Unity Catalog — Databricks on AWS</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-classification">Data Classification — Azure Databricks (Microsoft Learn)</a></li><li><a href="https://docs.databricks.com/gcp/en/data-governance/unity-catalog/data-classification-tags">Supported classification tags (PII, GDPR, HIPAA, PCI DSS, GLBA, DPDPA) — Databricks on Google Cloud</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/data-classification-commands">data-classification CLI command group — Databricks CLI reference</a></li></ul><p><strong>Databricks blog &amp; product</strong></p><ul><li><a href="https://www.databricks.com/blog/find-sensitive-data-scale-data-classification-unity-catalog">Find Sensitive Data at Scale with Data Classification in Unity Catalog — Databricks Blog</a> (Jacqueline Li, Viswesh Periyasamy, Xiao Zhu — Oct 2025)</li><li><a href="https://www.databricks.com/blog/whats-new-unity-catalog-data-ai-summit-2026">What’s new with Unity Catalog at Data + AI Summit 2026 — Databricks Blog</a></li><li><a href="https://www.databricks.com/product/unity-catalog">Unity Catalog — product overview</a></li></ul><p><strong>Community</strong></p><ul><li><a href="https://community.databricks.com/t5/announcements/find-sensitive-data-at-scale-with-data-classification-in-unity/td-p/136750">Find Sensitive Data at Scale — Databricks Community announcement</a></li></ul><p><em>Note: feature availability (GA vs. Preview) and exact documentation paths differ across AWS, Azure, and GCP — open the page matching your own deployment and confirm the current status before designing around any feature.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52875de105f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why rclone is the CLI tool every cloud engineer should know]]></title>
            <link>https://fernandodenitto.medium.com/why-rclone-is-the-cli-tool-every-cloud-engineer-should-know-9bbed6b608cf?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bbed6b608cf</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-engineering]]></category>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 15:29:03 GMT</pubDate>
            <atom:updated>2026-03-28T15:29:03.669Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(And how it saved me from manually deleting thousands of objects from cloud storage)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ctwus8CMrOMIqK2g" /><figcaption>Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">CHUTTERSNAP</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you’ve ever tried to delete thousands of files from a cloud storage bucket using a web UI, you already know the pain.</p><p>No “select all.” Pagination hell. Timeouts. Clicks… endless clicks.</p><p>I recently ran into this problem while working with <strong>Cloudflare R2</strong>. I needed to empty a bucket before deleting it — and the UI simply wasn’t built for that scale.</p><p>Cloudflare’s own docs suggest using the AWS CLI. But I prefer a different tool, one I keep coming back to for all kinds of cloud storage tasks:</p><p>👉 <a href="https://rclone.org/"><strong>rclone</strong></a></p><h3>TL;DR</h3><ul><li>Use rclone to manage any object storage from your CLI</li><li>Works with S3, Cloudflare R2, GCS, Azure Blob, Backblaze B2, Google Drive, Google Photos, and <a href="https://rclone.org/#providers">many more</a></li><li>Delete an entire bucket’s content with a single command:</li></ul><pre>rclone delete r2:/your-bucket/ -P</pre><h3>The real problem: cloud UIs don’t scale</h3><p>Most object storage services share a common limitation:</p><p>❌ You cannot delete a bucket until it is empty ❌ But deleting thousands of objects via UI is painfully slow — or outright impossible</p><p>Even modern platforms like Cloudflare R2 lack a simple “Delete All” button.</p><p>Sure, you can reach for provider-specific CLIs (like the AWS CLI), but then:</p><ul><li>You need to learn a different tool for each provider</li><li>Commands differ subtly between them</li><li>Your scripts become vendor-locked</li></ul><h3>Enter rclone: one CLI to rule them all</h3><p>What makes rclone powerful isn’t just that it works — it’s that it works <strong>everywhere</strong>.</p><p>With one tool, you can manage:</p><ul><li>S3-compatible storage (AWS S3, Cloudflare R2, MinIO, Backblaze B2)</li><li>Google Cloud Storage</li><li>Azure Blob Storage</li><li>Google Drive, Dropbox, OneDrive</li><li><strong>Google Photos</strong> ← yes, even this</li></ul><p>I’ve personally used rclone to upload photos to Google Photos and pull files down from Google Drive. It goes way beyond object storage.</p><p>👉 <a href="https://rclone.org/#providers">Full list of providers</a></p><p>Think of it as:</p><blockquote><em>🧠 “kubectl for storage” — a universal adapter for cloud data.</em></blockquote><h3>Real example: emptying a Cloudflare R2 bucket</h3><p>Let’s walk through a real use case.</p><h3>1. Get your credentials</h3><p>From the Cloudflare R2 dashboard:</p><ul><li>Create an <strong>Account API Token</strong></li><li>Permissions: <strong>Object Read &amp; Write</strong></li><li>Save your:</li><li>Access Key ID</li><li>Secret Access Key</li><li>Endpoint URL</li></ul><h3>2. Configure rclone</h3><p>You can use the interactive wizard:</p><pre>rclone config</pre><p>Or define it manually in your rclone config file:</p><pre>[r2]<br>type = s3<br>provider = Cloudflare<br>access_key_id = YOUR_ACCESS_KEY<br>secret_access_key = YOUR_SECRET_KEY<br>endpoint = https://&lt;account&gt;.r2.cloudflarestorage.com</pre><p>Verify the remote is set up correctly:</p><pre>rclone listremotes --long</pre><h3>3. Empty the bucket</h3><pre>rclone delete r2:/your-bucket/ -P</pre><p>That’s it.</p><p>✔ Fast ✔ Parallel ✔ Works even with massive buckets</p><p>Example output:</p><pre>Deleted: 37 (files), 0 (dirs), 41.254 MiB (freed)<br>Elapsed time: 1.5s</pre><h3>Why this matters beyond this example</h3><p>Once you start using rclone, you unlock a whole set of powerful workflows:</p><p><strong>Cross-cloud sync</strong></p><pre>rclone sync s3:bucket-a gcs:bucket-b</pre><p><strong>Migrations without writing any code</strong></p><pre>rclone copy azure:container s3:backup</pre><p><strong>Local backups</strong></p><pre>rclone sync /local/data remote:backup</pre><p><strong>Bulk cleanup by age</strong></p><pre>rclone delete remote:old-data --min-age 30d</pre><h3>rclone vs. native CLIs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3h7cDDG_ugoLTDDpUSDdng.png" /><figcaption>Fast comparison from ChatGPT</figcaption></figure><h3>When rclone shines the most</h3><p>You’ll really appreciate rclone when:</p><ul><li>You work across multiple cloud providers</li><li>You need automation or CI/CD scripts</li><li>You handle large datasets</li><li>You want speed + simplicity without learning a new tool every time</li></ul><h3>Final thought</h3><p>Most engineers discover rclone accidentally.</p><p>But once you start using it, it becomes one of those tools you can’t live without.</p><p>The moment you type:</p><pre>rclone delete remote:bucket -P</pre><p>…and avoid 10,000 clicks, you’re never going back.</p><h3>Install rclone</h3><p>👉 <a href="https://rclone.org/install/">rclone.org — Installation guide</a></p><p><em>Found this useful? Follow me for more practical cloud engineering tips.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bbed6b608cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Embed Your Latest Medium Articles on Your Personal Website for Free]]></title>
            <link>https://fernandodenitto.medium.com/how-to-embed-your-latest-medium-articles-on-your-personal-website-for-free-3e2ffbbe0efc?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e2ffbbe0efc</guid>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[blogging]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 08:31:53 GMT</pubDate>
            <atom:updated>2026-03-25T08:31:53.235Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQ8fK1-uA-Czku2lUtnMjg.png" /><figcaption>My recent Medium Articles from my personal website</figcaption></figure><p>If you’re a developer writing on Medium, you probably want to showcase your latest posts on your personal portfolio or website.</p><p>While there are paid widgets and complex plugins available, the simplest solution is often the best. In this article, I’ll show you how to fetch and display your latest Medium articles using purely vanilla JavaScript — with zero heavy dependencies.</p><h3>The Problem with Medium’s API</h3><p>Medium does not offer a straightforward public API for retrieving a user’s articles. However, they <em>do</em> provide an RSS feed for every user, accessible at:</p><p><a href="https://medium.com/feed/@your_username">https://medium.com/feed/@your_username</a></p><p>The catch is that reading an XML feed directly in the browser using JavaScript can be tedious and is often blocked by CORS policies.</p><h3>The Solution</h3><p>To bypass CORS and the complexities of XML parsing, we can use a free service called <a href="https://rss2json.com/">rss2json</a>. This service takes an RSS feed URL and returns beautifully formatted JSON.</p><h3>Step 1: Clone the Repository</h3><p>I have prepared a minimal, ready-to-use boilerplate. You can grab the code from this GitHub repository:</p><p>👉<strong> </strong><a href="https://github.com/fernandodenitto/medium-embed-widget"><strong>Click here for the repository</strong></a></p><h3>Step 2: The HTML Container</h3><p>In your website’s HTML, create an empty container where the articles will be injected.</p><pre>&lt;div id=&quot;medium-articles-container&quot;&gt;<br>    &lt;p id=&quot;medium-articles-loading&quot;&gt;Loading latest articles from Medium...&lt;/p&gt;<br>&lt;/div&gt;</pre><h3>Step 3: The Code Logic</h3><p>The logic is incredibly simple. We fetch the JSON from rss2json, grab the first 3 items, and inject HTML elements into our container.</p><pre>const mediumUsername = &#39;YOUR_MEDIUM_USERNAME_HERE&#39;; <br>const rssUrl = `https://medium.com/feed/@${mediumUsername}`;<br>const apiUrl = `https://api.rss2json.com/v1/api.json?rss_url=${encodeURIComponent(rssUrl)}`;<br>fetch(apiUrl)<br>    .then(response =&gt; response.json())<br>    .then(data =&gt; {<br>        // ... loop through data.items and append HTML to the DOM<br>    });</pre><p><em>(Check the repo linked above for the full code!)</em></p><h3>Step 4: Customize!</h3><p>The boilerplate code is completely unopinionated regarding styling. You can add your own CSS classes (like Tailwind utility classes) directly inside the JavaScript template literals to match your website’s aesthetic perfectly.</p><h3>That’s It!</h3><p>With fewer than 50 lines of JavaScript, you now have an auto-updating section on your website that showcases your latest thoughts and tutorials.</p><p>Feel free to fork the repository and tweak it to your needs!</p><p>Happy coding, and happy writing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e2ffbbe0efc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Mastering Data Partitioning in Databricks with Spark and Delta Lake (Without Losing Your Mind)]]></title>
            <link>https://fernandodenitto.medium.com/mastering-data-partitioning-in-databricks-with-spark-and-delta-lake-without-losing-your-mind-0f4f2bef4e12?source=rss-70dd2ca04d3e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f4f2bef4e12</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[delta]]></category>
            <dc:creator><![CDATA[Fernando De Nitto]]></dc:creator>
            <pubDate>Wed, 21 May 2025 11:51:56 GMT</pubDate>
            <atom:updated>2025-05-21T11:56:51.894Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jWWidHfIdD-yFs5LgaQ8vw.png" /></figure><p>When working with big data on Spark — especially on Databricks with Delta Lake — <strong>partitioning</strong> is one of the most powerful and often overlooked techniques for optimizing performance and controlling costs.</p><p>But what exactly is a partition? How do you create one? And more importantly: how do you use it correctly?</p><p>💡 One way to think about it: <strong>partitioning is like organizing a massive bookshelf</strong>.</p><p>If every book is just thrown on the floor (no partitions), finding what you need takes forever. But if books are neatly arranged by year and month, you can go directly to the right shelf — no scanning required. That’s the power of partitioning: smarter layout, faster access.</p><p>Let’s break it all down with clear explanations, simple examples, and real-world best practices.</p><h3>🔍 What is a Partition?</h3><p>Think of a partition as a <strong>subset of your data</strong> where all rows share the same value for a specific column. It’s also a physical layout trick: when Spark writes partitioned data, it <strong>stores it in separate folders</strong> based on that column’s value.</p><h3>Example Table:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*PQFcrgIV26AKYjgUvzJWfw.png" /></figure><p>If we partition this table by year, it’ll be stored like this:</p><pre>/year=2023/<br>/year=2024/</pre><p>So when querying data for a specific year, Spark will only look inside the relevant folder — skipping the rest. This is called <strong>partition pruning</strong> or <strong>partition skipping</strong>.</p><h3>🛠 How to Create a Partitioned Delta Table</h3><p>Creating a partitioned Delta Table in Spark is straightforward.</p><p>Using PySpark:</p><pre>df.write.format(&quot;delta&quot;) \<br>  .partitionBy(&quot;year&quot;) \<br>  .mode(&quot;overwrite&quot;) \<br>  .save(&quot;/mnt/datalake/events&quot;)</pre><p>Using SQL in Databricks:</p><pre>CREATE TABLE events<br>USING DELTA<br>PARTITIONED BY (year)<br>AS SELECT * FROM bronze_events;</pre><p>The data will be physically stored in /mnt/datalake/events/year=2023/, /year=2024/, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GN-Un1DW6-kHOIaOcg8O2w.png" /></figure><h3>🚀 Partition Skipping in Action</h3><p>Let’s say you want to fetch data only for 2023:</p><pre>SELECT * FROM events WHERE year = 2023;</pre><p>Spark will only scan the partition year=2023, skipping everything else — this can make queries significantly faster, especially on large datasets.</p><h3>🤔 What if I Filter on a Non-Partitioned Column?</h3><p>Suppose you filter by name instead:</p><pre>SELECT * FROM events WHERE name = &#39;Login&#39;;</pre><p>Spark must scan all partitions because it doesn’t know which one contains ‘Login’. That’s where <strong>Z-Ordering</strong> comes into play.</p><h3>🧠 Optimize + Z-Order = File Skipping Power</h3><p>While partitions help at the folder level, <strong>Z-Ordering</strong> improves performance inside the files.</p><p>You can use:</p><pre>OPTIMIZE events ZORDER BY (name);</pre><p>This clusters similar values close together inside the files, allowing <strong>data skipping at the block level</strong>.</p><p>If you’re interested in optimisation techniques, check <a href="https://medium.com/data-reply-it-datatech/mastering-delta-lake-optimizations-optimize-z-order-and-vacuum-f6ca110c9869">my other story</a>!</p><h3>🧩 Multi-Column Partitioning</h3><p>Need more granularity? You can partition on multiple columns, like year and month.</p><pre>df.write.format(&quot;delta&quot;) \<br>  .partitionBy(&quot;year&quot;, &quot;month&quot;) \<br>  .save(&quot;/mnt/datalake/events&quot;)</pre><p>This creates a structure like:</p><pre>/year=2024/month=5/<br>/year=2023/month=6/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4b--N3S7WSzu7gcT6j1YNA.png" /></figure><p>Each combination becomes its own physical directory.</p><h3>⚠️ The Danger of Over-Partitioning</h3><p>Partitioning can backfire if done incorrectly. If you choose high-cardinality columns (like id or user_id), you’ll create <strong>thousands of tiny folders</strong>.</p><h3>Example of Bad Partitioning:</h3><pre>df.write.partitionBy(&quot;id&quot;).save(&quot;/mnt/data&quot;)</pre><p>Now imagine millions of ids — your file system and query planner will struggle.</p><h4>How to Spot Over-Partitioning:</h4><ul><li>Thousands of partitions with less than 1 GB of data</li><li>Slower-than-expected queries</li><li>OPTIMIZE takes forever</li></ul><h4>How to Fix It:</h4><ul><li>Use low-cardinality columns (e.g., year, region, event_type)</li><li>Make sure each partition has at least <strong>1 GB of data</strong></li><li>Use OPTIMIZE regularly</li></ul><h3>🧹 DELETE with Partitions and Streaming: What You Need to Know</h3><p>Let’s say you want to remove all 2023 data:</p><pre>DELETE FROM events WHERE year = 2023;</pre><p>This operation <strong>doesn’t delete the data immediately</strong>. Delta Lake updates the <strong>metadata</strong>, but the files remain accessible via <strong>time travel</strong>:</p><pre>SELECT * FROM events VERSION AS OF 5;</pre><p>To remove the files physically:</p><pre>VACUUM events RETAIN 168 HOURS;</pre><h3>⚠️ Important for Streaming:</h3><p>If you’re using readStream and someone performs a DELETE, it can cause errors because streaming is <strong>append-only</strong> by default.</p><p>To prevent this, add the following option:</p><pre>spark.readStream \<br>  .format(&quot;delta&quot;) \<br>  .option(&quot;ignoreDeletes&quot;, &quot;true&quot;) \<br>  .load(&quot;/mnt/datalake/events&quot;)</pre><h3>✅ How to Choose the Right Partition Columns</h3><p>Here are some general rules:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*7sCiyVBCRG6Vp72FYhFKBA.png" /></figure><h3>📚 Recap: Think of Partitioning Like a Well-Organized Bookshelf</h3><p>To wrap things up, one of the easiest ways to understand data partitioning is to imagine a <strong>bookshelf</strong>.</p><p>If your data isn’t partitioned, it’s like walking into a bookshelf where <strong>every book is thrown into one big pile</strong>. To find what you’re looking for, you’d have to check every single book — slow, messy, and frustrating.</p><p>Now picture a properly organized bookshelf:</p><ul><li>Books are shelved by <strong>year</strong> (year=2023, year=2024)</li><li>Within each shelf, they’re grouped by <strong>month</strong> (month=5, month=6)</li><li>You can walk directly to the exact section and find the book you need — <strong>quickly and efficiently</strong></li></ul><p>That’s exactly what <strong>partitioning does for your data</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*MFfYUZRmkmg0aXlYWGoofw.png" /></figure><p>By organizing your data like a smart librarian, you enable Spark to <strong>scan less</strong>, <strong>work faster</strong>, and <strong>save resources</strong> — all while keeping things clean and scalable.</p><h3>🧾 TL;DR</h3><p>Partitioning is like organizing a bookshelf. If the books are sorted by year, it’s easy to find what you need. If they’re all over the place — good luck.</p><p>With the right strategy, partitioning can lead to <strong>faster queries, reduced costs</strong>, and simpler data operations.</p><h3>Key takeaways:</h3><ul><li>✅ Partition smartly with low-cardinality columns</li><li>✅ Use OPTIMIZE + ZORDER for even better performance</li><li>⚠️ Don’t over-partition — check partition sizes</li><li>⚠️ Handle DELETEs carefully if streaming is involved</li></ul><h3>📚 Further Reading</h3><ul><li>📘 <a href="https://docs.databricks.com/delta/partitioning.html">Delta Lake Partitioning — Databricks Docs</a></li><li>⚙️ <a href="https://docs.databricks.com/delta/optimize.html">Optimize and Z-Ordering</a></li><li>🔄 <a href="https://docs.databricks.com/structured-streaming/delta-lake.html#ignorechanges-and-ignoredeletes">Delta Lake Streaming &amp; Deletes</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f4f2bef4e12" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>