Introducing Forward Deployed Engineer on Databricks
A Career, Built Layer by Layer
Three volumes. Eight milestones. 156 topics. Nearly 1,000 pages. One company you will rebuild from a Python script to an AI platform. Today it is live.
It’s finally here.
I am excited to announce Forward Deployed Engineer on Databricks — a complete e-book series that takes you from your first line of Python to production pipelines, real-time streaming, AI/BI dashboards and generative AI on the lakehouse.
It is available now on Topmate.
But before I tell you what is inside, let me tell you why it exists.
There is a moment almost every data engineer goes through. You have finished the courses. You know the syntax. You have the certificates. Then you land in front of a real client, with real data and a real deadline — and you realise the tutorials prepared you for the exam, not for the room.
The client does not ask you to write a groupBy. They ask why last night's job took three hours instead of twenty minutes. They ask whether they can trust the revenue number on the CEO's dashboard. They ask how to move forty legacy Hive tables to Databricks without breaking finance.
This series was written for that room.
Wait — what is a Forward Deployed Engineer?
A Forward Deployed Engineer is the person who sits closest to the problem.
Not behind a ticket queue. Not three teams away from the business. Right there with the client — understanding what they actually need, designing the solution, building it on the platform, and staying until it works in production.
On Databricks, that means one person who can:
- Read a messy source system and model it properly
- Write the Python, SQL and Spark that move the data
- Build a governed lakehouse that finance will actually trust
- Stream real-time events when batch is not fast enough
- Put dashboards and Genie in front of business users
- Ship ML and generative AI on top of all of it
That is a rare combination. Most learning paths teach one slice. This series teaches the whole stack — in the order you will need it.
Why I wrote it
Over the years, I have mentored a lot of engineers. The pattern repeats.
Someone learns pandas in one course, SQL in another, Spark in a third, and Databricks from a set of disconnected videos. Every course uses a different dataset. Every course starts from zero. Nobody ever shows how the pieces connect.
So when they meet a real platform, they see eight separate tools instead of one system. And when something breaks, they do not know which layer it came from.
I wanted a path that works the way real platforms are built: layer by layer, each one resting on the last.
That is why the entire series follows one fictional company — RetailCo — and the same three business questions from start to finish:
- Which cities and categories drive revenue?
- Which customers are at risk?
- Can we trust the data?
In Milestone 1 you answer them with a Python script. By Milestone 8 you answer them with a governed lakehouse, streaming pipelines, dashboards, Genie and AI. Same company. Same questions. A bigger engine every time.
The building, layer by layer
Think of the series as a lakehouse. Raw foundations at the bottom. Refined engineering in the middle. Business-ready intelligence at the top. You cannot skip a floor — and you will not want to.
Phase 1 — Foundations (the Bronze layer)
Milestone 1 · Python for Data Engineering (24 topics). Not “Python for beginners.” Python for people who move data: files, JSON, APIs, pandas, error handling, logging and production-style code — ending with your first mini ETL pipeline.
Milestone 2 · SQL for Data Engineering (24 topics). From schema design to window functions, MERGE and slowly changing dimensions. The language every data platform still runs on.
Milestone 3 · Data Modeling & Git (20 topics). Facts, dimensions, star schemas and the medallion architecture — plus the Git, pull requests and CI checks that separate a hobby project from a team project.
Phase 2 — Big Data & Lakehouse (the Silver layer)
Milestone 4 · Big Data & Apache Spark (24 topics). Where your pipeline learns to scale. More on this one below.
Milestone 5 · Lakehouse on Databricks (18 topics). Delta Lake internals, Unity Catalog, governance, Auto Loader, Lakeflow pipelines and jobs, bundles and CI/CD, and what all of it costs.
Milestone 6 · Real-time Data Engineering (16 topics). Kafka concepts, Structured Streaming, watermarks, exactly-once, CDC and real-time Lakeflow pipelines — because the business stopped waiting for tomorrow’s batch.
Phase 3 — Analytics & AI (the Gold layer)
Milestone 7 · AI/BI Dashboards, Metric Views & Genie (12 topics). Serving the lakehouse to the people who make decisions — dashboards, alerts, metric views and Genie agents that business users can actually trust.
Get THE BRICK LEARNING’s stories in your inbox
Join Medium for free to get updates from this writer.
Milestone 8 · Machine Learning & Generative AI (18 topics). MLflow, feature tables, model serving, AI Functions in SQL, embeddings, RAG, agents, Agent Bricks, Databricks Apps and Lakebase — and how to keep all of it governed and responsible.
Every milestone ends with a capstone that upgrades RetailCo. By the last page, you have not read about a modern data platform. You have built one.
Sneak peek: inside Milestone 4 — Big Data & Apache Spark
If you want to know what “layer by layer” feels like in practice, Milestone 4 is the best example.
Most Spark courses start with spark.read.csv(). Ten minutes later you have a DataFrame — and no idea what happens when it meets 2 billion rows. This milestone builds Spark the way it should be learned: foundation first.
Layer 1 — Why big data exists (Topics 1–5). What breaks when data outgrows one machine, and the twenty-year road from Google’s papers to Hadoop, Spark, Delta Lake and Spark 4. Then the story every engineer should tell in one breath: warehouse → data lake → lakehouse.
Layer 2 — How Spark thinks (Topics 6–11). Driver, executors and SparkSession. RDDs, and why DataFrames replaced them. A bridge from pandas, so nothing you learned in Milestone 1 is wasted. And lazy evaluation — the moment Spark stops feeling random. You write ten transformations, they finish instantly, then .count() takes two minutes. Nothing was broken. Spark was building a plan.
Layer 3 — Pipelines that survive real data (Topics 12–18). Aggregations and windows, joins and skew, missing data, timestamps, nested JSON, UDFs vs pandas UDFs, and Spark SQL side by side with Python.
Layer 4 — Performance (Topics 19–23). Reading query plans, partitioning, shuffles, broadcast joins, AQE, caching on serverless, file formats and the small-files problem. This is the layer that gets you trusted with production.
Layer 5 — Build it yourself (Topic 24). A 6–8 hour capstone that rebuilds RetailCo end to end in PySpark: raw files into Bronze, cleaned and validated data with a quarantine table into Silver, business aggregates into Gold.
Then comes the Exercise Book: 50 questions, about 15 hours. The five scenario challenges are the ones I love most, because every one of them is a situation you will meet on a real project:
- A nightly job that joins 2 billion rows has grown from 20 minutes to 3 hours. Find out why, and fix it.
- A legacy Hadoop platform with 40 Hive tables needs a migration plan to Databricks.
- A partner starts sending files with text amounts like “3,800” and nested discount objects. Lose nothing, keep Silver strictly typed.
- A data scientist’s pandas notebook takes hours on 30 million orders. Rewrite it in PySpark.
- Make your capstone safely re-runnable for any date with a deduplicated MERGE.
That is not a quiz. That is a Tuesday on any data team — and exactly the kind of story that wins interviews.
What you get in every milestone
Every milestone follows the same rhythm, so you always know where you are:
- Overview slides with speaker notes — the whole milestone on a few pages, before you dive in
- Setup scripts — run once in Databricks and your RetailCo data is ready
- Step-by-step topics — learning objectives, why it matters on the job, core concepts, runnable code and tips from real projects
- A capstone — RetailCo gets upgraded with everything you just learned
- An Exercise Book — theory, practicals at two levels, and scenario challenges taken from real client situations
You commit your work to your own GitHub portfolio as you go. By the end, you do not just say you know Databricks. You can show it.
The series by the numbers:
- 3 volumes
- 8 milestones
- 156 topics
- 8 capstone projects
- Nearly 1,000 pages
- 1 company, rebuilt from a script to an AI platform
Who it is for — and where to start
Just starting out, or switching from analytics or software? Start with Volume 1. It assumes nothing and builds the foundation properly.
Already writing Spark but still guessing when jobs slow down? Jump into Volume 2, starting with Milestone 4.
Building on Databricks and asked to deliver dashboards, Genie or AI? Volume 3 is where the business value lives.
Aiming for a Forward Deployed Engineer or client-facing data role? Take the whole climb. That is what it was designed for.
Available now on Topmate
Forward Deployed Engineer on Databricks is live today. Grab the complete series, or start with the volume that matches where you are.
Get it on Topmate:
The next time a client asks why the job is slow, whether the numbers can be trusted, or how to put AI on top of their data — I want you to be the engineer in the room who already knows the answer. Because you built every layer yourself.
If this launch excites you, give it a clap, share it with one engineer who needs it, and follow me for deep dives into each milestone. And tell me in the comments: which layer are you climbing right now?
Tags: Databricks · Data Engineering · Apache Spark · Lakehouse · Generative AI
