<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manish Pansari on Medium]]></title>
        <description><![CDATA[Stories by Manish Pansari on Medium]]></description>
        <link>https://medium.com/@Manish_Pansari?source=rss-34a13a391539------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uQj-AjmdAYYak_dbA7vQLg.jpeg</url>
            <title>Stories by Manish Pansari on Medium</title>
            <link>https://medium.com/@Manish_Pansari?source=rss-34a13a391539------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:19:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Manish_Pansari/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Databricks Lakeflow Designer]]></title>
            <link>https://medium.com/@Manish_Pansari/hey-everyone-if-you-dont-know-me-yet-feel-free-to-check-out-my-linkedin-profile-manish-pansari-994b7c94b67f?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/994b7c94b67f</guid>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 13:41:56 GMT</pubDate>
            <atom:updated>2026-06-30T13:44:03.743Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><p><strong>Introduction :</strong></p><p>Every few months, a “low-code data engineering” tool shows up promising to finally bridge the gap between analysts who know the data and engineers who know how to move it. You get excited. You build a couple of toy pipelines. And then — right around the time you try to do something real — it falls apart. The abstraction breaks, the generated code is a nightmare, and you’re worse off than if you’d just written a Python script to begin with.</p><p>So when Databricks announced Lakeflow Designer, my first instinct was a familiar mix of curiosity and skepticism. I’d heard this pitch before.</p><p><strong>What Even Is Lakeflow? :</strong></p><p>If you’ve been following the Databricks world, you’ve probably noticed they’ve been consolidating their data engineering offerings under a unified brand called <strong>Lakeflow</strong>.</p><p>Lakeflow is the umbrella — and under it you’ll find:</p><ul><li><strong>Lakeflow Pipelines</strong> (what used to be Delta Live Tables — DLT)</li><li><strong>Lakeflow Jobs</strong> (the evolution of Databricks Workflows)</li><li><strong>Lakeflow Designer</strong> — the visual, canvas-based pipeline builder that’s the focus of this piece</li></ul><p>The key thing to understand is that Lakeflow Designer isn’t a standalone product that competes with Pipelines. It <em>generates</em> Lakeflow Pipelines.</p><p><strong>What Does Lakeflow Designer Actually Do?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ycAYhe7WJMcG-kvx1kqoQ.png" /></figure><p>At its core, Lakeflow Designer gives you a visual canvas where you can design an end-to-end data pipeline by connecting boxes. You pick a source, add transformation steps, define your targets, and the tool generates the Delta Live Tables code for you.</p><p>But that sentence doesn’t quite capture what makes it feel different in practice.</p><p>When I dropped in a source table and started adding transformations, the first thing I noticed was that the UI wasn’t fighting me. The drag-and-drop felt intentional. You can see your full data lineage laid out on the canvas — where data comes from, how it flows through each transformation, where it lands. It looks a bit like those architecture diagrams we all draw on whiteboards before writing a single line of code. Except this one actually runs.</p><p>You can write SQL transformations inline. You preview data at each node before committing to anything. Data quality expectations — one of the best features DLT brought to the table — are baked right in. You define your constraints visually, and they get enforced as part of the pipeline execution.</p><p>What you’re building isn’t a mock-up. It’s real, production-ready code.</p><p><strong>The Part That Actually Impressed Me :</strong></p><p>Here’s where it gets interesting. After building a pipeline visually, I clicked through to see the generated code.</p><p>It was readable. Clean. The kind of code I’d actually commit to a repository without wanting to disappear.</p><p>This is what most visual pipeline tools get catastrophically wrong. They either generate unmaintainable spaghetti that nobody wants to touch, or they lock your logic inside a proprietary format that’s impossible to version-control or review. Lakeflow Designer generates standard Delta Live Tables Python and SQL. Your engineers can open it, understand it, modify it, and own it.</p><p>That changes the conversation entirely. You’re not choosing between “visual tool for non-engineers” and “code for real engineers.” You’re choosing a faster way to get to code that real engineers can maintain. That’s a meaningfully different value proposition.</p><p><strong>Who Is This Actually For?</strong></p><p><strong>For data engineers</strong>, Lakeflow Designer is a rapid prototyping tool. You can sketch out a complex pipeline in minutes, see the data lineage clearly, and iterate on transformation logic without jumping between notebooks.</p><p><strong>For analytics engineers and senior analysts</strong>, it’s a path to building and owning pipelines without needing someone to translate your SQL into a proper engineering format. If you know SQL well enough, you can define transformations, set quality rules, and ship something to production. You’re not blocked waiting for a data engineer’s bandwidth.</p><p><strong>For data teams trying to move faster</strong>, it reduces the distance between “we need this pipeline” and “this pipeline runs in production.” That’s a real operational win, especially in organizations where the ratio of people who understand the data to people who can engineer pipelines is painfully skewed.</p><p><strong>Unity Catalog Integration :</strong></p><p>One thing worth calling out specifically: Lakeflow Designer is deeply integrated with Unity Catalog, Databricks’ unified governance layer.</p><p>What this means in practice is that when you’re building a pipeline, you’re operating within the full context of your organization&#39;s data assets. You can browse your catalog to find source tables, see column-level metadata, and understand data lineage that extends beyond just the pipeline you’re building. When the pipeline runs, lineage is automatically tracked and visible in Unity Catalog.</p><p><strong>Honest Limitations :</strong></p><p>No tool is perfect, and I’d be doing you a disservice if I pretended otherwise.</p><p><strong>Complex transformations get awkward.</strong> Inline SQL in a visual node works great for straightforward transformations. When your logic gets complex — think multi-level CTEs, dynamic schemas, or Pythonic transformations that benefit from being in a proper notebook — the visual abstraction starts to feel constraining rather than helpful. The ability to switch to a full notebook or edit the generated code directly is there, but it means you’re stepping out of the visual flow anyway.</p><p><strong>It’s a Databricks-native tool.</strong> If your data platform spans multiple clouds or tools, Lakeflow Designer won’t be your pipeline for everything. It works within the Databricks ecosystem.</p><p><strong>The Bigger Picture :</strong></p><p>The data engineering world has been grappling for years with a fundamental tension: the people who understand a business’s data the best are rarely the same people with the time and skills to build robust pipelines. Lakeflow Designer feels like Databricks’ serious attempt to push that boundary further, without sacrificing the production engineering standards that actually matter.</p><p>The fact that it generates real, editable, version-controllable code — and sits on top of Delta Live Tables, a genuinely solid execution engine — gives me confidence that this is more than a demo trick. It’s a tool that respects the engineers who will eventually maintain whatever you build with it.</p><p>If you’re already on Databricks and haven’t kicked the tyres on Lakeflow Designer yet, it’s worth an afternoon. Start with a pipeline you already have in production and try to recreate it visually. The process of doing that alone — mapping out your data flow on a canvas — is a useful exercise, regardless of whether you ship the visual version.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=994b7c94b67f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Sharing in Databricks]]></title>
            <link>https://medium.com/@Manish_Pansari/delta-sharing-in-databricks-a3496c97f0c4?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3496c97f0c4</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[delta-share]]></category>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 11:56:31 GMT</pubDate>
            <atom:updated>2026-04-01T11:56:31.001Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><p><strong>Introduction :</strong></p><p>I’ll be honest with you. The first time someone asked me to share a dataset with an external team, I did what most data engineers do — I exported it to CSV, dropped it in an S3 bucket, slapped an IAM policy on it, and sent an email with seven bullet points of instructions. Three days later they still couldn’t access it. We’ve all been there.</p><p>That’s the problem Delta Sharing was built to solve. Not just “make it easier,” but <strong>fundamentally rethink</strong> how data crosses organizational boundaries without losing your sanity — or your security posture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*oi_ecZY5RPVIm2tq50BrVQ.png" /></figure><p><strong>So, what exactly is Delta Sharing?</strong></p><p>At its core, Delta Sharing is an open protocol — think of it as a universal handshake for data. It lets you share live Delta Lake tables with anyone, on any platform, without copying the data, without giving them access to your cloud storage, and without requiring them to run Databricks.</p><p>That last part is the kicker. Your recipient could be using Python on their laptop, Tableau on a Windows machine, or a completely different cloud. It doesn’t matter. The data comes to them, securely, through a clean REST API.</p><p>The architecture diagram above tells the whole story visually. The data provider defines what to share, wraps it in a <strong>Share </strong>object, assigns <strong>Recipients</strong> with token-based credentials, and the Sharing Server acts as the secure middleman. Recipients get pre-signed URLs that let them read files directly from your storage — but they never see your storage credentials. It’s a beautifully simple idea.</p><p><strong>The three building blocks</strong></p><p>Before you write a single line of code, you need to understand three objects. Think of them like setting up a lending library:</p><p><strong>The Share</strong> is your curated shelf — a named collection of Delta tables you’ve chosen to expose. You control exactly which tables, which schemas, and even which partitions go into it.</p><p><strong>The Recipient</strong> is the library card — a named identity backed by a bearer token (or Unity Catalog identity for Databricks-to-Databricks sharing). Hand someone this token, and they can read your share. Revoke it anytime.</p><p><strong>The Sharing Server</strong> is the librarian — it validates the token, enforces permissions, and returns pre-signed URLs so the reader can grab the actual Parquet files from cloud storage directly.</p><p><strong>Setting it up in Databricks (Unity Catalog)</strong></p><p>If you’re on Databricks with Unity Catalog enabled, setup is refreshingly clean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*WAMz8wUDiKeggjwo_BWBhg.png" /></figure><p>That’s genuinely it on the provider side. No S3 policies, no VPC peering, no firewall rules. You create a share, add tables, create a recipient, grant access. Done.</p><p><strong>What the recipient sees</strong></p><p>Here’s the part that makes this protocol special. The recipient doesn’t need Databricks at all. They download a small credential file (a `.share` file in JSON format) from the activation link and then use it however fits their workflow:</p><p><strong>If they’re a Python/Pandas user:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*ZIIdkrxc05Ube_ixiIpmzA.png" /></figure><p><strong>If they’re on another Databricks workspace</strong> (Databricks-to-Databricks):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*B3PngZZEE16dueHVEsz7mA.png" /></figure><p>The recipient sees a live view. When you update your table, they see the update. No stale CSV exports. No “can you resend the file with last week’s numbers?” emails.</p><p><strong>The security model — and why it’s actually clever</strong></p><p>Here’s what I appreciate most about Delta Sharing from a security standpoint. Let me walk you through what *doesn’t* happen:</p><p>- Your recipient never gets your AWS/Azure/GCP credentials<br>- Your recipient never gets direct access to your storage bucket<br>- You never copy data to a “sharing” bucket<br>- You never open firewall ports or set up VPN tunnels</p><p>What <strong>does</strong> happen: the Sharing Server generates short-lived pre-signed URLs (think of them like temporary, one-use download links) scoped to exactly the files the recipient is allowed to read. These expire. They’re single-purpose. If the token is compromised, you revoke it immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*kE1b_TMR0-GqPJPZVon5ew.png" /></figure><p>You also get full audit trails through Unity Catalog — every read, every table access, every token use is logged. Try auditing that CSV-in-S3 approach.</p><p><strong>A real-world scenario that changed how I think about this</strong></p><p>Imagine you’re a fintech company. You process transaction data daily and have a regulatory requirement to share aggregated fraud signals with a consortium of banks — each running a different cloud, a different tech stack.</p><p>Old way: Export → encrypt → SFTP → hope they can parse your schema → repeat weekly. Someone always gets last week’s file. Your compliance team sends reminders. Everybody is miserable.</p><p><strong>Delta Sharing way: </strong>One share, five recipients (one per bank), each with their own token. They consume live data in whatever tool they already use. When your schema evolves, you update the share. They see it immediately. Your compliance dashboard shows exactly who accessed what and when.</p><p>That’s not just a technical improvement — it changes the <strong>relationship</strong> between data teams.</p><p><strong>Limitations you should know going in :</strong></p><p>Delta Sharing is genuinely powerful, but it’s not magic. A few honest caveats:</p><p>The protocol is <strong>read-only</strong>. Recipients can read your data but cannot write back. If you need bidirectional data exchange, you’ll need a different approach for the write path.</p><p><strong>Row-level security is coarse-grained</strong>. You can filter by partition, but you can’t easily say “this recipient sees only rows where `customer_country = ‘IN’`” without pre-creating a view that does the filtering and sharing that instead.</p><p><strong>Latency is not real-time</strong>. Recipients read Parquet files, not a streaming feed. If you need sub-second freshness, Delta Sharing isn’t your answer — but for most analytics use cases, the latency of your Delta table commit cycle is perfectly fine.</p><p><strong>The bigger picture:</strong></p><p>Delta Sharing is part of a broader shift that I find genuinely exciting — the move toward treating data as a product that you <strong>publish</strong> rather than files that you <strong>send</strong>. It’s the same philosophy behind data mesh, but with a concrete, working implementation underneath it.</p><p>When I first tried it, I expected another enterprise feature with ten configuration screens and a three-day setup. Instead I had a live share working in about fifteen minutes, and a colleague on a completely different platform reading it in Python within the hour.</p><p>Sometimes the best technology is the kind that just quietly gets out of your way and lets you do the actual work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3496c97f0c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide to Mastering the Databricks Data Engineer Spark Associate Exam (2025 Update)]]></title>
            <link>https://medium.com/@Manish_Pansari/a-guide-to-mastering-the-databricks-data-engineer-spark-associate-exam-2025-update-ec47fba11bd3?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec47fba11bd3</guid>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Mon, 24 Nov 2025 12:03:43 GMT</pubDate>
            <atom:updated>2025-11-24T12:03:43.911Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><h3><strong>Introduction :</strong></h3><p>Databricks recently announced the retirement of the Databricks Certified Associate Developer for Apache Spark 3.0 exam, effective April 15, 2025. While this initially caused some confusion among learners, the updated certification actually reflects the modern expectations of today’s data engineering landscape. With stronger emphasis on Structured Streaming, Spark SQL, performance tuning, and the Pandas API on Spark, the new exam aligns far more closely with real-world workloads in the Lakehouse platform.</p><h3><strong>What Has Changed?</strong></h3><ul><li>Databricks made significant updates to the exam to align it with modern workloads. Based on my research and preparation (note: I haven’t taken either version yet), here’s a simple difference-map between the <strong>retired exam</strong> and the <strong>new exam</strong>:</li><li>Key Differences Between the Old &amp; New Exam</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kEotpF1Pu1gkFMP0wsOz8A.png" /></figure><ul><li>This shift clearly shows that Databricks wants practitioners to be more aligned with <strong>streaming, optimization, and modern APIs </strong>— the skills used daily in real Databricks environments.</li></ul><h3><strong>Syllabus Breakdown for the New Exam (2025) :</strong></h3><p>Here is the official breakdown from Databricks:</p><p><strong>1.Apache Spark Architecture and Components — 20%</strong><br> Execution modes, lazy evaluation, fault tolerance, shuffle mechanics, broadcast, GC, and cluster fundamentals.</p><p><strong>2.Using Spark SQL — 20%</strong><br> SQL queries, functions, window operations, joins, and optimizations.</p><p><strong>3.Developing Spark DataFrame/Dataset API Applications — 30%</strong><br> Schema handling, data cleaning, transformations, aggregations, joins, reading/writing formats, UDFs, partitioning.</p><p><strong>4.Troubleshooting and Tuning — 10%</strong><br> Common errors, skew fixes, caching, physical plans, and performance patterns.</p><p><strong>5.Structured Streaming — 10%</strong><br> Micro-batching, triggers, watermarks, sources/sinks, streaming joins, checkpoints.</p><p><strong>6. Using Spark Connect — 5%</strong><br> Architecture, client/server model, Python client usage.</p><p><strong>7. Pandas API on Spark — 5%</strong><br> Pandas-style operations at scale, interoperability, when to use it.</p><h3><strong>Why These Changes Matter :</strong></h3><p>The Databricks ecosystem has evolved massively over the last few years. Streaming, ML-ready data, BI workloads, and AI-driven pipelines are at the core of modern data jobs. It makes sense that the certification reflects those expectations.</p><p>Some of the biggest modern changes Spark engineers encounter today include:</p><p>1. Transition from batch-only pipelines → <strong>continuous streaming/near-real-time systems</strong><br>2. Unified analytics at scale → <strong>Pandas API on Spark</strong><br>3. Distributed SQL workloads → <strong>Spark SQL advancements</strong><br>4. Lightweight client/server execution → <strong>Spark Connect</strong></p><p>If you’ve been working in Databricks or Spark environments recently, these updates will feel very natural.</p><h3><strong>Exam Details (2025 Version) :</strong></h3><p><strong>Total Scored Questions :</strong> 45<br><strong>Time :</strong> 90 minutes<br><strong>Format :</strong> Multiple choice<br><strong>Fees : </strong>$200<br><strong>Language :</strong> English<br><strong>Mode :</strong> Online or test center<br><strong>Validity :</strong> 2 years<br><strong>Recommended Experience :</strong> 6+ months hands-on Spark DataFrame work<br><strong>Prerequisites :</strong> None (but training is recommended)</p><h3><strong>My Preparation Strategy (Updated) :</strong></h3><p>Initially, my plan was strongly DataFrame-oriented — the older exam’s primary focus.<br>But with the updated syllabus, I’ve added deeper study in:</p><blockquote>Streaming concepts: watermarks, triggers, stateful processing<br>Spark SQL tuning and joins<br>Catalyst and physical plans<br>Pandas API migration patterns<br>Spark Connect architecture</blockquote><h3>Practical Exam Tips :</h3><p><strong>Familiarize Yourself with the Platform:</strong></p><blockquote><em>Use the Databricks Academy course or the Databricks Free Edition to get hands-on experience.</em></blockquote><p><strong>Mark and Review:</strong></p><blockquote><em>Quickly cover all questions in one round, marking those you’re unsure about for review. In the second round, revisit these marked questions and ensure no question is left unanswered before submitting.</em></blockquote><h3><strong>Summary :</strong></h3><p>Overall, the updated exam better reflects the reality of modern data engineering. It rewards practical skills that engineers use every day in the Databricks Lakehouse: streaming, SQL, optimization, and distributed computation.</p><h3>Reference:</h3><ul><li><a href="https://www.databricks.com/learn/certification/apache-spark-developer-associate">https://www.databricks.com/learn/certification/apache-spark-developer-associate</a></li><li>Practice Exam : <a href="https://www.udemy.com/course/databricks-certified-associate-developer-for-apache-spark-python/">https://www.udemy.com/course/databricks-certified-associate-developer-for-apache-spark-python/</a></li><li>Practice Exam : <a href="https://www.udemy.com/course/databricks-certified-data-engineer-associate-ultimate-practice-exams/">https://www.udemy.com/course/databricks-certified-data-engineer-associate-ultimate-practice-exams/</a></li><li><a href="https://www.databricks.com/sites/default/files/2025-04/Databricks-Certified-Associate-Developer-for-Apache-Spark_Exam-Guide_1_Apr_2025.pdf">https://www.databricks.com/sites/default/files/2025-04/Databricks-Certified-Associate-Developer-for-Apache-Spark_Exam-Guide_1_Apr_2025.pdf</a></li></ul><p>By following this guide and leveraging the recommended resources, you’ll be well-prepared to tackle the Databricks Data Engineer Spark Associate Exam and achieve certification. Good luck!</p><p><em>This article should provide a comprehensive and engaging read for anyone preparing for the Databricks Data Engineer Spark Associate Exam. If you have reached this far, thank you for reading this blog, I hope you found it insightful</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec47fba11bd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide to Mastering the Databricks GenAI Associate Exam]]></title>
            <link>https://medium.com/@Manish_Pansari/a-guide-to-mastering-the-databricks-genai-associate-exam-8793d76a2786?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/8793d76a2786</guid>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[genrative-ai]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[databricks-certification]]></category>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Sat, 19 Jul 2025 12:03:05 GMT</pubDate>
            <atom:updated>2025-07-19T12:05:19.993Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><p>Let me guess — you heard about the Databricks Certified Generative AI Engineer Associate exam and thought,<br>“It’s just another AI certification… How hard can it be?”</p><p>Well, spoiler alert: It’s not rocket science… but it will make you wish you had read those documentation pages you always skip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rf3WRCWjzMfoTW1UQm_nrg.jpeg" /></figure><p>Having recently earned this certification, I thought I’d share what actually matters — a practical guide with key topics, study resources, and yes, a few tips that might save you from last-minute panic Googling during your preparation.</p><h3>The Exam at a Glance :</h3><p><strong>Exam registration link :</strong> <a href="https://www.webassessor.com/databricks">Exam_Link</a></p><p>Proctored, multiple choice, and surprisingly scenario-heavy<br>45 questions | 90 minutes | $200 investment</p><p>Design Applications — 14%<br>Data Preparation — 14%<br>Application Development — 30% (you’ll feel this one)<br>Deployment &amp; Assembly — 22%<br>Governance — 8% (because compliance loves attention)<br>Evaluation &amp; Monitoring — 12%</p><h3><strong>Key Topics :</strong></h3><ol><li><strong>Prompt Engineering :</strong></li></ol><blockquote>Zero-shot vs. Few-shot prompting<br>Prompt chaining and system/user roles<br>Focus on how prompts drive LLM responses and how to enhance them in different contexts.</blockquote><p><strong>2. Retrieval-Augmented Generation (RAG) :</strong></p><blockquote>Understand end-to-end RAG pipelines in Databricks<br>Parsing &amp; Chunking — strategies like sentence or paragraph splitting, context window tuning<br>Vector Search with Databricks — indexing, querying, embeddings<br>Managing environment-specific authentication for dev &amp; prod</blockquote><p><strong>3. RAG on Structured Data :</strong></p><blockquote>Applying RAG techniques on structured datasets<br>Integration with Databricks Feature Store for augmenting structured information</blockquote><p><strong>4. Application Development :</strong></p><blockquote>Compound AI Systems — orchestrating multiple components<br>Multi-stage Reasoning — frameworks for chaining tasks<br>Agentic AI &amp; Design Patterns — using libraries like LangChain, LlamaIndex, OpenAI agents</blockquote><p><strong>5. LangChain Framework :</strong></p><blockquote>Structured workflows using LangChain’s core components: prompts, models, retrievers, tools<br>Typical pipeline: retrieval ➔ LLM ➔ post-processing</blockquote><p><strong>6. Model Deployment on Databricks :</strong></p><blockquote>Model Serving: API creation, dev/prod authentication, deployment patterns (batch &amp; streaming)<br>Performance optimization and scaling</blockquote><p><strong>7. Evaluation &amp; Monitoring of LLMs :</strong></p><blockquote>Lakehouse Monitoring with MLflow<br>Evaluation techniques:<br>Perplexity, Bleu, context precision, rouge, LLM-as-a-judge<br>Guardrails setup and license compliance</blockquote><h3><strong>Recommended Resources for Preparation :</strong></h3><p>During my preparation, I found the official Databricks resources to be the most reliable and structured. Here’s what worked for me:</p><p>✅ <strong>Databricks Academy Training :</strong></p><p>Generative AI Fundamentals Course — Get your basics right.<br>Generative AI Engineering with Databricks Course — Deep dive into topics like Application Development, RAG Implementation, and Data Preparation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A_94nckE3cDip24b0oY5zg.jpeg" /></figure><p><strong>✅ Practice Exams — To Test Your Nerves (and Knowledge) :</strong></p><p>I found Udemy practice exams and community-shared sample questions extremely helpful to get a feel of the actual exam difficulty.<br>These not only assess your readiness but also help you avoid surprises on exam day.</p><p><strong>Practice Exam Link : </strong><a href="https://www.udemy.com/course/databricks-certified-generative-ai-engineer-associate-practice-sets/"><em>https://www.udemy.com/course/databricks-certified-generative-ai-engineer-associate-practice-sets/</em></a></p><p><strong>My Suggestion : </strong><br>Complete the official Databricks training first — It builds a solid foundation.<br>Once done, dive into additional practice resources to sharpen your understanding and assess your confidence.</p><p>By following this guide and leveraging the recommended resources, you’ll be well-prepared to tackle the Databricks GenAI Associate Exam and achieve certification. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8793d76a2786" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimize Your Analytics: A Deep Dive into Databricks Clusters Part-1]]></title>
            <link>https://medium.com/@Manish_Pansari/optimize-your-analytics-a-deep-dive-into-databricks-clusters-part-1-69a82dbdc71a?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/69a82dbdc71a</guid>
            <category><![CDATA[databricks-delta-lake]]></category>
            <category><![CDATA[databricks-certification]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark-cluster]]></category>
            <category><![CDATA[databricks-basics]]></category>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Tue, 02 Jul 2024 19:44:29 GMT</pubDate>
            <atom:updated>2024-07-02T19:44:29.408Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><p>This article will explore the various compute options available in Databricks and guide you on selecting the right one based on your requirements. Additionally, it will cover best practices for managing compute resources and provide efficient configuration tips tailored to different workloads.</p><p><strong>Databricks Clusters :</strong></p><p>When working with Databricks, you’ll require a combination of resources and configurations to execute your data processing tasks. A Databricks Cluster simplifies this process by integrating these computational resources and configurations, enabling you to efficiently handle Data Science, Data Engineering, and Data Analytics workloads.</p><p><strong>Types of Databricks Clusters :</strong></p><p><strong>All-purpose Clusters:</strong> These clusters are designed for collaborative data analysis through interactive notebooks. They can be created using the CLI, UI, or REST API. All-purpose clusters can be manually terminated and restarted and are shareable among multiple users for interactive tasks.</p><blockquote>Best suited for ad hoc query execution and interactive notebook tasks during development and testing.<br>Should not be used for automated jobs.</blockquote><p><strong>Job Clusters: </strong>These clusters are optimized for running fast, automated tasks. They are created specifically when a job is run on a new Job Cluster and are automatically terminated once the job is complete. Job clusters cannot be restarted.</p><blockquote>Ideal for automated tasks, providing robust and efficient execution.<br>Should be used instead of all-purpose clusters for any automated job.</blockquote><p>When setting up a cluster, several key parameters need to be considered based on the workload’s size and type:</p><ol><li>Number of Nodes.</li><li>Size of Each Node.</li><li>Node Instance type.</li></ol><p>Once these primary parameters are selected, optimizing the following secondary parameters can enhance execution speed and improve the price-to-performance ratio with minimal additional effort:</p><ol><li>Photon Acceleration.</li><li>Disk Cache (also known as Delta cache).</li><li>Autoscaling.</li></ol><p>Lets discuss the above listed points one by one.</p><h4><strong>Number of Nodes :</strong></h4><p>Determining the optimal number of workers for a Spark job often involves some trial and error to understand the compute and memory requirements. Here are some guidelines to get you started:</p><ol><li>Avoid using a single worker for production jobs, as it poses a risk of being a single point of failure.</li><li>For small workloads — those without complex operations like joins or aggregations — begin with 2–4 workers.</li><li>For medium to large workloads that involve wide transformations, such as joins and aggregations, start with 8–10 workers and increase if needed.</li><li>Adjust shuffle partitions to fully utilize all available cluster cores when appropriate.</li><li>After running a few initial executions, you can estimate the approximate execution time.</li><li>If shuffle partitions are optimized and data skew and spill issues are managed, adding more workers won’t necessarily increase costs. Instead, it will proportionally decrease execution time, keeping costs relatively stable.</li></ol><h4><strong>Size of Nodes :</strong></h4><ol><li><strong>Driver Node:</strong> For handling large datasets or performing substantial broadcast joins, configure a driver node with 4–8 cores and 16–32GB of memory. This setup is generally sufficient and may require increasing spark.driver.maxResultSize.</li><li><strong>Shuffle-Heavy Workloads: </strong>Use fewer nodes with larger sizes to minimize network I/O during data transfers between nodes.</li><li><strong>Worker RAM: </strong>While workers with over 128GB of RAM can improve job performance, they may also lead to extended garbage collection (GC) pauses, which can degrade performance or even cause job failures. Therefore, it’s advisable not to use workers with more than 128GB of RAM.</li></ol><p><strong>General Advice: </strong>Scale up before scaling out. Using fewer machines with more resources reduces network overhead during shuffles.</p><h4><strong>Instance Type as Per Workload :</strong></h4><p>Please follow below image for instance type selection as per the databricks workload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*NP10G_tFpx1H8rvexKVpMA.jpeg" /><figcaption>Instance Types</figcaption></figure><h4><strong>Photon :</strong></h4><p>Photon is the advanced engine on the Databricks Lakehouse Platform designed to deliver exceptional query performance at a lower cost. It integrates seamlessly with Apache Spark APIs and works with existing Spark code, offering notable performance improvements over the standard Databricks Runtime.</p><p>Here are some of the key benefits of Photon:</p><ul><li><strong>Accelerated Queries:</strong> Photon significantly speeds up queries involving large datasets (&gt;100GB), especially those with aggregations and joins.</li><li><strong>Enhanced Performance with Delta Cache:</strong> It provides quicker access to data that is frequently retrieved from the Delta cache.</li><li><strong>Improved Scan/Read Performance:</strong> Photon optimizes performance for tables with numerous columns and many small files.</li><li><strong>Faster Delta Writes:</strong> It accelerates operations like UPDATE, DELETE, MERGE INTO, INSERT, and CREATE TABLE AS SELECT.</li><li><strong>Better Join Efficiency:</strong> Photon enhances the performance of join operations.</li></ul><p>For workloads with the following characteristics, it is advisable to enable Photon.</p><ul><li>ETL pipelines that include Delta MERGE operations.</li><li>Writing large volumes of data to cloud storage (Delta/Parquet).</li><li>Scanning large datasets, performing joins, aggregations, and decimal computations.</li><li>Using Auto Loader to efficiently process new data arriving in storage.</li><li>Running interactive or ad hoc SQL queries.</li></ul><h4><strong>Delta Cache :</strong></h4><p>Databricks employs disk caching to speed up data reads by storing copies of remote Parquet data files on the local storage of nodes. This process involves converting the data into a fast intermediate format. When a file is retrieved from a remote source, Databricks automatically caches it. Subsequent reads of this data are handled locally, greatly enhancing read performance. This caching mechanism is applicable to all Parquet data files, including Delta Lake tables.</p><h4><strong>Autoscaling :</strong></h4><p>Databricks offers robust autoscaling capabilities for clusters. Here’s how to effectively use autoscaling :</p><ol><li><strong>Ad Hoc Queries and Interactive Work:</strong></li></ol><blockquote>Always enable autoscaling for ad hoc queries, interactive notebook executions, and development/testing pipelines using interactive clusters.</blockquote><blockquote>Set the minimum number of workers to 1 to ensure that resources are adjusted as needed.</blockquote><p><strong>2. Production Job Clusters:</strong></p><blockquote>For job clusters in production, avoid setting the minimum instances to 1 if the job requires more than one worker. Instead, set a minimum value that meets the job’s basic compute requirements. This helps reduce scale-up time.</blockquote><p><strong>3. Fine-Tuned Job Clusters:</strong></p><blockquote>Autoscaling may not be necessary if Spark shuffle partitions are optimized to fully utilize all worker cores and the job has its own dedicated cluster. This approach provides precise cost control.</blockquote><blockquote>However, you can still use autoscaling as a precautionary measure. Set the minimum number of workers based on the expected daily data load and include a few extra workers as a buffer to handle unexpected data surges.</blockquote><p><strong>4. Shared Job Clusters in Workflows:</strong></p><blockquote>When multiple tasks or jobs within the same pipeline are running on a shared job cluster, enable autoscaling for the cluster. This allows the cluster to scale up and provide adequate resources for all parallel jobs.</blockquote><p>Thank you for reading!</p><p>We’ve covered some essential strategies for optimizing Databricks clusters, including autoscaling, disk caching, and leveraging Photon. Remember, the right configuration can make a significant difference in performance and cost-efficiency. Stay tuned for our next articles, where we’ll dive into instance pools and auto-termination, two more powerful tools for managing your Databricks environment.</p><p>If you have any questions or thoughts on these topics, feel free to share them in the comments below. Don’t forget to follow for more insights and updates. Happy optimizing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69a82dbdc71a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide to Mastering the Databricks Data Engineer Professional Exam]]></title>
            <link>https://medium.com/@Manish_Pansari/a-guide-to-mastering-the-databricks-data-engineer-professional-exam-129166496cc0?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/129166496cc0</guid>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 07:54:56 GMT</pubDate>
            <atom:updated>2024-06-03T07:54:56.724Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*COGUEL4LPfSOi4RacT7tzQ.jpeg" /></figure><p>The Databricks Data Engineer Professional Exam is the final step in the learning path for the Databricks Data Engineer certification. This exam is a significant milestone for any data professional aiming to demonstrate their proficiency in using the Lakehouse Platform for advanced data engineering tasks. Here’s a comprehensive guide to help you prepare and succeed in this exam.</p><p><strong>Note :</strong> Databricks Data Engineer Associate Certification is not an mandatory/pre-requisites, but you need all the skills of Associate Certification.</p><p><strong>Exam Overview :</strong><br><strong><em>Duration </em>:</strong> Testers have 120 minutes to complete the certification exam.<br><strong><em>Cost </em>:</strong> Each attempt costs $200, with additional taxes depending on your location. You can retake the exam as many times as needed, but each attempt requires the $200 fee.<br><strong><em>Registration </em>:</strong> To register for the certification exam, you need to create an account on <a href="https://webassessor.com/databricks">here</a> or log in if you already have one.</p><p><strong>Exam Topics :</strong></p><p>The exam covers several key areas, ensuring you have a broad understanding of the Databricks Lakehouse Platform and its applications. Here are the main topics:</p><p><strong>Databricks Tooling :</strong></p><blockquote>Delta Lake transaction log and cloud object storage.<br>Delta Lake’s Optimistic Concurrency Control isolation.<br>Delta Clones. <br>Delta Lake indexing optimizations including partitioning, zorder, bloom filters, and file sizes. <br>Delta tables optimizations for Databricks SQL.</blockquote><p><strong>Data Processing :</strong></p><blockquote>Multiple Strategies for Updating Records in a Spark Table (Type 1).<br>Implementing Common Design Patterns with Structured Streaming and Delta Lake.<br>Exploring and Tuning State Information with Stream-Static Joins in Delta Lake.<br>Implementing Stream-Static Joins and Deduplication Logic Using Spark Structured Streaming.<br>Enabling Change Data Feed (CDF) on Delta Lake Tables and Redesigning Data Processing Steps for CDC Output.<br>Leveraging CDF for Easy Propagation of Deletes.</blockquote><p><strong>Data Modeling :</strong></p><blockquote>Data Transformations During Promotion from Bronze to Silver.<br>Designing a Multiplex Bronze Table to Avoid Common Pitfalls in Streaming Workloads.<br>Implementing Best Practices for Streaming Data from Multiplex Bronze Tables.<br>Applying Incremental Processing, Quality Enforcement, and Deduplication from Bronze to Silver.<br>Making Informed Decisions on Enforcing Data Quality Based on Delta Lake’s Strengths and Limitations.<br>Adding Constraints to Delta Lake Tables to Prevent Bad Data from Being Written.<br>Diagramming Architectures and Operations for Implementing Slowly Changing Dimension Tables Using Delta Lake in Streaming and Batch Workloads.<br>Implementing SCD Type 0, 1, and 2 Tables.</blockquote><p><strong>Security &amp; Governance :</strong></p><blockquote>Dynamic views to perform data masking <br>Dynamic views to control access to rows and columns</blockquote><p><strong>Monitoring &amp; Logging :</strong></p><blockquote>Utilizing Spark UI for Performance Analysis, Application Debugging, and Tuning.<br>Inspecting Event Timelines and Metrics for Stages and Jobs on a Cluster.<br>Drawing Conclusions from Spark UI, Ganglia UI, and Cluster UI to Assess Performance Issues and Debug Failures.<br>Designing Systems to Control Costs and Meet Latency SLAs for Production Streaming Jobs.<br>Deploying and Monitoring Streaming and Batch Jobs.</blockquote><p><strong>Testing &amp; Deployment :</strong></p><blockquote>Implement a notebook dependency pattern to use Python file dependencies.<br>Implement Python code maintained as Wheels to direct imports using relative paths.<br>Repair and rerun failed jobs.<br>Create Jobs based on common use cases and patterns.<br>Create a multi-task job with multiple dependencies.<br>Configure the Databricks CLI and execute basic commands to interact with the workspace and clusters. <br>Execute commands from the CLI to deploy and monitor Databricks jobs. <br>Use REST API to clone a job, trigger a run, and export the run output.</blockquote><p><strong>Question Distribution :</strong><br>There are 64 multiple-choice questions on the certification exam. The questions will be distributed by high-level topic in the following way:</p><blockquote>Databricks Tooling — 20%<br>Data Processing — 30%<br>Data Modeling — 20%<br>Security and Governance — 10%<br>Monitoring and Logging — 10%<br>Testing and Deployment — 10%</blockquote><p><strong>Exam Pattern :</strong></p><p>The exam comprises two types of multiple-choice questions, conceptual and code. The conceptual questions challenge your understanding of Spark operations and the overall features of the Databricks Lakehouse platform. The code questions are all some form of “fill-in-the-blank,” “find the error,” or arranging the code statements in the appropriate order based on some fact pattern from the question. You will not be asked to write code during the exam. There are also no outside resources allowed or provided during the exam.</p><p><strong>Preparation Plan :</strong><br>To thoroughly prepare for the exam, allocate about 4–5 weeks for dedicated study and practice.</p><p><strong>Step 1: </strong>Take the <a href="https://partner-academy.databricks.com/learn/course/811/session/3021/advanced-data-engineering-with-databricks-class-13527-emea">Databricks Academy Data Engineering Course.</a><br>The Databricks Academy Data Engineering with Databricks course is an excellent starting point. This free course covers advance topics related to data engineering, including data collection, processing, and analysis.</p><p><strong>Step 2:</strong> Supplement with Additional Udemy Courses.<br>Consider taking <a href="https://www.udemy.com/course/databricks-certified-data-engineer-professional/?couponCode=IND21PM">this</a> Udemy course following courses for a more in-depth understanding.</p><p><strong>Step 3: </strong>Practice with Mock Test.</p><p><strong>Udemy course for practice questions:</strong> <a href="https://www.udemy.com/course/practice-exams-databricks-data-engineer-professional-k/?couponCode=IND21PM">https://www.udemy.com/course/practice-exams-databricks-data-engineer-professional-k/?couponCode=IND21PM</a></p><p><strong>Step 4: </strong>Go through some Youtube videos related to Databricks Data Engineer Professional Exam.</p><p>There are a lot of topics covered by the exam, and to make matters complex, the exam questions are LONG! There is a lot of context for each multiple-choice question. Be sure to take your time and understand what the question is asking of you. They are trying to trick you with answers that might seem right but are not the correct choice in the context of the question. Try to eliminate the wrong answers first.</p><p><strong>Exam Strategy :</strong><br><strong>Familiarize Yourself with the Platform:</strong></p><blockquote><em>Use the Databricks Academy course or the Databricks Community Edition to get hands-on experience.</em></blockquote><p><strong>Mark and Review:</strong></p><blockquote><em>Quickly cover all questions in one round, marking those you’re unsure about for review. In the second round, revisit these marked questions and ensure no question is left unanswered before submitting.</em></blockquote><p>By following this guide and leveraging the recommended resources, you’ll be well-prepared to tackle the Databricks Data Engineer Professional Exam and achieve certification. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=129166496cc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide to Mastering the Databricks Data Engineer Associate Exam]]></title>
            <link>https://medium.com/@Manish_Pansari/a-guide-to-mastering-the-databricks-data-engineer-associate-exam-ba01f96869b4?source=rss-34a13a391539------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba01f96869b4</guid>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks-data-engineer]]></category>
            <category><![CDATA[databricks-certification]]></category>
            <dc:creator><![CDATA[Manish Pansari]]></dc:creator>
            <pubDate>Sat, 25 May 2024 20:59:38 GMT</pubDate>
            <atom:updated>2024-05-25T21:07:57.747Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey everyone! If you don’t know me yet, feel free to check out my LinkedIn profile: <a href="https://www.linkedin.com/in/manish-pansari/">Manish Pansari</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*JEHTRhKytAHEWXIB__049A.jpeg" /></figure><p>The Databricks Data Engineer Associate Exam is a significant milestone for any data professional looking to demonstrate their proficiency in using the Lakehouse Platform for basic data engineering tasks. This certification verifies a thorough understanding of the platform, its tools, and the benefits it offers. Here’s a comprehensive guide to help you prepare and succeed in this exam.</p><p><strong>Exam Overview : </strong><br><strong><em>Duration:</em></strong> Testers have 90 minutes to complete the certification exam.<br>Cost: Each attempt costs $200, with additional taxes depending on your location. You can retake the exam as many times as needed, but each attempt requires the $200 fee.<br>Registration: To register for the certification exam, you need to create an account on <a href="https://webassessor.com/databricks">here</a> or log in if you already have one.</p><p><strong>Exam registration link :</strong> <a href="https://www.webassessor.com/databricks">Exam_Link</a></p><p><strong>Exam Topics :</strong><br>The exam covers several key areas, ensuring you have a broad understanding of the Databricks Lakehouse Platform and its applications. Here are the main topics:</p><p><strong>Databricks Lakehouse Platform:</strong></p><blockquote>Data Lakehouse: Understand its architecture, descriptions, and benefits.<br>Data Science and Engineering Workspace: Learn about clusters, notebooks, and data storage.<br>Delta Lake: Grasp general concepts, table management, manipulation, and optimizations.</blockquote><p><strong>ETL Pipelines using Apache Spark SQL and Python:</strong></p><blockquote>Relational Entities: Work with databases, tables, and views.<br>ELT Processes: Create tables, write data, clean data, combine and reshape tables, and use SQL UDFs.<br>Python Integration: Facilitate Spark SQL with string manipulation, control flow, and passing data between PySpark and Spark SQL.</blockquote><p><strong>Production Pipelines for Data Engineering:</strong></p><blockquote>Jobs: Learn about scheduling, task orchestration, and UI.<br>Dashboards: Understand endpoints, scheduling, alerting, and refreshing.</blockquote><p><strong>Security Practices:</strong></p><blockquote>Unity Catalog: Discover its benefits and features.<br>Entity Permissions: Manage team-based and user-based permissions.<br>Incremental Data Processing:</blockquote><p><strong>Structured Streaming:</strong></p><blockquote>Learn about general concepts, triggers, and watermarks.<br>Auto Loader: Understand streaming reads.<br>Multi-hop Architecture: Grasp the bronze-silver-gold architecture and streaming applications.<br>Delta Live Tables: Learn about its benefits and features.</blockquote><p><strong>Question Distribution :</strong><br>The exam consists of 45 multiple-choice questions, distributed as follows:</p><blockquote><strong>Databricks Lakehouse Platform</strong> : 24% (11 questions)<br><strong>ELT with Spark SQL and Python :</strong> 29% (13 questions)<br><strong>Incremental Data Processing : </strong>22% (10 questions)<br><strong>Production Pipelines : </strong>16% (7 questions)<br><strong>Data Governance :</strong> 9% (4 questions)</blockquote><p><strong>Preparation Plan :</strong><br>To thoroughly prepare for the exam, allocate about 2–3 weeks for dedicated study and practice.</p><p><strong>Step 1: </strong>Take the <a href="https://partner-academy.databricks.com/learn/course/internal/view/elearning/1266/data-engineering-with-databricks-v3">Databricks Academy Data Engineering Course</a>.<br>The Databricks Academy Data Engineering with Databricks course is an excellent starting point. This free course covers the basics of data engineering, including data collection, processing, and analysis.</p><p><strong>Step 2:</strong> Supplement with Additional Courses.<br>Consider taking <a href="https://www.udemy.com/course/databricks-certified-data-engineer-associate/?utm_source=adwords&amp;utm_medium=udemyads&amp;utm_campaign=LongTail_la.EN_cc.INDIA&amp;utm_content=deal4584&amp;utm_term=_._ag_77882236223_._ad_533093955804_._kw__._de_c_._dm__._pl__._ti_dsa-1007766171032_._li_9062136_._pd__._&amp;matchtype=&amp;gclid=CjwKCAjwm4ukBhAuEiwA0zQxk7AYDleuZFRhVDZ8GqDX9sAQYuHmePXlmNSIDBUfUKtvsuBOtjU0uRoCBf0QAvD_BwE&amp;couponCode=IND21PM">this</a> following courses for a more in-depth understanding.</p><p><strong>Step 3: </strong>Practice with Databricks Provided Resources.<br>Test your knowledge with the Databricks Certified Data Engineer Associate Practice Exam. These practice exam gives you a good sense of the types of questions you will encounter.</p><p><strong>Sample questions:</strong> <a href="https://github.com/Manish-Pansari/Databricks_Data_Engineer_Associate_Practice_Questions">https://github.com/Manish-Pansari/Databricks_Data_Engineer_Associate_Practice_Question</a><br>(Free)<br><strong>Udemy course for practice questions:</strong> <a href="https://www.udemy.com/course/databricks-certified-data-engineer-associate-practice-tests/?couponCode=IND21PM">https://www.udemy.com/course/databricks-certified-data-engineer-associate-practice-tests/?couponCode=IND21PM </a>(paid)</p><p><strong>Exam Strategy :</strong><br><strong>Familiarize Yourself with the Platform:</strong></p><blockquote>Use the Databricks Academy course or the Databricks Community Edition to get hands-on experience.</blockquote><p><strong>Mark and Review:</strong></p><blockquote>Quickly cover all questions in one round, marking those you’re unsure about for review. In the second round, revisit these marked questions and ensure no question is left unanswered before submitting.</blockquote><p>By following this guide and leveraging the recommended resources, you’ll be well-prepared to tackle the Databricks Data Engineer Associate Exam and achieve certification. Good luck!</p><p><em>This article should provide a comprehensive and engaging read for anyone preparing for the Databricks Data Engineer Associate Exam. If you have reached this far, thank you for reading this blog, I hope you found it insightful</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba01f96869b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>