<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Badrish Davay on Medium]]></title>
        <description><![CDATA[Stories by Badrish Davay on Medium]]></description>
        <link>https://medium.com/@badrish.davay?source=rss-6f53b904fff2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*y-kStl9ei9r0MaC8</url>
            <title>Stories by Badrish Davay on Medium</title>
            <link>https://medium.com/@badrish.davay?source=rss-6f53b904fff2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:57:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@badrish.davay/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM, RAG, AI Agent, Agentic AI: What Actually Changes at Each Step]]></title>
            <link>https://medium.com/@badrish.davay/llm-rag-ai-agent-agentic-ai-what-actually-changes-at-each-step-191c7cb6a26c?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/191c7cb6a26c</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[life-sciences]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 19:12:58 GMT</pubDate>
            <atom:updated>2026-09-21T19:12:58.490Z</atom:updated>
            <content:encoded><![CDATA[<p>Four terms get used as if they were four brands of the same thing. They are not. Each one adds a specific capability, and each added capability brings a specific new failure mode. If you work in a regulated industry, the failure modes matter more than the capabilities.</p><p>Here is the progression in one line: an LLM <strong>writes</strong>, RAG <strong>grounds</strong>, an agent <strong>acts</strong>, and agentic AI <strong>coordinates</strong>.</p><p>Let me take each one, show what is actually happening under the hood, and give a life sciences example that a pharma or biotech team would recognize.</p><h3>1. The LLM: a text predictor with no outside world</h3><p>An LLM takes your prompt, converts it to tokens, and predicts the next token over and over until it decides to stop. Everything it “knows” is baked into its weights during training. It has no access to your systems, no memory of yesterday, and no ability to do anything other than produce text.</p><p>append token, predict thenext oneUser prompt+ provided contextTokenizerLLMlearned parameters, frozenGenerated responseWhat it cannot do:look anything upremember past sessionschange state in a system</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r1FhC4hJ48-CS_R5CCKUDQ.png" /></figure><p><strong>Life sciences example: first-draft medical writing.</strong></p><p>A medical writer pastes the statistical output and key findings from a Phase II study into the model and asks for a plain-language summary suitable for a trial participant. The model produces a readable draft in seconds, at roughly an eighth-grade reading level, in the right tone.</p><p>What it does well: structure, tone, readability, consistency of voice across twenty summaries.</p><p>Where it breaks: ask it for the hepatic impairment dosing guidance for your compound and it will produce a confident, well-formatted, entirely invented answer. It has no idea what is in your label. It is predicting plausible text, and plausible is not the same as correct.</p><p>That gap is the whole reason RAG exists.</p><h3>2. RAG: grounding the answer in your documents</h3><p>Retrieval Augmented Generation puts a retriever in front of the model. The question goes to a search layer first, the search layer pulls the most relevant passages from an indexed knowledge base, and those passages get stitched into the prompt alongside the question. The model then answers using the text in front of it rather than the text in its weights.</p><p>top-k relevant passagesapprovedinsufficient evidenceClinical questionEmbed queryRetrievervector + keyword searchIndexed knowledgeapproved label, CSRs,standard response docs,SOPsPrompt assemblyquestion + retrievedcontextLLMGrounded answerwith source citationsHuman reviewResponse to requester</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nN_HnhhtjViWLCMGPNYusQ.png" /></figure><p><strong>Life sciences example: the medical information response desk.</strong></p><p>A healthcare provider calls in with a question about dose adjustment in moderate hepatic impairment. Today an associate searches the standard response document library, the approved label, and sometimes the clinical study report, then assembles a reply. Turnaround is measured in hours or days.</p><p>With RAG, the retriever pulls the four or five passages that actually address hepatic impairment, and the model drafts a response that cites the label section and the SRD ID it came from. The associate reviews, edits, and sends.</p><p>Two things to be honest about. First, grounded does not mean correct. If your index contains a superseded SRD, you get a confidently cited wrong answer, which is worse than an obviously wrong one. Content governance becomes the real project. Second, retrieval quality is the ceiling on answer quality. Most disappointing RAG pilots in pharma are not model problems, they are chunking, metadata, and document lifecycle problems.</p><p>RAG still only produces text. It cannot open a case, code a term, or update a record. For that you need an agent.</p><h3>3. AI Agent: a loop that can act</h3><p>An agent is a goal, a reasoning engine, a set of instructions, a running task state, and access to tools. The important word is loop. The agent decides what to do, does it, observes what came back, updates its understanding, and decides again. It keeps going until the goal is met or it hits a stopping condition.</p><p>goal metchoose an actionupdate task state, repeatAgent coreLLM reasoningand planningInstructions,policy, guardrailsTask stateand working memoryGoal: triage this inboundadverse event reportDecide next stepOutcome:coded, assessed, draftnarrativeAct: call a toolTools, data, environmentMedDRA coding service,safety database,duplicate check, literaturesearch, emailObserve resultHuman sign-offGxP quality gateValidated safety database</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*jz0GT-qBcI34YEySrIQ7sg.png" /></figure><p><strong>Life sciences example: adverse event case intake and triage.</strong></p><p>A case arrives by email from a call center. The agent reads it and works through a sequence it decides on rather than one you hard-coded:</p><ol><li>Extract the four minimum criteria for a valid case: identifiable patient, identifiable reporter, suspect product, event.</li><li>Call the duplicate check tool against the safety database. If a probable duplicate comes back, it changes course and drafts a merge recommendation instead of a new case.</li><li>Call the MedDRA coding service to code the reported terms, and check the returned preferred terms against the designated medical event list.</li><li>Apply the seriousness criteria and the expectedness check against the reference safety information.</li><li>Draft the narrative in house style and attach the evidence trail.</li><li>Stop, and hand the whole package to a safety physician for sign-off.</li></ol><p>The loop is what earns the label “agent.” At step two it chose a different path based on what a tool told it. A fixed workflow could not do that without someone anticipating every branch.</p><p>What this changes for you operationally: an agent that can call tools can also call them wrong. Scope tool permissions tightly, make every write operation reversible or gated, log every step, and treat the agent as a proposer rather than an approver for anything that touches a validated system. The value is in compressing the 80 percent of routine cases so your physicians spend their time on the 20 percent that are genuinely difficult.</p><h3>4. Agentic AI: coordination across many agents and workflows</h3><p>One agent handles one goal. Real work is a set of interlocking goals with dependencies, different specialist knowledge, and a shared definition of done. Agentic AI adds an orchestration layer above the agents: it decomposes the objective, routes work, maintains shared state that everyone reads and writes, evaluates progress against the objective, and replans when something comes back short.</p><p>Note the part people miss. An agentic system is not all agents. The best designs use deterministic workflow steps wherever the path is known and reserve agents for the parts that genuinely need judgment. Determinism is cheaper, faster, testable, and far easier to validate.</p><p>Agentic AI systemgaps found, replancriteria metcoordinate workread / update stateresults and statustool calls and resultsAgents and workflowsLiteratureagentClinical dataagentCMCagentDeterministicworkflow stepsformat, validate, diffObjective: assemble asubmission-ready dossiersectionOrchestratordecompose, route,sequenceShared task stateplan, artifacts, decisions,provenanceEvaluate progressagainst acceptance criteriaTools, data, environmentEDC, CTMS, eTMF, LIMS,document system,publishingHuman review gatesQC, medical, regulatoryOutcome: reviewed draft +full audit trail</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*041nQaJOzfucE-htMM7_Ug.png" /></figure><p><strong>Life sciences example: regulatory submission readiness.</strong></p><p>Preparing a submission section pulls from sources that live in different systems and different functions. An agentic setup might look like this:</p><p>The orchestrator breaks the objective into a plan and writes it to shared state. The literature agent runs the search strategy, screens hits against inclusion criteria, and posts a candidate evidence set. The clinical data agent pulls study-level results from the clinical data repository and reconciles them against what the literature agent found. The CMC agent assembles the manufacturing and stability narrative from the relevant batch records and specifications. Deterministic workflow steps handle formatting to the required structure, running the technical validation checks, and producing a diff against the previous version.</p><p>The evaluator checks the assembled output against explicit acceptance criteria: every claim traceable to a source, no unresolved data discrepancies, all required subsections present. Where it finds a gap it sends that specific piece back to the responsible agent rather than restarting everything. When criteria are met, the package goes to human QC, medical, and regulatory review gates. Nothing reaches a validated system without a named human signing for it.</p><p>The audit trail is not an afterthought here. In a GxP context, shared task state doubles as your evidence that the process was followed, which source produced which claim, and who approved what. Design it in from the first sprint or you will retrofit it painfully later.</p><h3>Which one do you actually need</h3><p>Most teams reach for the most sophisticated pattern when a simpler one would do. More autonomy means more surface area for things to go wrong, more cost per task, and a harder validation story. Start at the bottom of this tree and only move up when the answer forces you to.</p><p>NoYesNo, answer onlyYesYesNoNoYesWhat are you trying to do?Does the answer dependonyour proprietary orchanging data?Use an LLMdrafting, summarizing,reformatting, translatingDoes anything need tochange in a system?Use RAGgrounded Q&amp;A withcitationsIs the sequence of stepsknown in advance?Use a workflowwith LLM steps inside itcheaper, testable,validatableDoes it need multiplespecialisms workingagainst a shared goal?Use a single agentwith tightly scoped toolsUse an agentic systemorchestrator, shared state,evaluator, human gates</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*veZuboqnYuKJYmlkT1YqRQ.png" /></figure><h3>The comparison in one table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*S6W_FLmTzAakV0XHS05O5Q.png" /></figure><h3>Five things I would tell a life sciences team starting this</h3><p><strong>Grounded is not validated.</strong> A citation proves the model used a source. It does not prove the source was current, applicable, or correctly interpreted. Your content governance program is the real dependency.</p><p><strong>Put the human gate where the risk is, not everywhere.</strong> Blanket human review of every step kills the economics and trains reviewers to rubber-stamp. Identify the specific points where a wrong decision has regulatory or patient consequence and gate hard there.</p><p><strong>Deterministic beats agentic wherever the path is known.</strong> An agent that reinvents a fixed five-step process on every run is slower, more expensive, and less auditable than a workflow that just does the five steps. Use judgment where judgment is needed.</p><p><strong>Instrument from day one.</strong> Every tool call, every retrieved passage, every decision, every human override. You need this for debugging in month two and for your quality system in month six. Adding it later is a rewrite.</p><p><strong>Measure the handoff, not the demo.</strong> The metric that matters is not whether the agent produced a good draft. It is how much time the reviewer spent fixing it, and whether that number is falling week over week.</p><p>The progression from LLM to agentic AI is really a progression in how much of the work you are willing to delegate. Each step gives back more leverage and takes away more visibility. In life sciences the job is to keep the leverage while engineering the visibility back in, and that engineering is most of the work.</p><p><em>The four-pattern framing in this post was inspired by a ByteByteGo diagram. The diagrams and examples here are my own.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=191c7cb6a26c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[dbt and Prefect on Snowflake: where the line actually sits]]></title>
            <link>https://medium.com/@badrish.davay/dbt-and-prefect-on-snowflake-where-the-line-actually-sits-4fe438ebf084?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fe438ebf084</guid>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[dagster]]></category>
            <category><![CDATA[fivetran]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 19:39:13 GMT</pubDate>
            <atom:updated>2026-09-16T20:12:32.740Z</atom:updated>
            <content:encoded><![CDATA[<p><em>One of them compiles SQL and hands it to your warehouse. The other runs Python and hands work to everything else. Some ideas and a pile of code for the teams still treating this as a bake-off.</em></p><p>Every few months a platform team asks me to settle it: dbt or Prefect. The question usually arrives with an assumption baked in, which is that one of them will turn out to be the better version of the other. That is not what is going on. The two tools overlap on maybe a sixth of their surface area, and that sixth is where all the confusion lives.</p><p>Here is the sentence I keep repeating in architecture reviews. <strong>dbt decides what your tables should look like. Prefect decides when anything happens and what to do when it breaks.</strong> On Snowflake you will very likely want both, and the interesting work is not picking a winner, it is drawing the line between them and then defending that line when people start putting orchestration logic into macros.</p><p>This piece is the long version of that conversation. Diagrams first, because the boundary is a spatial idea and prose is bad at spatial ideas.</p><p>Colour key used in every diagram: amber means it runs on your infrastructure, teal means it runs on Snowflake compute, blue means objects dbt owns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FF4VbYNJjOyg_aAn1WhgmQ.png" /></figure><p><em>Figure 1. The only picture that matters. dbt-core is a process on your side of the line that produces SQL text; the work happens on the other side. Prefect never crosses the line either, it just tells things on both sides when to start.</em></p><h3>The distinction that explains all the others</h3><p>Ask where the code runs. Everything else follows from the answer.</p><p>dbt is a compiler with a database connection. You write SQL with Jinja in it, dbt resolves the references, works out a dependency graph, and sends plain SQL statements to Snowflake over a normal connection. Not one row of your data passes through the dbt process. If you kill dbt halfway through a model, Snowflake keeps running the statement it already received. That single fact explains why dbt scales beautifully on Snowflake and why it is useless for anything that is not SQL.</p><p>Prefect is the opposite shape. You write Python functions, decorate them, and Prefect turns them into runs it can schedule, retry, observe and react to. The Python executes on infrastructure you own, a container, a pod, an ECS task. Prefect Cloud or your self-hosted server holds the state, the schedules and the event rules, but it does not hold your data. So Prefect can call anything Python can call, which is everything, and it has no opinion whatsoever about whether your SQL is any good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6UZKvU83Xnbf6sJPvT0GQ.png" /></figure><p><em>Figure 2. Two execution models. The dbt chain ends in your warehouse; the Prefect chain ends in your cluster. Neither tool is a place where data lives, which is why neither one is a bottleneck and why both are cheap to run compared with the compute they trigger.</em></p><p>People get confused because dbt <em>feels</em> like an orchestrator. It has a DAG. It runs things in order. It skips downstream models when an upstream one fails. All true, and all scoped to a single invocation of a single project against a single warehouse. The moment your dependency is “the Salesforce extract finished” rather than “stg_orders finished”, dbt has no idea what you are talking about.</p><h3>Who owns which stage</h3><p>Map both tools onto the lifecycle and the overlap stops being mysterious. There is exactly one contested zone, transform, and even there the contest is about who presses the button, not who writes the SQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aglmkUOMWT14NLXvO3szlg.png" /></figure><p><em>Figure 3. Coverage bands across the lifecycle. Note that dbt’s band starts at Store, not at Extract. Everything to the left of the RAW schema is somebody else’s job, and that somebody is usually Fivetran, Snowpipe or a Prefect task.</em></p><h3>dbt on Snowflake, concretely</h3><p>An example is worth three paragraphs of positioning. Here is a small slice of a revenue pipeline: two sources, three staging models, one fact table, and the tests that stop bad data reaching a dashboard.</p><p>Sources first. This is the file people skip, and then they wonder why nobody notices when an upstream extract silently stops.</p><p>models/staging/_sources.yml</p><p>version<strong>:</strong> 2</p><p>sources<strong>:</strong><br> — name<strong>:</strong> salesforce<br> database<strong>:</strong> raw<br> schema<strong>:</strong> salesforce<br> loaded_at_field<strong>:</strong> _fivetran_synced<br> freshness<strong>:</strong><br> warn_after<strong>:</strong> <strong>{</strong>count<strong>:</strong> 6<strong>,</strong> period<strong>:</strong> hour<strong>}</strong><br> error_after<strong>:</strong> <strong>{</strong>count<strong>:</strong> 24<strong>,</strong> period<strong>:</strong> hour<strong>}</strong><br> tables<strong>:</strong><br> — name<strong>:</strong> orders<br> description<strong>:</strong> One row per order header, soft deletes included.<br> columns<strong>:</strong><br> — name<strong>:</strong> id<br> tests<strong>:</strong> <strong>[</strong>unique<strong>,</strong> not_null<strong>]</strong><br> — name<strong>:</strong> accounts</p><p>— name<strong>:</strong> stripe<br> database<strong>:</strong> raw<br> schema<strong>:</strong> stripe<br> loaded_at_field<strong>:</strong> _loaded_at<br> tables<strong>:</strong><br> — name<strong>:</strong> payments</p><p>Freshness is the cheapest early warning you will ever configure. dbt source freshness runs a max() query per table and fails the build before you spend credits transforming yesterday’s data.</p><p>models/staging/stg_orders.sql</p><p>{{ config(<strong>materialized</strong>=’view’) }}</p><p><strong>with</strong> <strong>source</strong> <strong>as</strong> (<br> <strong>select</strong> * <strong>from</strong> {{ <strong>source</strong>(‘salesforce’, ‘orders’) }}<br>),</p><p>renamed <strong>as</strong> (<br> <strong>select</strong><br> <strong>id</strong> <strong>as</strong> order_id,<br> account_id <strong>as</strong> customer_id,<br> cast(created_date <strong>as</strong> timestamp_ntz) <strong>as</strong> ordered_at,<br> upper(trim(status)) <strong>as</strong> order_status,<br> amount / 100.0 <strong>as</strong> order_amount_usd,<br> _fivetran_deleted <strong>as</strong> is_deleted<br> <strong>from</strong> <strong>source</strong><br>)</p><p><strong>select</strong> * <strong>from</strong> renamed<br><strong>where</strong> <strong>not</strong> is_deleted</p><p>Staging models stay boring on purpose: rename, cast, deduplicate, nothing else. Materialized as views they cost storage of approximately zero and always reflect RAW.</p><p>models/marts/fct_orders.sql</p><p>{{<br> config(<br> <strong>materialized</strong>=’incremental’,<br> incremental_strategy=’merge’,<br> unique_key=’order_id’,<br> on_schema_change=’append_new_columns’,<br> cluster_by=[‘ordered_at’],<br> snowflake_warehouse=’TRANSFORM_L’<br> )<br>}}</p><p><strong>with</strong> orders <strong>as</strong> (<br> <strong>select</strong> * <strong>from</strong> {{ ref(‘stg_orders’) }}</p><p>{% <strong>if</strong> is_incremental() %}<br> <strong>where</strong> ordered_at &gt;= (<strong>select</strong> dateadd(day, -3, max(ordered_at)) <strong>from</strong> {{ this }})<br> {% endif %}<br>),</p><p>payments <strong>as</strong> (<br> <strong>select</strong><br> order_id,<br> sum(amount_usd) <strong>as</strong> amount_paid_usd,<br> count(*) <strong>as</strong> payment_count<br> <strong>from</strong> {{ ref(‘stg_payments’) }}<br> <strong>group</strong> <strong>by</strong> 1<br>)</p><p><strong>select</strong><br> o.order_id,<br> o.customer_id,<br> o.ordered_at,<br> o.order_status,<br> o.order_amount_usd,<br> coalesce(p.amount_paid_usd, 0) <strong>as</strong> amount_paid_usd,<br> coalesce(p.payment_count, 0) <strong>as</strong> payment_count,<br> o.order_amount_usd — coalesce(p.amount_paid_usd, 0) <strong>as</strong> balance_usd<br><strong>from</strong> orders o<br><strong>left</strong> <strong>join</strong> payments p <strong>on</strong> o.order_id = p.order_id</p><p>Three Snowflake specific things in one config block: cluster_by for pruning on the date predicate, snowflake_warehouse to send this one heavy model to a larger warehouse while everything else stays on XS, and a merge strategy with a three day lookback window for late arriving payments. This is the kind of control that makes people stop writing stored procedures.</p><p>models/marts/_marts.yml</p><p>version<strong>:</strong> 2</p><p>models<strong>:</strong><br> — name<strong>:</strong> fct_orders<br> description<strong>:</strong> Order grain fact with payment rollups.<br> config<strong>:</strong><br> contract<strong>:</strong> <strong>{</strong>enforced<strong>:</strong> true<strong>}</strong><br> columns<strong>:</strong><br> — name<strong>:</strong> order_id<br> data_type<strong>:</strong> varchar<br> constraints<strong>:</strong><br> — type<strong>:</strong> not_null<br> tests<strong>:</strong><br> — unique<br> — name<strong>:</strong> customer_id<br> data_type<strong>:</strong> varchar<br> tests<strong>:</strong><br> — relationships<strong>:</strong><br> to<strong>:</strong> ref(‘dim_customers’)<br> field<strong>:</strong> customer_id<br> — name<strong>:</strong> order_status<br> data_type<strong>:</strong> varchar<br> tests<strong>:</strong><br> — accepted_values<strong>:</strong><br> values<strong>:</strong> <strong>[</strong>‘PLACED’<strong>,</strong> ‘SHIPPED’<strong>,</strong> ‘CANCELLED’<strong>,</strong> ‘RETURNED’<strong>]</strong><br> — name<strong>:</strong> balance_usd<br> data_type<strong>:</strong> number<br> tests<strong>:</strong><br> — dbt_utils.accepted_range<strong>:</strong><br> min_value<strong>:</strong> -1000<br> max_value<strong>:</strong> 100000</p><p>With contract: {enforced: true}, dbt asks Snowflake to enforce the column types at build time rather than discovering the drift three dashboards later.</p><p>Run dbt build and dbt walks that graph, materializing and testing in dependency order. The graph is the product. It is what gives you lineage for free, selective runs, and the ability to hand a junior analyst a model without them needing to know what a Snowflake task is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjL8VjqFYuyDtXzHOQ3VAA.png" /></figure><p><em>Figure 4. The dbt graph for the code above. Sources are contracts with the outside world, staging is a thin rename layer, intermediate models can be ephemeral and never become objects at all, and exposures point at what breaks when a model breaks.</em></p><h3>What dbt will not do for you</h3><p>dbt’s failure model is a batch job’s failure model. The gaps that send teams looking for an orchestrator are always the same five:</p><ul><li><strong>It cannot fetch anything.</strong> No API calls, no SFTP, no S3, no Kafka. If the data is not already in Snowflake, dbt is asleep.</li><li><strong>It has a weak sense of time.</strong> dbt Cloud has schedules, dbt Core has cron and your patience. Neither reacts to a file landing or a message arriving.</li><li><strong>Retries are coarse.</strong> A classic dbt build is one process. A transient Snowflake error in model 340 of 400 means rerunning with — select result:error+, by hand, at 2am, unless someone wired that up.</li><li><strong>It knows nothing outside its own project.</strong> “Wait for the ML feature job, then rebuild marts, then refresh the Tableau extract, then notify the finance channel” is four systems. dbt sees one.</li><li><strong>Python models are a compromise.</strong> dbt Python models run as Snowpark stored procedures. Useful, genuinely, but they are still nodes inside a transformation graph, not a home for a training loop with GPU requirements.</li></ul><p>Every one of those is a Prefect sentence.</p><h3>Prefect on the same platform</h3><p>A Prefect flow is a Python function. That is not marketing, it is the whole design. There is no DAG to declare, so the shape of the run can depend on what the data turns out to be, which matters more than people expect the first time a vendor adds a region and your static DAG cannot grow a branch.</p><p>flows/revenue_ingest.py</p><p><strong>from</strong> datetime <strong>import</strong> timedelta</p><p><strong>import</strong> httpx<br><strong>from</strong> prefect <strong>import</strong> flow, task, get_run_logger<br><strong>from</strong> prefect.cache_policies <strong>import</strong> INPUTS<br><strong>from</strong> prefect_snowflake.database <strong>import</strong> SnowflakeConnector</p><p>@task(retries=3, retry_delay_seconds=[10, 60, 300], cache_policy=INPUTS,<br> cache_expiration=timedelta(hours=1), tags=[“saas-api”])<br><strong>def</strong> extract_salesforce(since: str) -&gt; str:<br> <em>“””Pull changed orders, write newline delimited JSON to an internal stage.”””</em><br> logger = get_run_logger()<br> rows, cursor = [], None<br> <strong>while</strong> True:<br> resp = httpx.get(<br> “https://api.example.com/v1/orders”,<br> params={“updated_since”: since, “cursor”: cursor},<br> timeout=60,<br> )<br> resp.raise_for_status()<br> body = resp.json()<br> rows.extend(body[“records”])<br> cursor = body.get(“next_cursor”)<br> <strong>if</strong> <strong>not</strong> cursor:<br> <strong>break</strong><br> logger.info(“pulled %s order rows since %s”, len(rows), since)<br> <strong>return</strong> write_ndjson_to_stage(rows, prefix=”orders”)</p><p>@task(retries=2, tags=[“snowflake-wh”])<br><strong>def</strong> copy_into_raw(stage_path: str, target: str) -&gt; int:<br> <strong>with</strong> SnowflakeConnector.load(“prod-loader”) <strong>as</strong> conn:<br> conn.execute(f”””<br> copy into {target}<br> from @load_stage/{stage_path}<br> file_format = (type = json)<br> on_error = abort_statement<br> “””)<br> (loaded,) = conn.fetch_one(<br> “select sum(rows_loaded) from table(information_schema.copy_history(“<br> f”table_name =&gt; ‘{target}’, start_time =&gt; dateadd(hour, -1, current_timestamp())))”<br> )<br> <strong>return</strong> loaded <strong>or</strong> 0</p><p>@flow(name=”revenue-ingest”, log_prints=True)<br><strong>def</strong> revenue_ingest(since: str = “2026–09–01”):<br> orders_path = extract_salesforce(since)<br> payments_path = extract_stripe(since) <em># same shape, omitted</em><br> loaded = copy_into_raw(orders_path, “raw.salesforce.orders”)<br> loaded += copy_into_raw(payments_path, “raw.stripe.payments”)<br> <strong>if</strong> loaded == 0:<br> <strong>raise</strong> ValueError(“nothing landed, refusing to trigger the transform”)<br> <strong>return</strong> loaded</p><p>Retries with a backoff list, a cache policy keyed on inputs so a rerun in the same hour does not re-hammer the API, and a tag that ties into a global concurrency limit so you never have forty pods fighting over one warehouse. None of that is expressible in dbt, and none of it should be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHaaymOw40hi0anUeLPFHg.png" /></figure><p><em>Figure 5. Anatomy of a flow. Triggers on the left, execution in the middle, and the part teams underestimate on the right: state, automations and concurrency limits are the reason you run an orchestrator at all.</em></p><p>Scheduling is the least interesting thing an orchestrator does. Reacting is the interesting thing. A deployment can sit idle until a specific event shows up, which is how you stop guessing whether the vendor’s file arrived by 03:00.</p><p>prefect.yaml</p><p>deployments<strong>:</strong><br> — name<strong>:</strong> revenue-to-marts<br> entrypoint<strong>:</strong> flows/revenue_to_marts.py:revenue_to_marts<br> work_pool<strong>:</strong><br> name<strong>:</strong> eks-data-pool<br> job_variables<strong>:</strong><br> image<strong>:</strong> “{{ build_image.image }}”<br> cpu<strong>:</strong> 2048<br> memory<strong>:</strong> 8192<br> parameters<strong>:</strong><br> target<strong>:</strong> prod<br> schedules<strong>:</strong><br> — cron<strong>:</strong> “0 4 * * *”<br> timezone<strong>:</strong> America/New_York<br> triggers<strong>:</strong><br> — type<strong>:</strong> event<br> enabled<strong>:</strong> true<br> match<strong>:</strong><br> prefect.resource.id<strong>:</strong> “s3-bucket.vendor-drops”<br> expect<strong>:</strong><br> — “aws.s3.object-created”<br> parameters<strong>:</strong><br> since<strong>:</strong> “{{ event.payload.object_key }}”</p><p>Cron and event triggers on the same deployment. The nightly run is the floor; the event trigger is what makes the pipeline feel instant when the vendor uploads early.</p><h3>Putting them together, which is what you will actually do</h3><p>The pattern nearly every mature Snowflake platform converges on: Prefect owns the run, dbt owns the SQL. Prefect lands the data, calls dbt, reacts to what dbt reports, and tells the rest of the business.</p><p>Until recently the handoff was crude. You shelled out to dbt build, the whole project ran as one opaque task, and if model 340 hit a transient error you rebuilt from the top. That changed in May 2026, when Prefect shipped an orchestrator that parses the dbt manifest and runs <strong>each node as its own Prefect task</strong>, with per node retries, Prefect’s concurrency limits, and caching that hashes the model SQL, its config and its upstream dependencies so unchanged work is skipped entirely. Prefect’s own write up quotes a customer estimating it would cut roughly 30 percent off their annual Snowflake bill by refusing to rebuild things that had not changed. Treat vendor numbers as vendor numbers, but the mechanism is sound: the cheapest query is the one you do not run.</p><p>flows/revenue_to_marts.py</p><p><strong>from</strong> datetime <strong>import</strong> timedelta</p><p><strong>from</strong> prefect <strong>import</strong> flow<br><strong>from</strong> prefect.blocks.notifications <strong>import</strong> SlackWebhook<br><strong>from</strong> prefect_dbt.core.settings <strong>import</strong> PrefectDbtSettings<br><strong>from</strong> prefect_dbt.core._orchestrator <strong>import</strong> (<br> PrefectDbtOrchestrator,<br> ExecutionMode,<br> CacheConfig,<br>)</p><p><strong>from</strong> flows.revenue_ingest <strong>import</strong> revenue_ingest</p><p>@flow(name=”revenue-to-marts”)<br><strong>def</strong> revenue_to_marts(since: str, target: str = “prod”):<br> rows = revenue_ingest(since) <em># subflow: extract + COPY INTO</em></p><p>settings = PrefectDbtSettings(<br> project_dir=”./analytics”,<br> profiles_dir=”./analytics”,<br> target_path=”target”,<br> )</p><p>orchestrator = PrefectDbtOrchestrator(<br> settings=settings,<br> execution_mode=ExecutionMode.PER_NODE, <em># one Prefect task per model</em><br> retries=2,<br> retry_delay_seconds=60,<br> cache=CacheConfig(expiration=timedelta(hours=24)),<br> )</p><p>result = orchestrator.run_build(select=”source:salesforce+ source:stripe+”)</p><p>SlackWebhook.load(“data-alerts”).notify(<br> f”revenue-to-marts finished: {rows} rows landed, “<br> f”{len(result.nodes)} nodes evaluated”<br> )<br> <strong>return</strong> result</p><p>The import path sits in a private module today (prefect_dbt.core._orchestrator) and the API is still marked beta, so pin your version and read the release notes before you build a platform standard on it. If you would rather dbt drove its own execution while Prefect watched from outside, PrefectDbtRunner is the conservative choice and has been stable far longer.</p><p>What that buys you is worth spelling out, because it is the strongest argument for running dbt under Prefect rather than beside it:</p><ul><li>One retry policy for the whole pipeline instead of two half policies.</li><li>A failing model retries itself; only its genuine descendants get skipped.</li><li>dbt models respect the same concurrency limit as everything else pointed at that warehouse.</li><li>One alerting path, one run page, one place to answer “why is the dashboard stale”.</li><li>No second scheduler, which is the thing that quietly doubles your on call surface.</li></ul><p>Here is the whole cycle end to end, including the part everyone designs last and needs most: what happens when a test fails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRvrQge5BK7aC0Z2gxRbBw.png" /></figure><p><em>Figure 6. An event driven run, including the failure path. Notice how much of the value sits after step 7. dbt detects the problem, Prefect decides what it means for everything downstream.</em></p><p>Zoom out and the full platform looks like this. Prefect is a rail running down the side of the whole thing rather than a box inside it, which is the mental model I would most like people to take away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H3T7O9E3iD506rhvHf_JJw.png" /></figure><p><em>Figure 7. Reference architecture. dbt occupies one broad band inside Snowflake. Prefect runs alongside the entire stack and owns none of it, which is exactly why it can coordinate all of it.</em></p><h3>The ground moved in 2026, so check your assumptions</h3><p>If your view of these two products was formed in 2024, it is out of date in ways that change architecture decisions. Three things happened in the space of eight months.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RVkV__WlCf6TGMQ2gmjMA.png" /></figure><p><em>Figure 8. Eight months of consolidation: the merger announcement in October 2025, dbt Projects on Snowflake reaching GA in November 2025, the Prefect dbt orchestrator in May 2026, the merger closing alongside dbt Core v2.0 in June 2026, and Prefect acquiring Dagster Labs in July 2026.</em></p><p><strong>dbt Labs is now part of Fivetran.</strong> The all stock merger announced in October 2025 closed on 1 June 2026, with George Fraser as CEO and Tristan Handy as President, and the combined company operating as Fivetran + dbt Labs. dbt Core v2.0 shipped at the same time carrying the Fusion engine under Apache 2.0. The practical read for a Snowflake team: ingestion and transformation now have a single commercial owner, dbt Core remains open, and the roadmap will plausibly favour teams that buy both. Nobody should rip anything out over this, but if your architecture assumed dbt was a neutral transformation layer independent of your loader, that assumption is worth revisiting at renewal time.</p><p><strong>Prefect bought Dagster.</strong> Announced 13 July 2026, with both products continuing under their own names and licences and the combined company operating as Prefect from August. Dagster’s declarative, asset first model and Prefect’s imperative, Python first model were the two main answers to “what replaces Airflow”, and they now live in the same house. If you are picking an orchestrator this quarter, that is one fewer fork in the road, and it also means Prefect’s asset and lineage story is going to get considerably more serious.</p><p><strong>Snowflake will run dbt for you.</strong> dbt Projects on Snowflake went GA in November 2025. You can import a dbt project from git into a Snowsight Workspace, deploy it as a schema level DBT PROJECT object, and schedule it with a native task. No separate runtime, no licence, no per seat fee, just warehouse compute. This is the option most teams have not properly evaluated, and for a certain shape of team it is the right answer.</p><p><em>— deploy the project from a workspace, then schedule it with a task</em><br><strong>CREATE</strong> <strong>OR</strong> <strong>ALTER</strong> TASK analytics.prod.run_dbt_project<br> WAREHOUSE = transform_wh<br> SCHEDULE = ‘6 hours’<br><strong>AS</strong><br> <strong>EXECUTE</strong> DBT PROJECT analytics.prod.revenue args=’run — target prod’;</p><p><strong>CREATE</strong> <strong>OR</strong> <strong>ALTER</strong> TASK analytics.prod.test_dbt_project<br> WAREHOUSE = transform_wh<br> <strong>AFTER</strong> analytics.prod.run_dbt_project<br><strong>AS</strong><br> <strong>EXECUTE</strong> DBT PROJECT analytics.prod.revenue args=’test — target prod’;</p><p><strong>ALTER</strong> TASK analytics.prod.test_dbt_project <strong>RESUME</strong>;<br><strong>ALTER</strong> TASK analytics.prod.run_dbt_project <strong>RESUME</strong>;</p><p>Serverless tasks are not allowed here, you must name a user managed warehouse, and the task has to live in the same database and schema as the project object. Run history, logs and OpenTelemetry traces land in Snowflake alongside everything else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T27sntfy6VzxaJiya0SGZQ.png" /></figure><p><em>Figure 9. Three camps, heavy overlap at the edges. The centre of gravity is still different for each: moving and modelling data, running and recovering workflows, storing and governing it.</em></p><h3>Three architectures that actually work</h3><p>The real decision is not dbt or Prefect. It is which of these three shapes fits your team, and all three use dbt.</p><p><strong>Pattern A, Snowflake native.</strong> Fivetran or Snowpipe lands the data, dbt is deployed as a project object, Snowflake tasks schedule it, and traces and cost management live in Snowsight. No worker fleet, no second UI, no seats. Weak the moment something outside Snowflake matters.</p><p><strong>Pattern B, Prefect plus dbt Core.</strong> Prefect flows handle extract and load, the Prefect dbt orchestrator runs the project node by node, dbt Core runs in your own image, and everything else in the estate sits in the same control plane. One on call rota, one run page. You now operate a worker fleet.</p><p><strong>Pattern C, Prefect plus managed dbt.</strong> Engineering owns ingestion and ML in Prefect, analysts own models in dbt Cloud or dbt on Snowflake, and Prefect triggers the dbt job and waits. Two UIs and coarser retries, but analysts ship without filing a platform ticket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oC9g6JJBPdljUcNjFZLpPQ.png" /></figure><p><em>Figure 10. Three working shapes. Pattern A is the one people forget exists, and it is genuinely the cheapest way to run dbt on Snowflake if your ingestion is already handled.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oO7wt9cB-3oU31km0Tb8Vg.png" /></figure><p><em>Figure 11. The decision, compressed. Two questions get most teams to an answer.</em></p><p>You have drawn the line in the wrong place if any of this sounds familiar:</p><ul><li>A macro that calls an external API, or a post-hook that sends a Slack message. That is orchestration hiding inside a compiler.</li><li>A Prefect task containing two hundred lines of hand written SQL that should have been a model with tests on it.</li><li>Two schedulers racing: a Snowflake task and a Prefect deployment both building the same marts, discovered during an incident.</li><li>dbt Cloud jobs chained by sleep timers because nobody wanted to admit there was a cross system dependency.</li></ul><h3>Where the money actually goes</h3><p>This part gets mis-argued constantly. Teams spend a month comparing orchestrator pricing while the warehouse bill quietly triples. On a Snowflake platform, compute dominates, and both tools are mostly interesting for how much of it they let you avoid.</p><p>The four levers that move the big number, roughly in order of impact: materialization choice, skipping unchanged nodes, right sizing the warehouse per model, and pruning with predicates and clustering. Two of those are orchestrator features and two are dbt features, which is the most practical argument for running them together. Also worth saying out loud: a full rebuild triggered by a retry at 3am is a cost event, not just an availability event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fo46Czi7JA9hahgVcpB3EQ.png" /></figure><p><em>Figure 12. Spend, to scale. The orchestrator line item is small; its effect on the big line item is not.</em></p><h3>The comparison table, for the people who scrolled here first</h3><p>dbt</p><p>Prefect</p><p>Primary job</p><p>Transform data that is already in the warehouse</p><p>Run, retry and observe anything, in any order, anywhere</p><p>Language</p><p>SQL with Jinja; Python models via Snowpark</p><p>Python, ordinary functions with decorators</p><p>Where code runs</p><p>Compiles locally, executes on Snowflake compute</p><p>Your containers; state and schedules in Prefect Cloud or self hosted</p><p>Unit of work</p><p>A model, a test, a snapshot, a seed</p><p>A task, a flow, a deployment</p><p>Dependency model</p><p>Static graph derived from ref() and source()</p><p>Dynamic; the graph is whatever your Python calls</p><p>Failure handling</p><p>Skip downstream, exit non-zero</p><p>Retries with backoff, timeouts, transactional rollback, automations</p><p>Triggering</p><p>Cron in dbt Cloud, or whatever calls the CLI</p><p>Schedules, events, webhooks, API, upstream runs</p><p>Data quality</p><p>Native tests, contracts, freshness, unit tests</p><p>None of its own; it runs whatever you point it at</p><p>Lineage</p><p>Model level and column level, from the project</p><p>Asset and run level, across systems</p><p>Non-SQL work</p><p>Snowpark Python models only</p><p>Anything Python can reach</p><p>Typical owner</p><p>Analytics engineers, analysts, data engineers</p><p>Data engineers, ML engineers, platform teams</p><p>Cost shape</p><p>Core is free; Cloud is per seat; on Snowflake it is warehouse time only</p><p>Open source engine plus your infra; Cloud has a free tier and paid tiers</p><p>Ownership in 2026</p><p>Fivetran + dbt Labs, merged June 2026</p><p>Prefect, which acquired Dagster Labs in July 2026</p><p>Replaces the other?</p><p>No. It cannot fetch, wait, react or page anyone.</p><p>No. It will happily run your worst SQL forever.</p><h3>What I would actually do</h3><p>If someone handed me a greenfield Snowflake platform tomorrow, I would start at Pattern A and refuse to add an orchestrator until something forced me to. Get ingestion handled by Fivetran or Snowpipe, put dbt in git, deploy it as a project object, schedule it with a task, and spend the saved months on modelling and tests instead of on a Kubernetes work pool. Most teams reach for an orchestrator far too early and end up operating one for the sake of three cron jobs.</p><p>The forcing functions are specific and you will recognise them when they arrive. A vendor starts dropping files at unpredictable times. Someone needs a model retrained and scored on a schedule tied to the marts. Finance wants the close pipeline to page a human at 04:15 rather than fail quietly. A dependency appears between two systems that Snowflake cannot see. When two or three of those are true, move to Pattern B, put dbt under the Prefect dbt orchestrator so you get one retry policy, and delete the Snowflake tasks the same week so you never have two schedulers.</p><p>The thing I would guard hardest is the boundary itself. Transformation logic belongs in models, in git, with tests and a contract. Coordination logic belongs in flows. The moment you find a macro making an HTTP call, or a Prefect task holding a two hundred line SELECT, the design has started to rot, and it rots quietly for about six months before anyone admits it.</p><p>dbt tells you what good looks like. Prefect makes sure it happens, notices when it does not, and wakes someone up.</p><p>One last thing, because the answer genuinely changes the recommendation and no article can guess it for you. Two questions decide nearly everything above. How does data get into your Snowflake account today: Fivetran, Snowpipe, or Python that somebody wrote and nobody wants to maintain? And who is going to write the models: three data engineers, or thirty analysts? Answer those honestly and the pattern picks itself.</p><h3>Sources for the 2026 claims</h3><ul><li>Fivetran and dbt Labs, merger completion, 1 June 2026: <a href="https://www.fivetran.com/press/fivetran-dbt-labs-complete-merger-to-create-the-data-infrastructure-for-trusted-ai-agents">https://www.fivetran.com/press/fivetran-dbt-labs-complete-merger-to-create-the-data-infrastructure-for-trusted-ai-agents</a></li><li>Prefect acquires Dagster Labs, 13 July 2026: <a href="https://www.prefect.io/prefect-acquires-dagster">https://www.prefect.io/prefect-acquires-dagster</a> and <a href="https://dagster.io/blog/prefect-is-acquiring-dagster">https://dagster.io/blog/prefect-is-acquiring-dagster</a></li><li>Introducing the Prefect dbt Orchestrator, 13 May 2026: <a href="https://www.prefect.io/blog/dbt-orchestrator">https://www.prefect.io/blog/dbt-orchestrator</a> and the docs at <a href="https://docs.prefect.io/integrations/prefect-dbt/orchestrator">https://docs.prefect.io/integrations/prefect-dbt/orchestrator</a></li><li>dbt Projects on Snowflake GA, 6 November 2025: <a href="https://docs.snowflake.com/en/release-notes/2025/other/2025-11-06-dbt-projects-on-snowflake-ga">https://docs.snowflake.com/en/release-notes/2025/other/2025-11-06-dbt-projects-on-snowflake-ga</a> and EXECUTE DBT PROJECT syntax at <a href="https://docs.snowflake.com/en/sql-reference/sql/execute-dbt-project">https://docs.snowflake.com/en/sql-reference/sql/execute-dbt-project</a></li></ul><p><em>Product surfaces in this space are moving monthly. The PrefectDbtOrchestrator API was still in a private module at the time of writing, and the Fivetran and dbt Labs product lines are mid integration. Check the release notes before you standardise on any of it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fe438ebf084" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Strategy Is Failing. Your Data Is Why.]]></title>
            <link>https://medium.com/@badrish.davay/your-ai-strategy-is-failing-your-data-is-why-e51c6c89bf6a?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e51c6c89bf6a</guid>
            <category><![CDATA[data-cleanup]]></category>
            <category><![CDATA[data-mes]]></category>
            <category><![CDATA[ai-without-clean-data]]></category>
            <category><![CDATA[ai-strategy]]></category>
            <category><![CDATA[curated-data]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 03:28:08 GMT</pubDate>
            <atom:updated>2026-04-21T03:28:08.853Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data &amp; AI · Opinion</em></p><p>After 23 years building data and AI systems across Financial Services, Automotive, Life Sciences, and Banking — I keep watching the same story play out: companies pour millions into AI initiatives, then wonder why nothing works.</p><p>The answer is almost never the AI itself.</p><h3>The numbers nobody wants to say out loud</h3><ul><li><strong>80% of enterprise AI initiatives fail to deliver meaningful results</strong></li><li>Millions get spent on models, tools, and platforms before anyone fixes the foundation</li><li>And not a single organization has ever fixed an AI problem by adding more AI</li></ul><h3>The data mess nobody wants to admit</h3><p>Before we even get to AI, let’s talk about what most enterprise data environments actually look like. It’s not pretty.</p><ul><li><strong>No governance. No lineage.</strong> Nobody knows where data came from or whether to trust it.</li><li><strong>Pipelines running on vibes</strong> — nobody maintains them, nobody monitors them.</li><li>The same data duplicated across a dozen systems, all slightly different.</li><li>“Data lakes” that are really just data swamps — full of everything, useful for nothing.</li></ul><h3>AI doesn’t fix bad data. It amplifies it.</h3><p>Here’s the part most vendor pitches leave out: AI is extraordinarily good at learning from whatever you give it — including garbage.</p><ul><li>Messy data trains models to produce messy conclusions — just with false confidence.</li><li>Bad inputs don’t produce obviously wrong outputs. They produce <em>plausible-sounding</em> wrong outputs. That’s worse.</li><li>Automation at scale means your data quality problems become business problems <em>faster</em>.</li><li>Once trust erodes in an AI system, it’s nearly impossible to rebuild — even after you fix the data.</li></ul><blockquote><em>The dangerous part isn’t that AI gets things wrong. It’s that AI gets things wrong </em><strong><em>confidently, at scale, and repeatedly</em></strong><em> — until someone notices the business impact.</em></blockquote><h3>This isn’t an engineering problem. It’s a leadership one.</h3><p>The data mess isn’t a mystery. Everyone knows it exists. The real issue is that cleaning it up requires someone with power to make an uncomfortable call:</p><ul><li>Pause the AI pilots</li><li>Delay the shiny roadmap</li><li>Fund unglamorous cleanup work</li><li>Commit to data governance as a permanent discipline — not a one-time sprint</li></ul><p>Data governance isn’t a project you finish. It’s an operating muscle you build over time — and that requires ongoing leadership commitment, not a quarter of effort followed by neglect.</p><h3>The question to ask in your next AI strategy meeting</h3><p><strong>“When did we last actually <em>fix</em> our data?”</strong></p><p>Not migrate it. Not move it to a new tool. Not wrap it in a new dashboard.</p><p><em>Actually cleanse it, validate it, and trust it.</em></p><p>Watch what happens in the room when you ask that.</p><h3>The bottom line</h3><p>Most organizations don’t have an AI problem. They have a <strong>data problem</strong> they’re trying to solve with more AI — which is like painting over rust and wondering why the car still breaks down.</p><p>Data is the foundation of every AI initiative. Not a nice-to-have, not a parallel workstream, not something you retrofit after the pilot. <strong>It has to come first.</strong></p><p>Until that changes, every “AI transformation” is just expensive automation built on data nobody actually trusts.</p><p><em>When did your organization last prioritize data cleanup over a new AI feature? I’d love to hear what you’re seeing in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e51c6c89bf6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark Catalyst Optimizer in nutshell]]></title>
            <link>https://medium.com/@badrish.davay/apache-spark-catalyst-optimizer-in-nutshell-570926e0ac79?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/570926e0ac79</guid>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Thu, 03 Apr 2025 20:38:56 GMT</pubDate>
            <atom:updated>2025-04-03T20:47:24.400Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Apache Spark’s Catalyst Optimizer</h3><p>In the world of big data processing, Apache Spark stands out as a powerful framework for distributed computing. At the heart of Spark’s exceptional performance lies the Catalyst optimizer — an advanced query optimization engine that transforms how we work with massive datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AXZ3Q4GgDuNKDncw0bb-UQ.png" /></figure><h3>What is the Catalyst Optimizer?</h3><p>Catalyst is Spark’s query optimization framework that enables high-performance SQL queries and DataFrame operations. First introduced in Spark 1.0 and significantly enhanced since then, Catalyst serves as the brain behind Spark SQL, intelligently planning how to execute your data transformations.</p><h3>How Catalyst Works: The Optimization Pipeline</h3><p>The Catalyst optimization process follows a systematic workflow:</p><h3>1. Query Analysis</h3><p>When you write a DataFrame or SQL query, Catalyst first converts it into an unresolved logical plan. This initial representation captures your intent but doesn’t yet know about column types or data sources.</p><h3>2. Logical Optimization</h3><p>After resolving column references and data types, Catalyst applies a series of rule-based optimizations to the logical plan:</p><ul><li><strong>Constant folding</strong>: Evaluates expressions with constant values at compile time</li><li><strong>Predicate pushdown</strong>: Filters data as early as possible in the pipeline</li><li><strong>Column pruning</strong>: Drops unnecessary columns to reduce data transfer</li><li><strong>Partition pruning</strong>: Skips irrelevant partitions based on filter conditions</li><li><strong>Join reordering</strong>: Arranges joins to minimize intermediate result sizes</li></ul><h3>3. Physical Planning</h3><p>The logical plan is then converted into multiple physical plans. Each physical plan represents a different way to execute the query using the available physical operators like broadcast joins, sort-merge joins, and various aggregation strategies.</p><h3>4. Cost-Based Optimization</h3><p>Catalyst evaluates these physical plans using its cost model, which considers factors like:</p><ul><li>Data size and distribution</li><li>Available resources</li><li>Operation complexity</li><li>Statistics about the data</li></ul><p>The plan with the lowest estimated cost is selected for execution.</p><h3>5. Code Generation</h3><p>Finally, Spark generates efficient JVM bytecode for the selected plan using a technique called whole-stage code generation. This step collapses multiple operations into a single function wherever possible, reducing virtual function calls and optimizing CPU usage.</p><h3>Best Practices for Large Datasets</h3><p>When dealing with massive datasets, Catalyst becomes your greatest ally if you follow these optimization strategies:</p><ol><li><strong>Let Catalyst do the heavy lifting</strong>: Use DataFrame/Dataset APIs instead of RDDs to leverage Catalyst’s optimizations.</li><li><strong>Provide schema information</strong>: Explicitly defining your schema helps Catalyst make better decisions than inferring it at runtime.</li><li><strong>Collect statistics</strong>: Use ANALYZE TABLE COMPUTE STATISTICS commands to help Catalyst make data-informed decisions.</li><li><strong>Leverage predicate pushdown</strong>: Write your queries to filter early, allowing Catalyst to push predicates to data sources.</li><li><strong>Choose the right join strategy</strong>: For highly skewed data, use broadcast joins for small tables; for large tables, ensure proper partitioning.</li><li><strong>Partition wisely</strong>: Design partitioning schemes that match your query patterns to enable partition pruning.</li><li><strong>Avoid UDFs when possible</strong>: Standard Spark functions allow better optimization than custom UDFs.</li><li><strong>Monitor query plans</strong>: Use explain() to understand how Catalyst is executing your queries and identify opportunities for improvement.</li></ol><p>By understanding and leveraging Catalyst’s optimization capabilities, you can achieve remarkable performance gains when processing terabytes or even petabytes of data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=570926e0ac79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ultimate Guide to Data Catalogs: Making the Right Choice for Your Organization’s Data Products]]></title>
            <link>https://medium.com/@badrish.davay/the-ultimate-guide-to-data-catalogs-making-the-right-choice-for-your-organizations-data-products-cfe15d07bbae?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfe15d07bbae</guid>
            <category><![CDATA[data-as-a-product]]></category>
            <category><![CDATA[enterprise-catalog]]></category>
            <category><![CDATA[comparison-of-catalogs]]></category>
            <category><![CDATA[data-catalog]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 14:25:17 GMT</pubDate>
            <atom:updated>2025-03-03T14:25:17.605Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In today’s data-driven world, organizations are increasingly focused on treating data as a product — a valuable asset that requires proper management, governance, and accessibility. At the heart of this approach lies the data catalog, a critical tool that helps organizations discover, understand, and effectively utilize their data assets. However, with numerous options available in the market — from Atlan and DataHub to OpenMetadata and beyond — choosing the right data catalog can be a daunting task.</p><p>This blog aims to provide a comprehensive analysis of different data catalogs, the challenges organizations face in their data product journey, and a framework to help you determine which solution best aligns with your specific requirements.</p><h3>Understanding Data Catalogs and Their Role in Data Products</h3><p>A data catalog serves as an inventory of all data assets within an organization. It provides metadata, lineage information, data quality metrics, and other contextual information that helps users find, understand, and trust data. In the context of data products, a catalog is more than just a repository — it’s the foundation that enables data discovery, governance, and collaboration across teams.</p><h3>Core Functions of Modern Data Catalogs</h3><ol><li><strong>Data Discovery and Search</strong>: Enabling users to find relevant data assets quickly</li><li><strong>Metadata Management</strong>: Capturing technical and business metadata</li><li><strong>Data Lineage</strong>: Tracking data flow and transformations</li><li><strong>Data Quality Monitoring</strong>: Providing insights into the reliability of data</li><li><strong>Access Control</strong>: Managing who can view and use specific data assets</li><li><strong>Collaboration</strong>: Facilitating knowledge sharing among data users</li><li><strong>Integration Capabilities</strong>: Connecting with various data sources and tools</li></ol><h3>Common Challenges in the Data Product Lifecycle</h3><p>Before diving into specific data catalog solutions, let’s examine the challenges organizations typically face throughout the data product lifecycle:</p><h3>Discovery Phase Challenges</h3><ul><li><strong>Data Silos</strong>: Information trapped in isolated systems</li><li><strong>Undocumented Data</strong>: Lack of context about data’s meaning and origin</li><li><strong>Search Inefficiency</strong>: Difficulty finding relevant data assets</li><li><strong>Duplicate Data</strong>: Multiple versions of similar datasets</li></ul><h3>Development Phase Challenges</h3><ul><li><strong>Limited Understanding of Dependencies</strong>: Unclear data lineage</li><li><strong>Quality Uncertainties</strong>: Unknown data reliability</li><li><strong>Inconsistent Metadata</strong>: Varying standards across departments</li><li><strong>Integration Complexity</strong>: Connecting various tools and platforms</li></ul><h3>Deployment Phase Challenges</h3><ul><li><strong>Governance Issues</strong>: Ensuring compliance with regulations</li><li><strong>Access Control Complexity</strong>: Balancing security with accessibility</li><li><strong>Change Management</strong>: Tracking updates to data and models</li><li><strong>Performance Monitoring</strong>: Ensuring data products meet SLAs</li></ul><h3>Maintenance Phase Challenges</h3><ul><li><strong>Data Drift</strong>: Changes in data characteristics over time</li><li><strong>Outdated Documentation</strong>: Metadata that no longer reflects reality</li><li><strong>Trust Issues</strong>: Declining confidence in data products</li><li><strong>Scalability Concerns</strong>: Managing growing data volumes</li></ul><h3>Detailed Analysis of Leading Data Catalog Solutions</h3><p>Let’s examine some of the most prominent data catalog solutions in the market, focusing on their strengths, limitations, and ideal use cases.</p><h3>1. Atlan</h3><p><strong>Overview</strong>: Atlan positions itself as a third-generation data catalog with a strong focus on collaboration and modern data stack integration.</p><p><strong>Key Strengths</strong>:</p><ul><li>Highly intuitive, consumer-grade UI designed for business users</li><li>Strong collaboration features including discussions, annotations, and tasks</li><li>Robust API and extensive integration ecosystem</li><li>Active metadata capabilities with automation features</li><li>Embedded data quality profiling and monitoring</li></ul><p><strong>Limitations</strong>:</p><ul><li>Relatively newer player in the market</li><li>May require additional customization for highly regulated industries</li><li>Premium pricing compared to some open-source alternatives</li></ul><p><strong>Ideal For</strong>: Organizations with modern data stacks prioritizing collaboration and user experience, particularly those using cloud data warehouses and transformation tools.</p><h3>2. DataHub (LinkedIn/Open Source)</h3><p><strong>Overview</strong>: DataHub is an open-source metadata platform originally developed by LinkedIn that focuses on end-to-end data lineage and comprehensive metadata management.</p><p><strong>Key Strengths</strong>:</p><ul><li>Strong technical metadata capabilities</li><li>Excellent data lineage visualization</li><li>Highly extensible through plugins</li><li>Active open-source community</li><li>Scalable architecture designed for large enterprises</li></ul><p><strong>Limitations</strong>:</p><ul><li>Steeper learning curve for implementation</li><li>User interface less intuitive for business users</li><li>Requires significant technical resources to deploy and maintain</li><li>Documentation can be challenging for newcomers</li></ul><p><strong>Ideal For</strong>: Technically sophisticated organizations with strong engineering resources who value customizability and have complex lineage requirements.</p><h3>3. OpenMetadata</h3><p><strong>Overview</strong>: A newer open-source data catalog focused on APIs, simplicity, and modern architecture.</p><p><strong>Key Strengths</strong>:</p><ul><li>API-first approach making it highly integrable</li><li>Modern, microservices-based architecture</li><li>Strong data quality features</li><li>Growing ecosystem of connectors</li><li>Simplified deployment compared to some alternatives</li></ul><p><strong>Limitations</strong>:</p><ul><li>Smaller community compared to some competitors</li><li>Still evolving certain enterprise features</li><li>Limited commercial support options</li><li>Fewer out-of-the-box integrations than commercial alternatives</li></ul><p><strong>Ideal For</strong>: Organizations that value flexibility and want an open-source solution with modern architecture and strong API capabilities.</p><h3>4. Alation</h3><p><strong>Overview</strong>: One of the pioneers in the data catalog space, Alation offers a comprehensive enterprise solution with strong query intelligence.</p><p><strong>Key Strengths</strong>:</p><ul><li>Robust SQL query features including recommendations</li><li>Strong business glossary and semantic layer</li><li>Extensive enterprise adoption and proven scalability</li><li>Advanced governance capabilities</li><li>Comprehensive stewardship workflows</li></ul><p><strong>Limitations</strong>:</p><ul><li>Higher price point</li><li>Can be complex to implement fully</li><li>May include features unnecessary for smaller organizations</li><li>Some users report steep learning curve</li></ul><p><strong>Ideal For</strong>: Large enterprises with complex governance requirements and significant SQL usage across multiple database platforms.</p><h3>5. Collibra</h3><p><strong>Overview</strong>: Collibra began as a data governance platform and expanded to include catalog capabilities, making it particularly strong for regulated industries.</p><p><strong>Key Strengths</strong>:</p><ul><li>Industry-leading governance capabilities</li><li>Comprehensive business glossary</li><li>Strong workflow automation</li><li>Robust policy management</li><li>Deep integration with enterprise systems</li></ul><p><strong>Limitations</strong>:</p><ul><li>Primary focus on governance can make it feel less intuitive for pure discovery use cases</li><li>Implementation complexity can be high</li><li>Premium pricing</li><li>Technical metadata capabilities less developed than some alternatives</li></ul><p><strong>Ideal For</strong>: Highly regulated industries (finance, healthcare) where governance and compliance are primary concerns.</p><h3>6. Amundsen (Lyft/Open Source)</h3><p><strong>Overview</strong>: Created by Lyft, Amundsen focuses on data discovery with a simple, search-centric approach.</p><p><strong>Key Strengths</strong>:</p><ul><li>Lightweight and focused on core discovery needs</li><li>Graph-based architecture enabling powerful relationship mapping</li><li>Strong search capabilities</li><li>Good for technical users</li><li>Relatively simple deployment compared to some alternatives</li></ul><p><strong>Limitations</strong>:</p><ul><li>More limited feature set compared to comprehensive solutions</li><li>Less business metadata functionality</li><li>Smaller ecosystem of integrations</li><li>Requires more custom development for advanced use cases</li></ul><p><strong>Ideal For</strong>: Organizations seeking a lightweight, search-focused solution or those starting their data catalog journey with basic discovery needs.</p><h3>7. Microsoft Purview (formerly Azure Data Catalog)</h3><p><strong>Overview</strong>: Microsoft’s unified data governance service that combines catalog capabilities with compliance features.</p><p><strong>Key Strengths</strong>:</p><ul><li>Seamless integration with Microsoft ecosystem</li><li>Strong compliance and regulatory features</li><li>Automated scanning and classification</li><li>Built-in data lineage capabilities</li><li>Good balance of technical and business metadata</li></ul><p><strong>Limitations</strong>:</p><ul><li>Primarily Azure-focused, though expanding</li><li>Less effective in highly heterogeneous environments</li><li>UI can be complex for non-technical users</li><li>More focused on governance than collaboration</li></ul><p><strong>Ideal For</strong>: Organizations heavily invested in Microsoft technologies and seeking integrated governance capabilities.</p><h3>8. AWS Glue Data Catalog</h3><p><strong>Overview</strong>: Amazon’s serverless data catalog integrated with their broader data services.</p><p><strong>Key Strengths</strong>:</p><ul><li>Tight integration with AWS ecosystem</li><li>Serverless architecture requiring minimal management</li><li>Cost-effective for AWS customers</li><li>Automated schema discovery</li><li>Seamless ETL capabilities</li></ul><p><strong>Limitations</strong>:</p><ul><li>Limited business metadata capabilities</li><li>Less collaborative features</li><li>Primarily technical focus</li><li>Limited functionality outside AWS ecosystem</li></ul><p><strong>Ideal For</strong>: AWS-centric organizations using services like S3, Redshift, and EMR who want an integrated, low-maintenance solution.</p><h3>Comparative Analysis: Data Catalog Feature Matrix</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4zNfvY1whhCZPPs841K-g.png" /></figure><p>Rating Scale:<br>★★★★★ = Excellent<br>★★★★☆ = Very Good<br>★★★☆☆ = Good<br>★★☆☆☆ = Basic<br>★☆☆☆☆ = Minimal</p><h3>Tailored Recommendations</h3><h3>For Enterprise Organizations</h3><ol><li>Alation: For those prioritizing SQL intelligence and enterprise-grade features</li><li>Collibra: For heavily regulated industries with strong governance needs</li><li>Atlan: For those modernizing their data stack while maintaining enterprise requirements</li></ol><h3>For Mid-Market Companies</h3><ol><li>Atlan: For collaborative teams seeking quick time-to-value</li><li>Microsoft Purview: For Microsoft-centric organizations</li><li>DataHub: For technically strong teams willing to invest in open-source</li></ol><h3>For Startups and Small Businesses</h3><ol><li>OpenMetadata: For technical teams seeking a lightweight open-source solution</li><li>Amundsen: For basic discovery needs with minimal overhead</li><li>AWS Glue Data Catalog: For AWS-focused organizations seeking integration</li></ol><h3>For Data-Intensive Industries (Finance, Healthcare)</h3><ol><li>Collibra: For comprehensive governance and regulatory capabilities</li><li>Alation: For strong lineage and control requirements</li><li>Microsoft Purview: For its built-in classification and compliance features</li></ol><h3>How to Choose the Right Data Catalog for Your Organization</h3><p>Selecting the most appropriate data catalog requires a thoughtful evaluation process aligned with your organization’s specific needs. Here’s a structured approach to guide your decision-making:</p><h3>1. Assess Your Organization’s Maturity and Needs</h3><p>Begin by evaluating where your organization stands in its data journey:</p><ul><li><strong>Data Volume and Complexity</strong>: How much data do you manage, and how diverse are your data sources?</li><li><strong>User Base</strong>: Who will use the catalog (technical teams, business analysts, executives)?</li><li><strong>Existing Tools</strong>: What systems need to integrate with your catalog?</li><li><strong>Regulatory Requirements</strong>: What compliance standards must you meet?</li><li><strong>Data Culture</strong>: Is your organization collaborative or more siloed?</li></ul><h3>2. Prioritize Your Requirements</h3><p>Not all features carry equal weight for every organization. Rank these aspects based on your specific context:</p><ul><li><strong>Discovery vs. Governance</strong>: Is finding data or controlling it more critical?</li><li><strong>Technical vs. Business Metadata</strong>: Which type of metadata is more important?</li><li><strong>Self-Service vs. Managed</strong>: Do you want users to contribute or a centralized team?</li><li><strong>Integration Depth</strong>: How seamlessly must it connect with existing tools?</li><li><strong>Implementation Resources</strong>: What technical capabilities do you have for deployment?</li></ul><h3>3. Consider Total Cost of Ownership</h3><p>Look beyond the license fees to understand the full investment required:</p><ul><li><strong>Implementation Costs</strong>: Time and resources needed for setup</li><li><strong>Maintenance Overhead</strong>: Ongoing management requirements</li><li><strong>Training Requirements</strong>: Learning curve for administrators and users</li><li><strong>Customization Needs</strong>: Development work for specific requirements</li><li><strong>Scaling Costs</strong>: How expenses grow as your data ecosystem expands</li></ul><h3>4. Evaluate the Ecosystem and Community</h3><p>The surrounding support system can significantly impact long-term success:</p><ul><li><strong>Vendor Support</strong>: Quality and responsiveness of official support</li><li><strong>Community Size</strong>: For open-source options, how active is the community?</li><li><strong>Documentation Quality</strong>: How comprehensive and clear are the guides?</li><li><strong>Release Cadence</strong>: How frequently is the product updated?</li><li><strong>Partner Network</strong>: Available implementation partners and integrations</li></ul><h3>5. Test with Proof of Concept</h3><p>Before committing fully:</p><ul><li>Select 2–3 promising solutions based on your analysis</li><li>Implement small-scale pilots with real data and users</li><li>Gather feedback on usability, performance, and feature adequacy</li><li>Assess the vendor relationship during the evaluation process</li></ul><h3>Tailored Recommendations for Different Organization Types</h3><h3>For Enterprise Organizations</h3><p><strong>Primary Considerations</strong>: Scalability, governance, integration with legacy systems, and comprehensive support</p><p><strong>Top Contenders</strong>:</p><ol><li><strong>Alation</strong>: For those prioritizing SQL intelligence and enterprise-grade features</li><li><strong>Collibra</strong>: For heavily regulated industries with strong governance needs</li><li><strong>Atlan</strong>: For those modernizing their data stack while maintaining enterprise requirements</li></ol><h3>For Mid-Market Companies</h3><p><strong>Primary Considerations</strong>: Balance of features, ease of implementation, and growth flexibility</p><p><strong>Top Contenders</strong>:</p><ol><li><strong>Atlan</strong>: For collaborative teams seeking quick time-to-value</li><li><strong>Microsoft Purview</strong>: For Microsoft-centric organizations</li><li><strong>DataHub</strong>: For technically strong teams willing to invest in open-source</li></ol><h3>For Startups and Small Businesses</h3><p><strong>Primary Considerations</strong>: Cost-effectiveness, ease of use, and essential functionality</p><p><strong>Top Contenders</strong>:</p><ol><li><strong>OpenMetadata</strong>: For technical teams seeking a lightweight open-source solution</li><li><strong>Amundsen</strong>: For basic discovery needs with minimal overhead</li><li><strong>AWS Glue Data Catalog</strong>: For AWS-focused organizations seeking integration</li></ol><h3>For Data-Intensive Industries (Finance, Healthcare)</h3><p><strong>Primary Considerations</strong>: Governance, compliance, and security features</p><p><strong>Top Contenders</strong>:</p><ol><li><strong>Collibra</strong>: For comprehensive governance and regulatory capabilities</li><li><strong>Alation</strong>: For strong lineage and control requirements</li><li><strong>Microsoft Purview</strong>: For its built-in classification and compliance features</li></ol><h3>Conclusion: Building a Future-Proof Data Catalog Strategy</h3><p>The right data catalog is not merely a tool but a foundational element of your data strategy. As you evaluate options, consider not just your current needs but your organization’s trajectory:</p><ol><li><strong>Start with clarity on your data product vision</strong>: What role will data play in your organization’s future?</li><li><strong>Build versus buy carefully</strong>: Consider whether customizing an open-source solution or purchasing a commercial product better serves your long-term goals.</li><li><strong>Plan for adoption</strong>: The most feature-rich catalog is worthless without user adoption — prioritize usability and integration with existing workflows.</li><li><strong>Consider hybrid approaches</strong>: Some organizations benefit from multiple specialized tools rather than a single comprehensive solution.</li><li><strong>Focus on outcomes</strong>: Measure success not by features implemented but by improvements in data discovery, quality, and decision-making.</li></ol><p>Remember that a data catalog is a living system that evolves with your organization. The most successful implementations are those that grow organically with user feedback, adapting to changing needs while maintaining core principles of discoverability, governance, and trust.</p><p>By thoughtfully evaluating your requirements against the strengths and limitations of available solutions, you can select a data catalog that not only meets your current needs but also supports your data product journey for years to come.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfe15d07bbae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Databricks Lakehouse Reference Architectures: A Comprehensive Guide using Databricks…]]></title>
            <link>https://medium.com/@badrish.davay/understanding-databricks-lakehouse-reference-architectures-a-comprehensive-guide-1e7a63b506e2?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e7a63b506e2</guid>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Mon, 17 Feb 2025 16:03:35 GMT</pubDate>
            <atom:updated>2025-02-17T16:04:36.670Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Databricks Lakehouse Reference Architectures: A Comprehensive Guide using Databricks on AWS</h3><p>The modern data landscape demands sophisticated solutions that can handle diverse data types, processing needs, and analytical requirements. The Databricks lakehouse architecture provides a unified platform that combines the flexibility of data lakes with the reliability of data warehouses. This article explores the architectural components and capabilities of the Databricks lakehouse, particularly focusing on its implementation on AWS.</p><h3>Core Architecture Components</h3><p>The Databricks lakehouse architecture is organized into distinct functional layers that work together to provide a complete data solution. Let’s explore each component in detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJoWZV1_in2ejQHgiWhMkA.png" /></figure><h3>Source Layer</h3><p>The architecture supports various data sources, categorized into:</p><ul><li>Semi-structured and unstructured data (sensors, IoT devices, media files, logs)</li><li>Structured data (traditional RDBMS, business applications)</li><li>SQL sources that can be integrated through lakehouse federation without ETL</li><li>Cross-cloud data sources</li></ul><h3>Ingestion Layer</h3><p>Data ingestion in the Databricks lakehouse occurs through multiple pathways:</p><ol><li>Databricks LakeFlow Connect: This offers built-in connectors for enterprise applications and databases, with pipelines governed by Unity Catalog and powered by serverless compute and Delta Live Tables.</li><li>Auto Loader: Enables direct loading of files delivered to cloud storage, providing an efficient path for batch data ingestion.</li><li>Streaming Ingestion: Supports real-time data processing through Databricks Structured Streaming, handling data from various sources like Kafka and sensor networks.</li></ol><h3>Storage Layer</h3><p>The storage layer primarily utilizes cloud storage systems, implementing the medallion architecture to organize data in Delta files and tables. This approach ensures data quality and provides clear lineage as data moves through different processing stages.</p><h3>Transform and Query Processing Layer</h3><p>This layer leverages two powerful engines:</p><ul><li>Apache Spark for distributed processing</li><li>Photon for high-performance query execution</li></ul><p>Delta Live Tables (DLT) serves as a declarative framework for building reliable, maintainable data processing pipelines. The platform supports both SQL queries via SQL warehouses and mixed workload processing through workspace clusters.</p><h3>Serving Layer</h3><p>The serving layer provides multiple options for data access:</p><p>Data Warehouse and BI:</p><ul><li>Databricks SQL with both serverless and traditional SQL warehouses</li><li>Support for various BI tools and dashboarding solutions</li></ul><p>Machine Learning:</p><ul><li>Model serving for scalable, real-time inference</li><li>Mosaic AI Gateway for managing generative AI models</li></ul><p>Data Sharing:</p><ul><li>Delta Sharing for secure partner data access</li><li>Databricks Marketplace for data product exchange</li></ul><h3>Unified Governance and Management</h3><h3>Unity Catalog</h3><p>At the heart of Databricks’ governance strategy lies Unity Catalog, providing:</p><ul><li>Centralized data access policy management</li><li>Cross-workspace governance</li><li>Comprehensive asset management (tables, volumes, features, models)</li><li>Runtime data lineage tracking</li><li>Integration with enterprise catalogs</li></ul><h3>Monitoring and Observability</h3><p>The platform includes robust monitoring capabilities:</p><ul><li>Data quality monitoring across tables</li><li>Machine learning model performance tracking</li><li>System tables for operational data analytics</li><li>Comprehensive observability features</li></ul><h3>Integration Capabilities</h3><p>The Databricks platform offers extensive integration options:</p><p>Identity Management:</p><ul><li>Support for standard identity providers</li><li>Single Sign-On (SSO) capabilities</li></ul><p>AI Services Integration:</p><ul><li>Direct integration with OpenAI, LangChain, HuggingFace</li><li>Support for custom AI model deployment</li></ul><p>Orchestration:</p><ul><li>Comprehensive REST API</li><li>Connectors for external orchestration tools like Apache Airflow</li></ul><h3>Data Intelligence Features</h3><p>The platform incorporates advanced intelligence capabilities:</p><p>DatabricksIQ:</p><ul><li>Combines generative AI with lakehouse benefits</li><li>Understands unique data semantics</li></ul><p>Databricks Assistant:</p><ul><li>Context-aware AI assistance</li><li>Available in notebooks, SQL editor, and file editor</li></ul><h3>Deployment Patterns and Use Cases</h3><h3>1. Data Ingestion with LakeFlow Connect</h3><p>Provides streamlined data ingestion with:</p><ul><li>Built-in enterprise application connectors</li><li>Unity Catalog governance</li><li>Serverless compute optimization</li><li>Incremental processing capabilities</li></ul><h3>2. Batch ETL Processing</h3><p>Implements robust data processing through:</p><ul><li>Source-specific adapters</li><li>Auto Loader integration</li><li>DLT-powered transformations</li><li>Unity Catalog governance</li></ul><h3>3. Streaming and CDC</h3><p>Enables real-time data processing using:</p><ul><li>Spark Structured Streaming</li><li>Integration with event queues (Kafka, Kinesis)</li><li>Change Data Capture support</li><li>Real-time transformation capabilities</li></ul><h3>4. Machine Learning and AI</h3><p>Provides comprehensive ML capabilities through:</p><ul><li>Mosaic AI integration</li><li>Feature Store functionality</li><li>Model registry</li><li>AutoML capabilities</li><li>MLflow integration</li></ul><h3>5. Generative AI Applications</h3><p>Supports advanced AI implementations with:</p><ul><li>State-of-the-art libraries</li><li>Prompt engineering capabilities</li><li>Model fine-tuning support</li><li>Vector search integration</li><li>RAG implementation support</li></ul><h3>6. BI and SQL Analytics</h3><p>Enables sophisticated analytics through:</p><ul><li>Databricks SQL</li><li>Integration with BI tools</li><li>Unity Catalog governance</li><li>Serverless query processing</li></ul><h3>Best Practices and Implementation Considerations</h3><p>When implementing the Databricks lakehouse architecture, consider:</p><ol><li>Start with Unity Catalog implementation for consistent governance</li><li>Leverage serverless compute where possible for cost optimization</li><li>Implement proper monitoring through system tables</li><li>Use Delta Live Tables for reliable pipeline development</li><li>Plan for proper data organization using the medallion architecture</li></ol><h3>Conclusion</h3><p>The Databricks lakehouse architecture provides a comprehensive framework for modern data management and analytics. By understanding its components and capabilities, organizations can effectively implement solutions that meet their data processing, analytics, and AI needs while maintaining governance and scalability.</p><p>This architecture’s flexibility allows it to adapt to various use cases while providing enterprise-grade security, performance, and governance. Organizations can use these reference architectures as blueprints, adapting them to their specific requirements while maintaining architectural integrity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e7a63b506e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modernizing Data Architecture with Databricks: A Holistic Approach Data as a Product]]></title>
            <link>https://medium.com/@badrish.davay/modernizing-data-architecture-with-databricks-a-holistic-approach-a30645411cc1?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a30645411cc1</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-modernization]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[data-as-a-product]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Mon, 17 Feb 2025 15:17:30 GMT</pubDate>
            <atom:updated>2025-02-17T15:52:50.467Z</atom:updated>
            <content:encoded><![CDATA[<h3>Market Context and Opportunity</h3><p>Our market analysis reveals a significant gap in the data modernization landscape. While many organizations have invested heavily in cloud migrations, they’re not seeing the expected returns on their investments. The root causes we’ve identified include:</p><ol><li>Technology-first approaches that neglect people and processes</li><li>Lack of proper data product management practices</li><li>Disconnected governance and quality frameworks</li><li>Insufficient focus on business value realization</li></ol><p>Other Critical Issues we see :</p><ul><li>Data quality and trustworthiness issues persist despite modern technology</li><li>Higher than expected cloud costs</li><li>Difficulty in presenting data as a product to business users</li><li>Loss of good practices during migration from legacy systems</li><li>Gaps in existing platform capabilities</li></ul><p>This creates a unique opportunity for us to position ourselves as strategic partners who can help organizations truly transform their data operations, not just their technology stack.</p><h3>Our Proposed Solution</h3><p>We’ve developed a comprehensive reference architecture and implementation approach that addresses these challenges by combining Databricks’ technical capabilities with our expertise in organizational transformation. Our solution stands out because:</p><ol><li>It treats data as a product, not just an asset, emphasizing quality, usability, and business value</li><li>It integrates seamlessly with Databricks’ latest features while filling critical gaps</li><li>It provides a clear path from assessment to implementation</li><li>It addresses both technical excellence and organizational change</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3NhXGMrALDOd6cXobKwlQg.png" /></figure><p>Let me explain the different layers of the reference architecture and their significance:</p><p><strong>Business Value Layer (Top Layer) </strong>This layer represents where value is ultimately delivered to the business through:</p><ul><li>Data Marketplace: Where data products are discovered and accessed</li><li>Analytics Dashboards: For business intelligence and reporting</li><li>ML Applications: For advanced analytics and AI use cases</li></ul><p><strong>Data Product Layer</strong> This crucial middle layer implements our data product approach through:</p><ul><li>Data Contracts: Formal agreements defining quality, access, and usage</li><li>Quality Metrics: Measurements and monitoring of data quality</li><li>Data Products: The actual data assets built to contract specifications</li></ul><p><strong>Databricks Platform Layer</strong> This layer leverages key Databricks capabilities:</p><ul><li>Unity Catalog: For governance, discovery, and access control</li><li>Delta Live Tables: For reliable data pipeline development</li><li>Lakehouse Monitor: For quality and performance monitoring</li><li>Auto Loader: For efficient data ingestion</li></ul><p><strong>Infrastructure Layer (Foundation)</strong> The base layer providing essential services:</p><ul><li>Cloud Storage: For data persistence and management</li><li>Compute Resources: For processing and analytics</li><li>Security Services: For protection and compliance</li></ul><p>The connecting lines between layers show how components interact:</p><ul><li>Business applications consume data products</li><li>Data products are built on Databricks platform capabilities</li><li>Platform services utilize infrastructure resources</li></ul><p><strong>Some key points about this architecture</strong>:</p><p><strong>Layered Separation</strong>: Each layer has distinct responsibilities but works together cohesively. This makes it easier to:</p><ul><li>Manage changes within each layer</li><li>Scale components independently</li><li>Maintain clear governance boundaries</li></ul><p><strong>Platform Integration</strong>: The architecture leverages Databricks’ strengths while adding necessary components for enterprise needs:</p><ul><li>Built-in governance through Unity Catalog</li><li>Automated quality control with Delta Live Tables</li><li>Comprehensive monitoring capabilities</li></ul><p><strong>Business Focus</strong>: The architecture maintains a clear line of sight to business value:</p><ul><li>Every lower layer supports business outcomes</li><li>Clear paths for data product development</li><li>Emphasis on quality and usability</li></ul><h3>Strategic Benefits</h3><p>For our practice:</p><ul><li>Positions us as thought leaders in data modernization</li><li>Creates opportunities for long-term client partnerships</li><li>Enables us to build repeatable solutions and accelerators</li><li>Differentiates us from pure technology implementers</li></ul><p>For our clients:</p><ul><li>Reduces time to value for cloud investments</li><li>Improves data quality and trust</li><li>Enables true self-service analytics</li><li>Creates a foundation for AI/ML initiatives</li></ul><h3>Understanding Data Products in Modern Enterprises</h3><p>A Data product is fundamentally “a product that facilitates an end goal through the use of data.” This encompasses not just traditional tabular data, but extends to ML models, dashboards, and other data-driven assets. The success of these data products hinges on five essential characteristics:</p><h3>Core Characteristics of Enterprise Data Products</h3><p><strong>Quality and Observability</strong> Quality encompasses accuracy, consistency, reliability, and timeliness. In our reference architecture, we implement this through:</p><ul><li>Defined quality metrics with continuous monitoring</li><li>Clear documentation and metadata management</li><li>Automated quality checks using Delta Live Tables (DLT)</li><li>Integration with Lakehouse Monitoring for proactive quality assurance</li></ul><p><strong>Semantic Consistency</strong> For a lakehouse architecture to succeed, data products must speak the same language. We achieve this through:</p><ul><li>Standardized terminology across data products</li><li>Unified governance rules in Unity Catalog</li><li>Consistent data modeling patterns</li><li>Shared reference data and master data management</li></ul><p><strong>Privacy and Security</strong> Modern data products must balance accessibility with protection:</p><ul><li>Granular access controls through Unity Catalog</li><li>PII detection and handling</li><li>Data residency management</li><li>Encryption and anonymization where required</li><li>Compliance with regulations like GDPR and CCPA</li></ul><p><strong>Discoverability and Usability</strong> Data products must be easily found and understood:</p><ul><li>Central discovery through Unity Catalog</li><li>Rich metadata and documentation</li><li>Sample notebooks and usage examples</li><li>Clear lineage visualization</li><li>Integration with enterprise catalogs like Alation and Collibra</li></ul><h3>The Reference Architecture</h3><p>Our reference architecture is designed to support the complete data product lifecycle while maintaining these core characteristics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOjJdf-L6SsglH9-eanejg.png" /></figure><h3>Foundation Layer</h3><p><strong>Databricks Lakehouse Platform</strong></p><ul><li>Delta Lake for ACID compliance and versioning</li><li>Unity Catalog for governance and discovery</li><li>MLflow for model lifecycle management</li><li>Databricks SQL for analytics workloads</li></ul><p><strong>Data Product Development Framework</strong></p><ul><li>Delta Live Tables for reliable pipeline development</li><li>Auto Loader for incremental data ingestion</li><li>Quality checks and expectations built into pipelines</li><li>Automated testing and validation</li></ul><p><strong>Operational Excellence</strong></p><ul><li>Lakehouse Monitoring for quality metrics</li><li>System Tables for usage analytics</li><li>Automated lineage tracking</li><li>Performance optimization tools</li></ul><h3>Implementation Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*ccHq6AfLpywiXbk60sJtGQ.png" /></figure><p>The architecture supports a structured data product lifecycle:</p><p><strong>Inception Phase</strong></p><ul><li>Business value definition</li><li>Owner assignment</li><li>Performance metrics establishment</li><li>Quality criteria definition</li></ul><p><strong>Design and Creation</strong></p><ul><li>Data contract development</li><li>Schema design and validation</li><li>Pipeline implementation using DLT</li><li>Quality checks integration</li></ul><p><strong>Publication and Governance</strong></p><ul><li>Certification process implementation</li><li>Access control configuration</li><li>Documentation publishing</li><li>Usage tracking setup</li></ul><p><strong>Operations and Evolution</strong></p><ul><li>Continuous monitoring</li><li>Performance optimization</li><li>User feedback integration</li><li>Iterative improvements</li></ul><h3>Data Contracts: The Foundation of Trust</h3><p>Data contracts serve as the cornerstone of our architecture, providing a formal agreement between producers and consumers. Each contract includes:</p><p><strong>Comprehensive Documentation</strong></p><ul><li>Detailed data descriptions</li><li>Schema specifications</li><li>Quality metrics and SLAs</li><li>Usage policies and guidelines</li></ul><p><strong>Technical Specifications</strong></p><ul><li>Data formats and structures</li><li>Update frequencies</li><li>Access patterns</li><li>Performance expectations</li></ul><p><strong>Governance Requirements</strong></p><ul><li>Security controls</li><li>Privacy requirements</li><li>Compliance considerations</li><li>Usage restrictions</li></ul><h3>Implementation Best Practices</h3><p>To successfully implement this architecture:</p><p><strong>Start with Governance</strong> Establish a data governance team that includes business owners, compliance experts, and technical leaders. This team should:</p><ul><li>Define certification processes</li><li>Establish quality standards</li><li>Monitor compliance</li><li>Guide data contract development</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*s0uCX3-6Kou65KpRpnWC9w.png" /></figure><p><strong>Leverage Automation</strong> Utilize Databricks’ capabilities for automated:</p><ul><li>Quality monitoring</li><li>Pipeline testing</li><li>Documentation generation</li><li>Access control management</li></ul><p><strong>Focus on User Experience</strong> Create a seamless experience for data consumers through:</p><ul><li>Clear documentation</li><li>Example notebooks</li><li>Self-service access</li><li>Support channels</li></ul><h3>Looking Forward</h3><p>As organizations continue their data modernization journeys, the role of well-designed data products becomes increasingly critical. Our reference architecture provides a framework that combines Databricks’ powerful capabilities with organizational best practices to ensure that data products are not just technically sound, but truly valuable to the business.</p><p>The architecture is designed to evolve with your organization’s needs and capabilities. Regular assessment and adjustment ensure it continues to meet both current requirements and future aspirations. By focusing on quality, trust, and usability from the start, organizations can build a foundation that supports not just current analytics needs, but future AI and machine learning initiatives as well.</p><p>Remember that successful data modernization is a journey that requires commitment to both technical excellence and organizational transformation. By treating data as a product and implementing proper governance and quality controls, organizations can maximize the value of their data assets and accelerate their path to becoming truly data-driven enterprises.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a30645411cc1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dataproducts Development uisng Databricks , Asset Bundles , Dataproductmanager catalog]]></title>
            <link>https://medium.com/@badrish.davay/dataproducts-development-uisng-databricks-asset-bundles-dataproductmanager-catalog-55927ae47473?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/55927ae47473</guid>
            <category><![CDATA[databricks-asset-bundles]]></category>
            <category><![CDATA[data-product-manager]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-product-platform]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 20:40:48 GMT</pubDate>
            <atom:updated>2025-01-27T20:40:48.961Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dataproducts Development uisng Databricks , Asset Bundles , Dataproductmanager catalog</h3><p>Today’s data engineering shifted from building monolithic data pipeline structures to modular <em>data products</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ow-UEj5uJB03I8lt1NrJ1A.png" /></figure><p>A <a href="https://www.datamesh-architecture.com/#data-product">data product</a> is the deliverable that contains everything around a business concept to fulfill a data consumer’s need:</p><ul><li>tables to actually store data</li><li>code that transform data</li><li>tests to verify and monitor that data is correct</li><li>output ports to make data accessable</li><li>input ports to ingest data from source systems or access other data products</li><li>data contracts to describe the API</li><li>documentation</li><li>meta information, such as ownership</li></ul><p>A data product is usually managed in one Git repository.</p><p>Databricks is one of the most popular modern data platforms, now how can we engineer a professional data product with Databricks?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e8Xxaqigc-RKmmTg.png" /></figure><p>In this article, we will use Data Contracts and the new Databricks Asset Bundles that are a great fit to implement data products. All source code of this example project is available on <a href="https://github.com/datamesh-architecture/databricks-dataproduct-example">GitHub</a>.</p><h3>Define the Data Contract</h3><p>Before we start implementing, let’s discuss and define the business requirements. What does our data consumer need from us, what is their use case, what do they expect as a data model. And we need to make sure, that we understand and share the same semantics, quality expectations, and expected service levels.</p><blockquote><em>We call this approach </em>contract-first<em>. We start designing the interface of the provided data model and its metadata as a data contract. We use the data contract to drive the implementation.</em></blockquote><p>In our example, the COO of an e-commerce company wants to know if there is an issue with articles that are not sold for a longer period, i.e., articles with no sale during the last three months, the so-called <em>shelf warmers</em>.</p><p>In collaboration with the data consumer, we define a data contract as YAML, using the <a href="https://datacontract.com/">Data Contract Specification</a>:</p><pre>dataContractSpecification: 0.9.3<br>id: urn:datacontract:fulfillment:stock-last-sales<br>info:<br>  title: Last Sales<br>  version: 1.0.0<br>  description: |<br>    The data model contains all articles that are in stock.<br>    For every article the last sale timestamp is defined.<br>  owner: Fulfillment<br>  contact:<br>    name: John Doe (Data Product Owner)<br>    url: https://teams.microsoft.com/l/channel/19%3Ad7X0bzOrUrZ-QAGu0nTxlWACe5HOQ-8Joql71A_00000%40thread.tacv2/General?groupId=4d213734-d5a1-4130-8024-00000000&amp;tenantId=b000e7de-5c4d-41a2-9e67-00000000<br>servers:<br>  development:<br>    type: databricks<br>    host: dbc-abcdefgh-1234.cloud.databricks.com<br>    catalog: acme<br>    schema: stock-last-sales<br>terms:<br>  usage: &gt;<br>    Data can be used for reports, analytics and machine learning use cases.<br>    Order may be linked and joined by other tables<br>  limitations: &gt;<br>    Not suitable for real-time use cases.<br>  billing: free<br>  noticePeriod: P3M<br>models:<br>  articles:<br>    description: One record per article that is currently in stock<br>    type: table<br>    fields:<br>      sku:<br>        description: The article number (stock keeping unit)<br>        type: string<br>        primary: true<br>        pattern: ^[A-Za-z0-9]{8,14}$<br>        minLength: 8<br>        maxLength: 14<br>        example: &quot;96385074&quot;<br>      quantity:<br>        description: The total amount of articles that are currently in stock in all warehouses.<br>        type: long<br>        minimum: 1<br>        required: true<br>      last_sale_timestamp:<br>        description: The business timestamp in UTC when there was the last sale for this article. Null means that the article was never sold.<br>        type: timestamp<br>      processing_timestamp:<br>        description: The technical timestamp in UTC when this row was updated<br>        type: timestamp<br>        required: true<br>servicelevels:<br>  availability:<br>    percentage: 99.9%<br>  retention:<br>    period: 1 year<br>  freshness:<br>    threshold: 25 hours<br>    timestampField: articles.processing_timestamp<br>  frequency:<br>    description: Data is updated once a day<br>    type: batch<br>    cron: 0 0 * * *<br>examples:<br>  - type: csv<br>    data: |<br>      sku,quantity,last_sale_timestamp,processing_timestamp<br>      1234567890123,5,2024-02-25T16:16:30.171798,2024-03-25T16:16:30.171807<br>      2345678901234,10,,2024-03-25T15:16:30.171811<br>      3456789012345,15,2024-03-02T12:16:30.171814,2024-03-25T14:16:30.171816<br>      4567890123456,20,,2024-03-25T13:16:30.171817<br>      5678901234567,25,2024-03-08T08:16:30.171819,2024-03-25T12:16:30.171821<br>      6789012345678,30,,2024-03-25T11:16:30.171823<br>      7890123456789,35,2024-03-14T04:16:30.171824,2024-03-25T10:16:30.171826<br>      8901234567890,40,,2024-03-25T09:16:30.171830<br>      9012345678901,45,2024-03-20T00:16:30.171833,2024-03-25T08:16:30.171835<br>      0123456789012,50,,2024-03-25T07:16:30.171837<br>quality:<br>  type: SodaCL<br>  specification:<br>    checks for articles:<br>      - row_count &gt; 1000</pre><p><em>datacontract.yaml</em></p><p>The dataset will contain all articles that currently are in stock and it includes the last_sale_timestamp, the attribute that is most relevant for the COO. The COO can easily filter in their BI tool (such as PowerBI, redash, ...) for articles with last_sale_timestamp older than three months. Terms and service Level attributes make it clear that the dataset is update daily at midnight.</p><h3>Create the Databricks Asset Bundle</h3><p>Now it is time to develop a data product that <em>implements</em> this data contract. Databricks recently added the concept of <a href="https://docs.databricks.com/en/dev-tools/bundles/index.html"><em>Databricks Asset Bundles</em></a> that are a great fit to structure and develop data products. As time of writing in March 2024, they are in Public Preview, meaning ready for production-use.</p><p>Databricks Asset Bundles include all the infrastructure and code files to actually deploy data transformations to Databricks:</p><ul><li>Infrastructure resources</li><li>Workspace configuration</li><li>Source files, such as notebooks and Python scripts</li><li>Unit tests</li></ul><p>The Databricks CLI bundles these assets and deploys them to Databricks Platform, internally it uses Terraform. Asset Bundles are well-integrated into the Databricks Platform, e.g., it is not possible to edit code or jobs directly in Databricks, which enables a strict version control of all code and pipeline configuration.</p><p>Bundles are extremely useful, when you have multiple environments, such as dev, staging, and production. You can deploy the same bundle to multiple targets with different configurations.</p><p>To create a bundle, let’s init in a new bundle:</p><pre>databricks bundle init</pre><p>We use this configuration:</p><ul><li>Template to use: <strong>default-python</strong></li><li>Unique name for this project: <strong>stock_last_sales</strong></li><li>Include a stub (sample) notebook in ‘stock_last_sales/src’: <strong>yes</strong></li><li>Include a stub (sample) Delta Live Tables pipeline in ‘stock_last_sales/src’: <strong>no</strong></li><li>Include a stub (sample) Python package in ‘stock_last_sales/src’: <strong>yes</strong></li></ul><p>When we look into the bundle structure, let’s have a quick look at the most relevant files:</p><ul><li><strong>databricks.yml</strong> The bundle configuration and deployment targets</li><li><strong>src/</strong> The folder for the transformation code</li><li><strong>tests/</strong> The folder to place unit tests</li><li><strong>resources/</strong> The job definition for the workflow definition</li></ul><blockquote><em>Note: We recommend to maintain an internal bundle as template that incorporates the company’s naming conventions, global policies, best practices, and integrations.</em></blockquote><p>With asset bundles, we can write our code locally in our preferred IDE, such as VS Code (using the Databricks extension for Visual Studio Code), PyCharm, or IntelliJ IDEA (using Databricks Connect).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u19QQHoK_3p1fZCe.png" /></figure><p>To set up a local Python environment, we can use venv and install the development dependencies:</p><pre>python3.10 -m venv .venv<br>source .venv/bin/activate<br>pip install -r requirements-dev.txt</pre><h3>Generate Unity Catalog Table</h3><p>How do we organize the data for our data product? In this example, we use Unity Catalog to manage storage as managed tables. On an isolation level, we decide that one data product should represent one <em>schema</em> in Unity Catalog.</p><p>We can leverage the data contract YAML to generate infrastructure code:</p><p>The <em>model</em> defines the table structure of the target data model. With the <a href="https://cli.datacontract.com/">Data Contract CLI</a> tool, we can generate the SQL DDL code for the CREATE TABLE statement.</p><pre>datacontract export --format sql datacontract.yaml</pre><pre>-- Data Contract: urn:datacontract:fulfillment:stock-last-sales<br>-- SQL Dialect: databricks<br>CREATE OR REPLACE TABLE acme.stock_last_sales.articles (<br>  sku STRING primary key,<br>  quantity BIGINT not null,<br>  last_sale_timestamp TIMESTAMP,<br>  processing_timestamp TIMESTAMP not null<br>);</pre><p>The <a href="https://cli.datacontract.com/">Data Contract CLI</a> tool is also available as a Python Library datacontract-cli. So let&#39;s add it to the requirements-dev.txt and use it in directly in a Databricks notebook to actually create the table in Unity Catalog:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jMYmO2CFp-YByAEq.png" /></figure><p>The Unity Catalog tables is a managed tables that internally uses the Delta format for efficient storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AabNGFm2TfxMVf_N.png" /></figure><h3>Develop Transformation Code</h3><p>Now, let’s write the core transformation logic. With Python-based Databricks Asset Bundles, we can develop our data pipelines as:</p><ul><li>Databricks Notebooks,</li><li>Delta Live Tables, or</li><li>Python files</li></ul><p>In this data product, we’ll write plain Python files for our core transformation logic that will be deployed as Wheel packages.</p><p>Our transformation takes all available stocks that we get from an input port, such as the operational system that manages the current stock data, and left-joins the dataframe with the latest sales timestamp for every sku. The sales information are also an input port, e.g., another upstream data product provided by the checkout team. We store the resulting dataframe in the previously generated table structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2CBt7MSTCmCuk5zw.png" /></figure><p>With that option, the code remains reusable, easy to test with unit tests, and we can run it on our local machines. As professional data engineers, we make sure that the calculate_last_sales() function works as expected by writing good unit tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lJQHFKq2EqOXDhbX.png" /></figure><p>We update the job configuration to run the Python code as a python_wheel_task and configure the scheduler and the appropriate compute cluster.</p><pre># The main job for stock_last_sales.<br>resources:<br>  jobs:<br>    stock_last_sales_job:<br>      name: stock_last_sales_job</pre><pre>      schedule:<br>        # Run every day at midnight<br>        quartz_cron_expression: &#39;0 0 0 * * ?&#39;<br>        timezone_id: Europe/Amsterdam</pre><pre>      tasks:<br>        - task_key: create_unity_catalog_table<br>          job_cluster_key: job_cluster<br>          notebook_task:<br>            notebook_path: ../src/create_unity_catalog_table.ipynb<br>          libraries:<br>            - pypi:<br>                package: datacontract-cli</pre><pre>        - task_key: main_task<br>          depends_on:<br>            - task_key: create_unity_catalog_table<br>          job_cluster_key: job_cluster<br>          python_wheel_task:<br>            package_name: stock_last_sales<br>            entry_point: main<br>          libraries:<br>            - whl: ../dist/*.whl</pre><pre>      job_clusters:<br>        - job_cluster_key: job_cluster<br>          new_cluster:<br>            spark_version: 13.3.x-scala2.12<br>            node_type_id: i3.xlarge<br>            autoscale:<br>                min_workers: 1<br>                max_workers: 4</pre><p><em>resources/stock_last_sales_job.yml</em></p><p>When we are confident, we can deploy the bundle to our Databricks dev instances (manually for now):</p><pre>databricks bundle deploy</pre><p>And let’s trigger a manual run of our workflow:</p><pre>databricks bundle run stock_last_sales_job</pre><p>In Databricks, we can see that the workflow run was successful:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hh2LefRV_b6MtTTB.png" /></figure><p>And we have data in our table that we created earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8XnmLyCvYg1O5UXs.png" /></figure><h3>Test the Data Contract</h3><p>We are not quite finished with our task. How do we know, that the data is correct? While we have unit tests that give us confidence on the transformation code, we also need an acceptance test to verify, that we implemented the agreed data contract correctly.</p><p>For that, we can use the <a href="https://cli.datacontract.com/">Data Contract CLI</a> tool to make this check:</p><pre>export DATACONTRACT_DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/b053xxxxxxx<br>export DATACONTRACT_DATABRICKS_TOKEN=dapia1926f7c64b7595880909xxxxxxxxxx<br>datacontract test datacontract.yaml</pre><p>The datacontract tool takes all the schema and format information from the <em>model</em>, the quality attributes, and the metadata, and compares them with the actual dataset. It reads the connection details from the <em>servers</em> section and connects to Databricks executes all the checks and gives a comprehensive overview.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cf7wCqjTlv8Y8Qme.png" /></figure><p>We want to execute this test with every pipeline run, so once again, let’s make a Notebook task for the test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x49TYiDLxUA1P-4u.png" /></figure><h3>Deploy with CI/CD</h3><p>To automatically test, deploy the Asset Bundle to Databricks, and finally run the job once, we set up a CI/CD pipeline in GitHub, using a <a href="https://docs.databricks.com/en/dev-tools/bundles/ci-cd.html">GitHub Action</a>.</p><pre>name: &quot;Deploy Databricks Assets Bundle&quot;<br>on:<br>  workflow_dispatch:<br>  push:<br>    branches: [ &quot;main&quot; ]</pre><pre>jobs:<br>  test:<br>    name: &quot;Run unit tests&quot;<br>    runs-on: ubuntu-latest<br>    steps:<br>      - uses: actions/checkout@v4<br>      - name: Set up Python ${{matrix.python-version}}<br>        uses: actions/setup-python@v5<br>        with:<br>          python-version: &quot;3.10&quot;<br>      - name: Install dependencies<br>        run: |<br>          python -m pip install --upgrade pip<br>          pip install -r requirements-dev.txt<br>      - name: Test with pytest<br>        run: pytest<br>        env:<br>          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}<br>          DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}<br>          DATABRICKS_CLUSTER_ID: ${{ secrets.DATABRICKS_CLUSTER_ID }}</pre><pre>  deploy:<br>    name: &quot;Deploy bundle to DEV&quot;<br>    runs-on: ubuntu-latest</pre><pre>    needs:<br>      - test</pre><pre>    steps:<br>      - uses: actions/checkout@v4<br>      - uses: databricks/setup-cli@main<br>      - run: databricks bundle deploy<br>        working-directory: .<br>        env:<br>          DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}<br>          DATABRICKS_BUNDLE_ENV: dev<br>  run_pipieline:<br>    name: &quot;Run pipeline&quot;<br>    runs-on: ubuntu-latest</pre><pre>    needs:<br>      - deploy</pre><pre>    steps:<br>      - uses: actions/checkout@v4<br>      - uses: databricks/setup-cli@main<br>      - run: databricks bundle run stock_last_sales_job --refresh-all<br>        working-directory: .<br>        env:<br>          DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}<br>          DATABRICKS_BUNDLE_ENV: dev</pre><p><em>.github/workflows/deploy.yml</em></p><p>Now, every time we update the code, the Asset Bundle is automatically deployed to Databricks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uWhINt52AB-lLlF0.png" /></figure><h3>Publish Metadata</h3><p>For others to find, understand, and trust data products, we want to register them in a data product registry.</p><p>In this example, we use <a href="https://www.datamesh-manager.com/?ref=dma-databricks-dataproduct">Data Mesh Manager</a>, a platform to register, manage, and discover data products, data contracts, and global policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VZsxL22JGiIRc8WK.png" /></figure><p>Again, let’s create a notebook task (or Python code task) to publish the metadata to Data Mesh Manager and add the task to our workflow. We can use <a href="https://docs.databricks.com/en/security/secrets/index.html">Databricks Secrets</a> to make the API Key available in Databricks.</p><pre>databricks secrets create-scope datamesh_manager<br>databricks secrets put-secret datamesh_manager api_key</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SG--ygP1KP8OpYJL.png" /></figure><h3>Summary</h3><p>Now, the COO can connect to this table with a BI tool (such as PowerBI, Tableau, Redash, or withing Databricks) to answer their business question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1GvRjGoYVi4x-Eq3.png" /></figure><p>Databricks Asset Bundles are a great fit to develop professional data products on Databricks, as it bundles all the resources and configurations (code, tests, storage, compute, scheduler, metadata, …) that are needed to provide high-quality datasets to data consumers.</p><p>It is easy to integration Data Contracts for defining the requirements and the <a href="https://cli.datacontract.com/">Data Contract CLI</a> to automate acceptance tests.</p><p>Find the source code for the example project on <a href="https://github.com/datamesh-architecture/databricks-dataproduct-example">GitHub</a>.</p><h3>Learn More</h3><ul><li><a href="https://github.com/datamesh-architecture/databricks-dataproduct-example">Source Code for the examples project</a></li><li><a href="https://docs.databricks.com/en/dev-tools/bundles/index.html">Official databricks documentation on Databricks Asset Bundles</a></li><li><a href="https://www.datamesh-architecture.com/tech-stacks/databricks">Databricks Tech Stack</a></li><li><a href="https://datacontract.com/">Data Contract Specification</a></li><li><a href="https://cli.datacontract.com/">Data Contract CLI</a></li><li><a href="https://www.datamesh-manager.com/">Data Mesh Manager</a></li></ul><p><strong>References</strong>:</p><ul><li><a href="https://www.datamesh-architecture.com/howto/build-a-dataproduct-with-databricks">Build a Data Product with Databricks</a></li><li><a href="https://www.datamesh-architecture.com/#data-product">Data Mesh Architecture</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55927ae47473" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Medallion Architecture in Databricks]]></title>
            <link>https://medium.com/@badrish.davay/medallion-architecture-in-databricks-e56be1b7516b?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e56be1b7516b</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 16:59:10 GMT</pubDate>
            <atom:updated>2025-01-27T17:12:42.653Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing Medallion Architecture in Databricks: A Comprehensive Guide</h3><p>In the world of modern data engineering, organizing and processing large-scale data efficiently has become increasingly crucial. Enter the Medallion Architecture, a powerful design pattern that has gained significant traction, especially within the Databricks ecosystem. This blog post will dive deep into what Medallion Architecture is, why you should consider it, and how to implement it effectively in Databricks.</p><h3>What is Medallion Architecture?</h3><p>Medallion Architecture is a data design pattern that organizes data into three distinct layers, each representing a different level of data refinement. These layers are often referred to as Bronze, Silver, and Gold — hence the “medallion” naming convention. Each layer serves a specific purpose and adds value to your data processing pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EkWY8nEW17SgbewI.jpeg" /></figure><h3>Bronze Layer: The Foundation</h3><p>The Bronze layer represents your raw data in its most primitive form. Think of it as a digital landing zone where data arrives from various sources. Key characteristics include:</p><p>• Raw data ingestion from source systems</p><p>• Minimal to no transformations</p><p>• Append-only tables with full history</p><p>• Schema inference and initial metadata capture</p><p>• Support for both batch and streaming ingestion</p><blockquote># Example Bronze layer table creation in Databricks<br> spark.sql(“””<br> CREATE TABLE IF NOT EXISTS bronze.sales_raw (<br> raw_data STRING,<br> source_system STRING,<br> ingestion_timestamp TIMESTAMP,<br> _metadata STRUCT&lt;filename: STRING&gt;<br> )<br> USING DELTA<br> PARTITIONED BY (source_system)<br> “””)<br> <br> # Example streaming ingestion<br> df = spark.readStream<br> .format(“cloudFiles”)<br> .option(“cloudFiles.format”, “json”)<br> .load(“/path/to/landing/zone”)<br> <br> df.writeStream<br> .format(“delta”)<br> .option(“checkpointLocation”, “/checkpoints/bronze_sales”)<br> .table(“bronze.sales_raw”)</blockquote><h3>Silver Layer: The Refined State</h3><p>The Silver layer is where we begin to add value through data quality rules, cleansing, and standardization. This layer focuses on:</p><p>• Data validation and quality checks</p><p>• Schema enforcement and evolution</p><p>• Business logic application</p><p>• Data enrichment and standardization</p><p>• Error handling and rejection tracking</p><blockquote># Example Silver layer transformation<br> def process_to_silver():<br> spark.sql(“””<br> MERGE INTO silver.sales_cleaned target<br> USING (<br> SELECT <br> from_json(raw_data, ‘struct&lt;sale_id:string, amount:decimal(10,2)&gt;’) as parsed_data,<br> source_system,<br> ingestion_timestamp<br> FROM bronze.sales_raw<br> WHERE _metadata.processed = false<br> ) source<br> ON target.sale_id = source.parsed_data.sale_id<br> WHEN MATCHED THEN UPDATE SET *<br> WHEN NOT MATCHED THEN INSERT *<br> “””)</blockquote><h3>Gold Layer: Business-Ready Data</h3><p>The Gold layer represents your data in its most refined form, optimized for specific business use cases:</p><p>• Aggregated tables for reporting</p><p>• Feature tables for machine learning</p><p>• Business-specific dimensional models</p><p>• Optimized query performance</p><p>• Self-service analytics support</p><blockquote># Example Gold layer aggregation<br> spark.sql(“””<br> CREATE OR REPLACE VIEW gold.daily_sales_metrics AS<br> SELECT <br> date_trunc(‘day’, transaction_timestamp) as sale_date,<br> region,<br> product_category,<br> COUNT(*) as total_transactions,<br> SUM(amount) as total_revenue,<br> AVG(amount) as avg_transaction_value<br> FROM silver.sales_cleaned<br> GROUP BY 1, 2, 3<br> “””)</blockquote><h3>Best Practices for Implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*Y1MPtHEko4Sb6s985C7WbA.png" /></figure><h3>1. Data Quality Management</h3><p>Implement comprehensive data quality checks at each layer:</p><blockquote>def validate_silver_data(df):<br> return df.withColumn(<br> “quality_check_passed”,<br> (col(“amount”) &gt; 0) &amp;<br> (col(“sale_id”).isNotNull()) &amp;<br> (col(“transaction_timestamp”).isNotNull())<br> )</blockquote><h3>2. Performance Optimization</h3><p>Optimize your Delta tables for better query performance:</p><blockquote># Example optimization commands<br> spark.sql(“””<br> OPTIMIZE gold.sales_metrics<br> ZORDER BY (sale_date, region)<br> “””)<br> <br> # Vacuum old files while retaining 7 days of history<br> spark.sql(“VACUUM gold.sales_metrics RETAIN 168 HOURS”)</blockquote><h3>3. Metadata Management</h3><p>Maintain comprehensive metadata for data lineage and governance:</p><blockquote>def track_lineage(input_table, output_table, transformation_type):<br> spark.sql(f”””<br> INSERT INTO metadata.lineage_tracking<br> VALUES (<br> ‘{input_table}’,<br> ‘{output_table}’,<br> ‘{transformation_type}’,<br> current_timestamp()<br> )<br> “””)</blockquote><h3>Advanced Features and Integration</h3><h3>Delta Lake Integration</h3><p>Databricks’ Delta Lake provides ACID transactions and versioning capabilities crucial for Medallion Architecture:</p><blockquote># Time travel capabilities<br> spark.sql(“””<br> SELECT * FROM silver.sales_cleaned VERSION AS OF 123<br> UNION ALL<br> SELECT * FROM silver.sales_cleaned@v2.0<br> “””)</blockquote><h3>Workflow Orchestration</h3><p>Use Databricks Workflows to orchestrate your medallion pipeline:</p><blockquote># Example workflow configuration<br> {<br> “name”: “medallion_pipeline”,<br> “tasks”: [<br> {<br> “task_key”: “ingest_to_bronze”,<br> “notebook_task”: {<br> “notebook_path”: “/path/to/bronze_ingestion”<br> }<br> },<br> {<br> “task_key”: “process_to_silver”,<br> “notebook_task”: {<br> “notebook_path”: “/path/to/silver_processing”<br> },<br> “depends_on”: [<br> {“task_key”: “ingest_to_bronze”}<br> ]<br> }<br> ]<br> }</blockquote><h3>Monitoring and Maintenance</h3><p>Implement comprehensive monitoring for your medallion pipeline:</p><blockquote>def monitor_pipeline_health():<br> metrics = spark.sql(“””<br> SELECT<br> layer,<br> COUNT(*) as record_count,<br> COUNT(CASE WHEN quality_check_passed THEN 1 END) as valid_records,<br> MAX(processing_timestamp) as last_update<br> FROM monitoring.pipeline_metrics<br> GROUP BY layer<br> “””)<br> return metrics</blockquote><h3>Medallion Architecture Disadvantages</h3><ul><li>Does not replace dimensional modeling techniques: schemas and tables within each layer must still be modeled. Medallion architecture provides a framework for data cleaning, not data architecture.</li><li>Uses large amounts of storage: though, as many have proclaimed, “storage is cheap,” a medallion architecture effectively triples the amount of storage used in a data lake. For that reason, it might not be practical for data teams with intensive storage demands.</li><li>Often requires additional downstream processing: If data engineers are maintaining the medallion architecture, there needs to be a place for analysts/analytics engineers to build business-focused transformations that power BI. Some teams might prefer those processes remain separate, rather than having analysts develop in the gold layer. As such, a medallion architecture is not a drop-in replacement for existing data transformation solutions.</li><li>Implies a data lakehouse architecture: The medallion architecture is built on the premise of a data lakehouse. If a lakehouse is impractical for your team, this architecture might not make sense. Nonetheless, medallion architectures <em>can</em> be used effectively in hybrid data lake/warehouse implementations.</li></ul><h3>Conclusion</h3><p>Medallion Architecture in Databricks provides a robust framework for building scalable, maintainable data pipelines. By following these patterns and best practices, you can create a data architecture that is both powerful and flexible, capable of handling modern data engineering challenges while providing clean, reliable data for your business needs.</p><p>Remember that this architecture is not a one-size-fits-all solution — adapt these patterns to your specific use cases and requirements. The key is to maintain the core principles of data quality, reproducibility, and scalability throughout your implementation.</p><h3>Additional Resources</h3><p>• Databricks Documentation</p><p>• Delta Lake Documentation</p><p>• Unity Catalog Integration Guide</p><p>• Databricks Workflow API Reference</p><p><em>This blog post is part of our Data Engineering Best Practices series. For more insights on modern data architecture, follow our blog and join our community of data practitioners.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e56be1b7516b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Testing in Two Worlds: Traditional TDD vs. LLM Application Testing with promptfoo ]]></title>
            <link>https://medium.com/@badrish.davay/testing-in-two-worlds-traditional-tdd-vs-llm-application-testing-with-promptfoo-793d9a244d62?source=rss-6f53b904fff2------2</link>
            <guid isPermaLink="false">https://medium.com/p/793d9a244d62</guid>
            <dc:creator><![CDATA[Badrish Davay]]></dc:creator>
            <pubDate>Wed, 22 Jan 2025 19:31:41 GMT</pubDate>
            <atom:updated>2025-01-22T20:18:24.271Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*QC7DWmTHPxu4SfFt.jpg" /></figure><p>In the ever-evolving landscape of software development, testing methodologies have had to adapt to new paradigms. While Test-Driven Development (TDD) has been a cornerstone of traditional software engineering for decades, the emergence of Large Language Models (LLMs) has introduced new challenges and approaches to testing. Let’s explore how testing differs between these two domains, with a particular focus on using promptfoo for LLM testing.</p><h3>Traditional Test-Driven Development</h3><h3>The Classic TDD Cycle</h3><p>Traditional TDD follows a well-established “Red-Green-Refactor” cycle:</p><ol><li>Write a failing test (Red)</li><li>Write minimal code to make the test pass (Green)</li><li>Refactor the code while maintaining passing tests</li></ol><p>This approach has several key characteristics:</p><ul><li><strong>Deterministic Results</strong>: Traditional software functions produce the same output given the same input</li><li><strong>Binary Pass/Fail</strong>: Tests either pass or fail based on exact matches or precise conditions</li><li><strong>Clear Requirements</strong>: Test cases are derived from well-defined specifications</li><li><strong>Immediate Feedback</strong>: Tests run quickly and provide instant results</li><li><strong>Isolation</strong>: Units can be tested independently of each other</li></ul><h3>Benefits of Traditional TDD</h3><p>The traditional approach offers several advantages:</p><ul><li>Ensures code correctness and maintainability</li><li>Provides documentation through test cases</li><li>Catches regressions early</li><li>Promotes modular design</li><li>Builds confidence in code changes</li></ul><h3>Testing LLM Applications with promptfoo</h3><h3>A New Testing Paradigm</h3><p>Testing LLM applications requires a fundamentally different approach due to their probabilistic nature. promptfoo has emerged as a valuable tool for this purpose, offering unique capabilities for testing across different models.</p><h3>Key Differences from Traditional TDD</h3><h4>1. Output Variability</h4><p>Unlike traditional software, LLMs may produce different outputs for the same input:</p><ul><li>Traditional TDD: sum(2, 2) always returns 4</li><li>LLM Response: “What’s 2+2?” might yield various phrasings of “4” or even explanations</li></ul><h4>2. Test Success Criteria</h4><p>Success metrics become more nuanced:</p><ul><li>Traditional TDD: Exact matches or precise conditions</li><li>LLM Testing:</li><li>Semantic similarity scoring</li><li>Token overlap assessment</li><li>Sentiment analysis</li><li>Format compliance</li><li>Content safety evaluation</li></ul><h4>3. Testing Methodology</h4><p>promptfoo introduces new testing patterns:</p><pre>javascriptCopy// Example promptfoo test configuration<br>{<br>  &quot;tests&quot;: [<br>    {<br>      &quot;description&quot;: &quot;Basic arithmetic test&quot;,<br>      &quot;prompt&quot;: &quot;What is 2+2?&quot;,<br>      &quot;assert&quot;: {<br>        &quot;type&quot;: &quot;contains&quot;,<br>        &quot;value&quot;: &quot;4&quot;<br>      }<br>    }<br>  ]<br>}</pre><h3>Key Features of promptfoo:</h3><h3>Core Testing Capabilities</h3><h3>1. Multi-Model Testing</h3><ul><li>Simultaneous testing across different LLM providers:</li><li>OpenAI models (GPT-3.5, GPT-4)</li><li>Anthropic models (Claude series)</li><li>Open-source models (Llama, Mistral)</li><li>Local models via API</li><li>Consistent comparison framework for all models</li><li>Unified output format for easy analysis</li><li>A python code using RAG</li></ul><h3>2. Assertion Types</h3><h4>Basic Assertions</h4><pre>tests:<br>  - description: &quot;Basic equality test&quot;<br>    assert:<br>      - type: equals<br>        value: &quot;Expected exact response&quot;<br>      - type: contains<br>        value: &quot;Must include this phrase&quot;<br>      - type: icontains<br>        value: &quot;Case-insensitive contains&quot;</pre><h4>Advanced Assertions</h4><pre>tests:<br>  - description: &quot;Complex validation&quot;<br>    assert:<br>      - type: javascript<br>        value: &quot;response.length &lt; 100 &amp;&amp; response.includes(&#39;key phrase&#39;)&quot;<br>      - type: webhook<br>        value: &quot;https://api.example.com/validate&quot;<br>      - type: regex<br>        value: &quot;^[A-Z][a-z]+$&quot;</pre><h3>3. Prompt Management</h3><ul><li>Version control for prompts</li><li>Prompt templating support</li><li>Variable substitution</li><li>Context window management</li><li>Prompt chaining capabilities</li></ul><h3>Configuration Features</h3><h3>1. Provider Configuration</h3><pre>providers:<br>  - openai:gpt-4<br>  - anthropic:claude-2<br>  - local:llama2<br>  <br>options:<br>  maxConcurrency: 5<br>  maxRetries: 3<br>  timeout: 30000</pre><h3>2. Test Suite Organization</h3><pre>suites:<br>  basic:<br>    - test1.yaml<br>    - test2.yaml<br>  advanced:<br>    - complex_tests/*.yaml</pre><h3>3. Environment Variables</h3><pre>PROMPTFOO_OPENAI_API_KEY=sk-...<br>PROMPTFOO_ANTHROPIC_API_KEY=sk-...</pre><h3>Analysis Tools</h3><h3>1. Evaluation Metrics</h3><ul><li>Response latency measurement</li><li>Token usage tracking</li><li>Cost analysis per request</li><li>Success rate calculation</li><li>Consistency scoring</li></ul><h3>2. Output Analysis</h3><pre>analysis:<br>  similarity:<br>    threshold: 0.85<br>    algorithm: &quot;cosine&quot;<br>  safety:<br>    enabled: true<br>    threshold: &quot;high&quot;<br>  formatting:<br>    json_schema: &quot;./schema.json&quot;</pre><h3>3. Reporting Features</h3><ul><li>HTML report generation</li><li>CSV export capabilities</li><li>JSON output format</li><li>Command-line summary</li><li>Detailed error logging</li></ul><p>Below is an example that I ran on promptfoo and compared the results from multiple LLM models even one custom python code uisng RAG. Below image shows the results published after the tests are run for each prompt against all the models speficied in my config file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dDU8P_Y01xwSC61cLs1MsQ.png" /></figure><h3>Advanced Features</h3><h3>1. Custom Evaluators</h3><pre>module.exports = {<br>  evaluate: async (response, expected) =&gt; {<br>    // Custom evaluation logic<br>    return {<br>      pass: true/false,<br>      score: 0.95,<br>      reason: &quot;Explanation&quot;<br>    };<br>  }<br>};</pre><h3>2. Grading and Scoring</h3><ul><li>Customizable scoring systems</li><li>Weighted test cases</li><li>Aggregate performance metrics</li><li>Historical performance tracking</li></ul><h3>3. Integration Capabilities</h3><ul><li>CI/CD pipeline integration</li><li>GitHub Actions support</li><li>API endpoint testing</li><li>Webhook integration</li><li>Custom output formatters</li></ul><h3>Debugging Tools</h3><h3>1. Verbose Mode</h3><pre>promptfoo test --verbose</pre><ul><li>Detailed execution logs</li><li>Request/response timing</li><li>Token usage breakdown</li><li>Error stack traces</li></ul><h3>2. Interactive Mode</h3><ul><li>Real-time test execution</li><li>Live result viewing</li><li>On-the-fly test modification</li><li>Direct model comparison</li></ul><h3>3. Snapshot Testing</h3><pre>tests:<br>  - description: &quot;Snapshot test&quot;<br>    prompt: &quot;Complex prompt&quot;<br>    assert:<br>      - type: snapshot<br>        file: &quot;./snapshots/test1.snap&quot;</pre><h3>Safety and Compliance</h3><h3>1. Content Filtering</h3><ul><li>Profanity detection</li><li>Sensitive content alerts</li><li>Content policy compliance</li><li>Custom filter rules</li></ul><h3>2. Rate Limiting</h3><pre>rateLimit:<br>  requests: 10<br>  period: &quot;1m&quot;<br>  perModel: true</pre><h3>3. Cost Management</h3><ul><li>Budget constraints</li><li>Usage alerts</li><li>Cost optimization suggestions</li><li>Billing reports</li></ul><h3>Best Practices Implementation</h3><h3>1. Test Organization</h3><pre># project structure<br>tests/<br>  ├── common/<br>  │   ├── helpers.js<br>  │   └── assertions.js<br>  ├── suites/<br>  │   ├── basic.yaml<br>  │   └── advanced.yaml<br>  └── config.yaml</pre><h3>2. Prompt Templates</h3><pre>yamltemplates:<br>  base: |<br>    System: {{system}}<br>    User: {{query}}<br>  variables:<br>    system: &quot;You are a helpful assistant&quot;<br>    query: &quot;What is 2+2?&quot;</pre><h3>3. Error Handling</h3><pre>errorHandling:<br>  retry:<br>    maxAttempts: 3<br>    backoff: &quot;exponential&quot;<br>  fallback:<br>    enabled: true<br>    provider: &quot;local:llama2&quot;</pre><h3>Unique Selling Points</h3><p><strong>Comprehensive Testing Framework</strong></p><ul><li>All-in-one solution for LLM testing</li><li>Extensive assertion library</li><li>Flexible configuration options</li></ul><p><strong>Developer-Friendly</strong></p><ul><li>Clear documentation</li><li>Intuitive CLI</li><li>Extensible architecture</li><li>Active community support</li></ul><p><strong>Enterprise Features</strong></p><ul><li>Team collaboration support</li><li>Role-based access control</li><li>Audit logging</li><li>Compliance reporting</li></ul><h3>Future Roadmap Features</h3><p><strong>Enhanced Analytics</strong></p><ul><li>Advanced visualization tools</li><li>Machine learning-based analysis</li><li>Automated optimization suggestions</li></ul><p><strong>Extended Integration</strong></p><ul><li>More LLM provider support</li><li>Additional testing frameworks</li><li>Expanded API capabilities</li></ul><p><strong>Advanced Automation</strong></p><ul><li>Automated test generation</li><li>Self-healing tests</li><li>Continuous optimization</li></ul><p>This comprehensive feature set makes promptfoo a powerful tool for testing LLM applications, offering both basic testing capabilities and advanced features for complex testing scenarios.</p><h3>promptfoo’s Unique Features</h3><h4>Model Comparison</h4><ul><li>Ability to test the same prompt across multiple models:</li><li>GPT-4</li><li>Claude</li><li>Open source models like Llama</li><li>Fine-tuned variants</li></ul><h4>Evaluation Metrics</h4><ul><li>Response time comparison</li><li>Token usage analysis</li><li>Cost assessment</li><li>Output consistency measurement</li><li>Safety compliance checking</li></ul><h3>Practical Differences in Implementation</h3><h3>Test Writing Approach</h3><p>Traditional TDD:</p><pre>pythonopydef test_add_numbers():<br>    assert calculator.add(2, 2) == 4</pre><p>promptfoo LLM Testing:</p><pre>tests:<br>  - description: &quot;Addition explanation&quot;<br>    prompt: &quot;Explain why 2+2=4&quot;<br>    assert:<br>      - type: &quot;similarity&quot;<br>        threshold: 0.8<br>        value: &quot;The sum of two and two equals four because...&quot;<br>      - type: &quot;safe&quot;<br>      - type: &quot;maxTokens&quot;<br>        value: 100</pre><h3>Handling Edge Cases</h3><p>Traditional software testing can exhaustively cover edge cases, while LLM testing must account for:</p><ul><li>Prompt variations</li><li>Context length limitations</li><li>Model-specific behaviors</li><li>Response formatting inconsistencies</li></ul><h3>Best Practices for LLM Testing</h3><h3>1. Comprehensive Test Suites</h3><p>Create test cases that cover:</p><ul><li>Basic functionality</li><li>Edge cases</li><li>Different prompt styles</li><li>Various input lengths</li><li>Multiple languages</li><li>Safety requirements</li></ul><h3>2. Version Control</h3><ul><li>Track prompt versions</li><li>Document model versions</li><li>Keep test cases in version control</li><li>Record baseline performances</li></ul><h3>3. Continuous Testing</h3><ul><li>Integrate LLM tests into CI/CD pipelines</li><li>Monitor model drift</li><li>Track performance metrics over time</li><li>Implement automated quality checks</li></ul><h3>Looking Forward</h3><p>As LLM applications become more prevalent, testing methodologies will continue to evolve. Key areas of development include:</p><ul><li>Better metrics for semantic evaluation</li><li>Automated prompt optimization</li><li>Improved testing tools and frameworks</li><li>Standardized testing patterns</li><li>Integration with traditional testing approaches</li></ul><h3>Conclusion</h3><p>While traditional TDD and LLM testing share the common goal of ensuring software quality, their implementations differ significantly. promptfoo and similar tools are pioneering new approaches to testing that accommodate the unique characteristics of LLM applications. As the field matures, we can expect testing methodologies to become more sophisticated, incorporating elements from both traditional software testing and new LLM-specific approaches.</p><p>The key is recognizing that both methodologies have their place in modern software development, and choosing the right approach depends on the specific requirements of your application. Understanding these differences allows developers to build more robust and reliable applications, whether they’re working with traditional software or LLM-powered systems.</p><p>Tags: #Testing #AI #SoftwareEngineering #LLM #TDD #promptfoo #QualityAssurance</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793d9a244d62" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>