<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pradeep Singh on Medium]]></title>
        <description><![CDATA[Stories by Pradeep Singh on Medium]]></description>
        <link>https://medium.com/@dbxdev?source=rss-04869b77d4b2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*59-zpP6Nid9CpYrKg79fTQ.jpeg</url>
            <title>Stories by Pradeep Singh on Medium</title>
            <link>https://medium.com/@dbxdev?source=rss-04869b77d4b2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 02:19:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dbxdev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Context Failure Modes: Why Agents Break, and How to Fix It on Databricks]]></title>
            <link>https://dbxdev.medium.com/context-failure-modes-why-agents-break-and-how-to-fix-it-on-databricks-90d7d631f457?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/90d7d631f457</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[lakebase]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-11-09T12:31:01.509Z</atom:updated>
            <content:encoded><![CDATA[<h3>Objective</h3><p>If you work with agents, chances are you have faced on the following Context Failure Modes — Poisoning, Distraction, Confusion, and Clash</p><p>The blog in an attempt to demystify what they mean, when they usually occur and how you can resolve these failure modes when building agents on databricks .</p><p>Its also an important topic to understand if you are prepraring for Databricks Context Engineer Associate with some question in the end to test and build a better understanding</p><h4>Simple Explanation &amp; Real-World Example</h4><p>Imagine an air traffic controller (ATC) managing a busy airport. Their radar screen (the context window) can only display so much information at once. Now imagine four ways things go wrong:</p><ul><li>Context Poisoning = A faulty radar transponder sends the wrong altitude for Flight 302. The ATC now has <em>incorrect data</em> on their screen and routes another plane into a dangerous path. The information <em>looks</em> legitimate but is wrong. → Corrupted information enters the context.</li><li>Context Distraction = The radar screen is cluttered with 200 private Cessnas doing touch-and-goes at a nearby airfield. The ATC can’t focus on the 12 commercial jets that actually matter. The relevant data is there, but it’s buried. → Irrelevant material dilutes attention.</li><li>Context Confusion = Two flights both have callsign “Delta 47” — one arriving from Chicago, one departing to Miami. The ATC accidentally gives the Chicago arrival the Miami departure’s clearance. → Similar but distinct entities get mixed together.</li><li>Context Clash = The tower supervisor’s standing order says “hold all departures on Runway 27” but the ATIS broadcast says “Runway 27 is active for departures.” The ATC receives genuinely contradictory instructions. → Two context elements directly contradict each other.</li></ul><h4>How It Works Under the Hood</h4><p>LLMs process context through attention mechanisms that assign weights to different tokens. As the context window fills:</p><ul><li>Poisoning corrupts the <em>signal</em> — the model treats incorrect information as ground truth because it appears in the same context as authoritative sources. The model has no built-in fact-checking; it trusts what’s in the window.</li><li>Distraction dilutes the <em>attention budget</em> — with finite attention heads, irrelevant tokens compete with relevant ones. Models exhibit a “lost in the middle” phenomenon where information in the center of long contexts receives less attention.</li><li>Confusion breaks <em>entity boundaries</em> — when two customers, two contracts, or two sessions share similar attributes, the model’s attention mechanism can’t reliably separate them, leading to cross-contamination of facts.</li><li>Clash creates <em>contradictions</em> — when the system prompt says one thing and retrieved context says another, the model must resolve the conflict, often unpredictably. This is especially dangerous because the model may silently pick the wrong side.</li></ul><h4>Databricks Technical Implementation</h4><p>Databricks addresses each failure mode through specific platform capabilities:</p><p>Preventing Context Poisoning → Unity Catalog Governance</p><p>Unity Catalog provides metadata, access controls, data-quality tags, and PII handling for all retrieval sources. By constraining an agent’s retrieval space to authoritative sources (certified tables, governed volumes), you prevent stale or untrusted data from entering context.</p><p>Example: Tag authoritative sources in Unity Catalog .Only tables tagged as ‘authoritative’ are available to the agent</p><pre>ALTER TABLE catalog.schema.policy_documents<br>SET TAGS (&#39;data_quality&#39; = &#39;authoritative&#39;, &#39;refresh_cadence&#39; = &#39;daily&#39;);</pre><p>In agent configuration, filter retrieval to authoritative sources only .This prevents derived/stale tables from poisoning context</p><p>Preventing Context Distraction → AI Search + Tool Result Scoping</p><p>Databricks AI Search with proper chunking and just-in-time retrieval ensures only relevant context enters the window. Tool result scoping trims large tool outputs before they enter the message history.</p><p>Example: Scope tool results to reduce distraction .Instead of dumping entire query results into context:</p><pre>def scoped_search_tool(query: str, max_results: int = 3) -&gt; str:<br>    &quot;&quot;&quot;Retrieve only the top-k most relevant results,<br>    returning summaries rather than full documents.&quot;&quot;&quot;<br>    results = ai_search_index.similarity_search(<br>        query=query,<br>        num_results=max_results,<br>        columns=[&quot;title&quot;, &quot;summary&quot;, &quot;relevance_score&quot;]  # Not full_text!<br>    )<br>    return format_concise_results(results)</pre><p>Preventing Context Confusion → Lakebase Session Isolation</p><p>Lakebase provides durable, structured memory with clear session boundaries. By separating current-session state from cross-session memory, you prevent the model from mixing facts across customers, contracts, or conversations.</p><p>Example: LangGraph checkpointer with Lakebase for session isolation</p><pre>from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver<br>async with AsyncPostgresSaver.from_conn_string(lakebase_conn) as checkpointer:<br>    await checkpointer.setup()<br>    # Each thread_id isolates one conversation&#39;s state<br>    graph = workflow.compile(checkpointer=checkpointer)<br>    config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;customer_session_abc123&quot;}}<br>    result = await graph.ainvoke(input_state, config)</pre><p>Preventing Context Clash → Clear Instruction Hierarchy + MLflow Evaluation</p><p>Design system prompts with explicit priority rules. Use MLflow 3 to trace and evaluate whether contradictions are occurring.</p><pre>System instructions override retrieved context.<br>Retrieved regulatory text overrides analyst commentary.<br>When sources conflict, state the conflict explicitly instead of choosing silently.</pre><p>MLflow 3 trace inspection for clash detection</p><pre>import mlflow<br># Log agent runs and inspect traces for contradictory outputs<br>with mlflow.start_run():<br>    mlflow.log_param(&quot;system_prompt_version&quot;, &quot;v2.3&quot;)<br>    mlflow.log_param(&quot;retrieval_source&quot;, &quot;uc_governed_only&quot;)</pre><h4>Client Scenario &amp; Trade-offs</h4><p>A major bank deploys a Databricks agent to help compliance officers review loan documents against current regulations. The agent retrieves regulatory text from a Unity Catalog volume and answers questions about whether specific loan terms comply.</p><p>The Problem: After three months in production, compliance officers report the agent is citing regulations that were amended weeks ago (poisoning), burying relevant clauses in pages of retrieved text (distraction), mixing up terms from commercial vs. residential loan regulations (confusion), and occasionally contradicting itself when the system prompt says “use only retrieved clauses” but retrieved context includes commentary from a prior analyst session (clash).</p><p>Recommendations</p><ul><li>For Poisoning: Switch the AI Search index from TRIGGERED to CONTINUOUS pipeline sync, or schedule triggered refreshes to run immediately after the regulatory ingestion pipeline completes. Tag all regulatory sources with freshness metadata in Unity Catalog. <em>Trade-off:</em> CONTINUOUS sync costs more compute but eliminates the stale-data window. For a compliance use case, the cost is justified — a single regulatory citation error could trigger an audit finding.</li><li>For Distraction: Reduce retrieval from top-10 to top-3 chunks, and implement query-conditioned summarization of tool results before they enter context. <em>Trade-off:</em> Fewer chunks means occasional recall misses, but for compliance, precision matters more than recall — it’s better to say “I don’t have enough context” than to bury the right answer in noise.</li><li>For Confusion: Add explicit metadata filters to retrieval (loan_type = ‘commercial’ vs. ‘residential’) and use Lakebase-backed session isolation so prior review sessions don’t bleed into current ones. <em>Trade-off:</em> Stricter filtering requires better metadata curation upfront, adding ~2 weeks to the data engineering pipeline.</li><li>For Clash: Establish a clear instruction hierarchy in the system prompt: “System instructions override retrieved context. Retrieved regulatory text overrides analyst commentary. When in doubt, state the conflict explicitly.” <em>Trade-off:</em> This adds ~200 tokens to the system prompt but prevents the most dangerous failure mode in a regulated environment.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90d7d631f457" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why External Secrets in Unity Catalog Matter]]></title>
            <link>https://dbxdev.medium.com/why-external-secrets-in-unity-catalog-matter-33ea071dd2b6?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/33ea071dd2b6</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 16:44:48 GMT</pubDate>
            <atom:updated>2026-09-13T16:44:48.145Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern data and AI workloads rarely operate in isolation. Pipelines connect to operational databases. Notebooks call third-party APIs. Agents invoke external tools. Applications use tokens, passwords, certificates, and API keys to reach the systems around them.</p><p>Every one of those integrations creates the same uncomfortable question: <strong>where should the credential live?</strong></p><p>Many organizations already have a cloud-standard answer, such as AWS Secrets Manager or Azure Key Vault. But when a workload moves into a data platform, teams often copy the same secret into a platform-specific store. That creates two sources of truth, two permission models, two audit trails, and a rotation process that can fail in several places.</p><p><a href="https://docs.databricks.com/aws/en/security/secrets/external-secrets">External secrets in Unity Catalog</a> are designed to remove that duplication. The secret value remains in the external cloud secret manager, while Unity Catalog presents it as a governed securable object that Databricks workloads can discover and reference.</p><p>That sounds like a small integration feature. It is actually an important step toward treating credentials as governed infrastructure rather than application configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETT2KStKiV-T_s4OhPeOhw.png" /></figure><p>Without an external-secret integration, a common operating model looks like this:</p><p>1. A security or platform team creates a credential in the enterprise secret manager.<br>2. A data team copies that credential into Databricks.<br>3. Permissions are configured again in a second system.<br>4. Rotation requires both copies to be updated in the correct order.<br>5. Auditors must reconcile cloud logs, workspace configuration, code usage, and a separate access model.</p><p>This model works until it does not. A missed rotation can stop a production pipeline. An old copy can remain active after the source secret changes. A broad permission inherited in one platform can undermine a narrow policy in another. An incident investigation can become a manual exercise in joining events across tools.</p><p>The technical problem is duplication. The customer problem is <strong>operational uncertainty:</strong> teams cannot easily prove that the right workload used the right credential under the right policy at the right time.</p><p><strong>What external secrets change</strong></p><p>External secrets separate the secret’s <strong>system of record</strong> from its <strong>governance and consumption interface.</strong></p><p>- The cloud secret manager remains responsible for storing and changing the value.<br>- Unity Catalog provides a three-level name, such as `catalog.schema.secret`, and governs access with privileges including `READ SECRET` and `REFERENCE SECRET`.<br>- Databricks retrieves the current value from the external manager when a workload reads it instead of storing the value in Unity Catalog.<br>- Databricks audit events retain the Databricks user identity, while cloud-side logs record the connection’s service credential.</p><p>This gives customers a practical bridge between cloud security controls and data-platform governance. Existing secret-management investments remain in place, while data and AI teams gain a consistent Unity Catalog interface.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33ea071dd2b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ABAC DENY Policies: Closing Unity Catalog’s Additive-Only Governance Gap]]></title>
            <link>https://dbxdev.medium.com/abac-deny-policies-closing-unity-catalogs-additive-only-governance-gap-2881a2d04aa1?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2881a2d04aa1</guid>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Sat, 12 Sep 2026 03:34:36 GMT</pubDate>
            <atom:updated>2026-09-12T03:37:07.426Z</atom:updated>
            <content:encoded><![CDATA[<p>For as long as Unity Catalog has existed, its access model has been fundamentally additive. Every mechanism a platform team had — GRANT statements, inherited privileges from catalogs and schemas, group membership, ownership — could only widen access, never narrow it below what an admin, an owner, or a careless GRANT ALL PRIVILEGES had already handed out. Row filters and column masks constrained what data a query could see, but nothing constrained who could change the access rules themselves. As of this week, that gap has a fix: ABAC DENY policies, now in Beta on both AWS and Azure Databricks.</p><h3>What is an ABAC DENY policy</h3><p>A DENY policy is a new object type, authored with a CREATE POLICY statement, that explicitly blocks a privilege on a securable object whenever the object’s governed tags match a condition:</p><pre>CREATE POLICY deny_manage_access_control_sensitive_tables<br>ON CATALOG catalog_a<br>TO `account users`<br>EXCEPT `data_admins`<br>DENY MANAGE ACCESS CONTROL<br>FOR TABLES<br>WHEN has_tag(&#39;sensitive&#39;);</pre><p>In this Beta release, exactly one privilege is denyable: MANAGE ACCESS CONTROL. That single privilege is more consequential than it sounds — it governs granting and revoking access, transferring ownership, authoring further ABAC policies, configuring access-request destinations, and managing workspace-catalog bindings. In other words, it’s the privilege that controls who gets to change who has access. Policies attach at the catalog or schema level and inherit downward to tables, views, volumes, functions, models, and secrets; per-object attachment isn’t supported yet.</p><p>The evaluation semantics are what make this genuinely new rather than a rebranded REVOKE. A DENY policy does not remove a grant — it sits alongside the grant path and is checked independently. If any applicable DENY policy matches, the action is blocked, full stop, regardless of how many grants, inherited privileges, or ownership claims would otherwise have allowed it. The only built-in exemption is metastore admins. And unlike a REVOKE, which is a point-in-time mutation to the grant table, a DENY policy is a standing rule: it stays in force until someone explicitly drops it, so it survives careless or well-intentioned grants issued after the fact. The diagram below shows the evaluation path a MANAGE ACCESS CONTROL action now takes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-5mjLFObpspsiQ_XWX1Ng.png" /></figure><p>Two operational details matter if you’re planning to adopt this.</p><p>First, SHOW GRANTS and the standard permissions APIs do not reflect DENY policies — they only show the grant side of the picture. You need SHOW EFFECTIVE POLICIES to see the actual, resolved access a principal has, which means any existing tooling, dashboards, or audit scripts built against SHOW GRANTS now give an incomplete answer for objects with DENY policies attached.</p><p>Second, Beta support requires classic compute on Databricks Runtime 18 LTS or newer for SQL operations, and there are hard quotas: 10,000 policies per metastore, 100 per catalog or schema.</p><h3>Why this matters architecturally</h3><p>Most Unity Catalog governance incidents don’t come from a missing grant — they come from an excess one that nobody walked back. A table owner adds a teammate to a group for a two-week project and forgets to remove them. A well-meaning admin grants broad catalog-level access to unblock a demo and never scopes it down. Data classification tags (pii, sensitive, restricted) have existed in UC for a while as metadata, but until now nothing enforced them as a hard boundary against permission creep — only against permission use, via row filters and column masks.</p><p>DENY policies let a platform or security team draw a governance perimeter that survives organizational entropy: tag your regulated data once, write one DENY policy per classification, and no downstream grant — however generous, however accidental — can widen who gets to change access to it. That’s a meaningfully different failure mode to defend against than “did we mask the SSN column,” and it’s the piece that was missing from UC’s ABAC story since row-filtering and column-masking policies went GA.</p><p>The catch, and it’s an important one for anyone evaluating this for a compliance use case: denying MANAGE ACCESS CONTROL does not close every path through which a principal can move data out from under governance. A principal who already holds SELECT can still run CREATE TABLE … AS SELECT into a table they control, build a view over the protected object, or — if they separately hold APPLY TAG / tag ASSIGN privileges — simply remove the tag that triggered the DENY policy in the first place. Treat DENY policies as an access-governance control (who can change permissions), not a data-loss-prevention control (who can copy data). The two need to be designed together, and today that means pairing DENY policies with tight APPLY TAG grants and row-filter/column-mask policies on the same objects, rather than treating DENY as a standalone data protection mechanism.</p><h3>Recommended rollout pattern</h3><p>Don’t roll this out metastore-wide on day one. Start narrow: pick two or three catalogs that already carry a sensitive or pii governed tag, write a single DENY policy scoped to MANAGE ACCESS CONTROL on those tags, and validate with SHOW EFFECTIVE POLICIES — not SHOW GRANTS — that the policy actually blocks a test grant attempt from a non-admin principal. Audit who currently holds APPLY TAG / ASSIGN on those same objects, since that’s the escape hatch that lets someone untag their way around the policy. Only after that combination is verified in a lower environment should this move toward catalogs that back production compliance boundaries — and even then, remember this is Beta: policy semantics, quotas, and tooling support (SHOW EFFECTIVE POLICIES, audit log coverage) should be expected to keep evolving before GA.</p><p>#Databricks #UnityCatalog #Governance</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2881a2d04aa1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Context Engineering with Databricks: Part 1]]></title>
            <link>https://dbxdev.medium.com/context-engineering-with-databricks-part-1-architecture-core-concepts-95e60021a610?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/95e60021a610</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 14:53:59 GMT</pubDate>
            <atom:updated>2026-07-31T15:23:04.539Z</atom:updated>
            <content:encoded><![CDATA[<p>This is first in the series of articles i plan to write about Context Engineering on Databricks</p><p>Databricks recently launched a new certification- <a href="https://www.databricks.com/learn/certification/context-engineer-associate"><em>Databricks Certified Context Engineer Associate</em></a></p><p>The official certification provides lot of inputs on how to prepare for the certification . But first things first . What exactly is Context engineering and how does it different from Prompt Engineering .</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*BoEalPJe1R-wDNFbLwcryQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*bnLT0ka9HWTre9GeZgUoWw.png" /></figure><p>Prompt engineering is writing a good question on an exam. Context engineering is designing the entire exam room, including the reference materials on the desk, the instructions on the board, and the rules about what resources are allowed.</p><p>In a nutshell- <em>Context engineering is the practice of making sure every token in the input window is earning its place.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95e60021a610" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The COIE Framework: A Practical Structure for Better AI Prompts]]></title>
            <link>https://dbxdev.medium.com/the-coie-framework-a-practical-structure-for-better-ai-prompts-97316f670ddc?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/97316f670ddc</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 14:46:04 GMT</pubDate>
            <atom:updated>2026-07-09T14:46:04.030Z</atom:updated>
            <content:encoded><![CDATA[<p>The <strong>COIE (Context, Outcome, Instruction, Example)</strong> framework is a structured approach for turning general AI prompts into precise requests. When you treat an AI assistant as a capable collaborator that follows instructions literally, the quality of the output tends to reflect the clarity of the input. COIE provides a consistent way to supply that clarity.</p><h3>The Four Parts of COIE</h3><p>The effectiveness of COIE rests on four distinct components:</p><ul><li><strong>Context :</strong> Provides the <strong>background and situational details</strong> needed for the model to understand the purpose behind a request. Without context, models often default to safe, generic answers that lack specific utility.</li><li><strong>Outcome (O):</strong> Defines <strong>what success looks like</strong>. Stating the goal helps the assistant adjust tone, level of detail, and formatting for the intended audience.</li><li><strong>Instruction (I):</strong> Specifies the <strong>exact task to complete</strong>. Clear instructions reduce ambiguity and make the response easier to act on.</li><li><strong>Example (E):</strong> Shows <strong>what good output looks like</strong>. Examples are often the most effective form of instruction, because they let the model mirror a preferred style or structure through few-shot prompting.</li></ul><h3>Applying COIE: A Technical Prompt</h3><p>The following prompt uses the COIE framework to convert a Snowflake transformation into a Databricks Silver-layer table.</p><h3>The COIE Prompt</h3><p><strong>Context:</strong> We are migrating our data processing pipelines from Snowflake to the Databricks Lakehouse. We use a <strong>Medallion architecture</strong>, and this specific task targets the <strong>Silver (validated) layer</strong>. All tables must be managed by <strong>Unity Catalog</strong> using the three-level namespace: catalog.schema.table_name.</p><p><strong>Outcome:</strong> I need production-ready Spark SQL that creates a <strong>managed Delta table</strong>. The output should be optimized for the <strong>Photon engine</strong> and use ANSI-compliant SQL.</p><p><strong>Instruction:</strong> Convert the provided Snowflake SQL to Databricks Spark SQL.</p><ol><li>Change Snowflake’s colon-based semi-structured access (e.g., data:index) to the Databricks VARIANT type or get_json_object where applicable.</li><li>Ensure the CREATE TABLE statement includes TBLPROPERTIES for Delta optimization.</li><li>Use the main.silver prefix for the target table.</li></ol><p><strong>Example:</strong> The transformation below illustrates the expected pattern.</p><p><em>Snowflake input:</em></p><pre>CREATE TABLE raw_data AS SELECT payload:id::int as id FROM source_stream;</pre><p><em>Databricks output:</em></p><pre>CREATE TABLE main.silver.raw_data USING DELTA AS<br>SELECT CAST(get_json_object(payload, &#39;$.id&#39;) AS INT) as id<br>FROM main.bronze.source_stream;</pre><h3>Source SQL to Convert</h3><pre>CREATE OR REPLACE TABLE sales_october AS <br>SELECT <br>    transaction_id,<br>    customer_details:phone_number::string as phone,<br>    TO_VARIANT(metadata) as raw_meta,<br>    current_timestamp() as load_time<br>FROM public.raw_sales<br>WHERE status = &#39;SUCCESS&#39;;</pre><h3>Comparing Prompts: A Business Decision</h3><p>To see how the framework changes results, consider a scenario in which a sales professional needs to decide which hospital system to prioritize for the upcoming quarter.</p><h3>Without COIE</h3><blockquote><em>“Which account should I prioritize this quarter: Account A or Account B?”</em></blockquote><p><strong>Typical result:</strong> A shallow response such as, “Prioritize Account A because it has higher revenue potential.” This is hard to trust because the <strong>reasoning stays implicit</strong>, and the model has no context about resource constraints or deal dynamics.</p><h3>With COIE</h3><p>With COIE applied, the same question becomes a structured analytical request:</p><ul><li><strong>Context:</strong> Our team has limited bandwidth this quarter and must focus on opportunities with the highest likelihood of conversion. <strong>Account A</strong> is a large academic system with <strong>$2.5M potential</strong>, but it follows a lengthy 12-month procurement process. <strong>Account B</strong> is a community network with <strong>$1.2M potential</strong>, but it is already dissatisfied with a competitor and has an engaged procurement team.</li><li><strong>Outcome:</strong> I need a reasoned recommendation on which account to prioritize, formatted as a brief I can share with my manager.</li><li><strong>Instruction:</strong> Use a <strong>chain-of-thought approach</strong> to evaluate both options. Specifically:</li></ul><ol><li>Compare short-term and long-term revenue potential.</li><li>Assess the barriers to closing each deal.</li><li>Recommend a primary focus and explain the strategic logic.</li></ol><ul><li><strong>Example:</strong> “I recommend Account [X] because while [Factor 1] is a challenge, [Factor 2] provides a faster path to impact.”</li></ul><h3>Why Structured Prompts Produce Better Results</h3><p>COIE shifts the interaction from a simple query to a <strong>collaborative reasoning process</strong>. By pairing clear <strong>Instructions (I)</strong> with an <strong>Example (E)</strong> and asking for step-by-step reasoning, you can review the model’s logic and check the result against known facts. This structured approach works across chat-style LLMs and generally produces more reliable, defensible outputs than unstructured requests.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97316f670ddc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[392 Tokens Per Second: How Databricks and the B300 Just Reset the AI Speed Limit]]></title>
            <link>https://dbxdev.medium.com/392-tokens-per-second-how-databricks-and-the-b300-just-reset-the-ai-speed-limit-7d84285b7e6e?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d84285b7e6e</guid>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 17:23:43 GMT</pubDate>
            <atom:updated>2026-06-29T19:45:37.490Z</atom:updated>
            <content:encoded><![CDATA[<h3>When Speed Becomes the Smartest Thing About Your Model</h3><p>For a long time, picking an AI model felt like an old-school engineering trade-off. You could have intelligence, or you could have speed, but pretending you could have both at once usually meant something was about to break in production.</p><p>That trade-off is quietly dying. And in the enterprise, “smart but slow” is starting to feel like a luxury nobody can actually afford. A model that reasons brilliantly in 14 seconds is fine for a research demo. It’s useless behind a developer’s cursor, or inside an agent that has to think five times before it does anything at all.</p><p>The bottleneck has moved. It used to be the model. Now it’s the stack underneath it.</p><p>This is the backdrop against which Databricks just did something genuinely interesting.</p><h3>392 tokens per second, and why that number matters</h3><p>Databricks recently took the #1 spot on the Artificial Analysis leaderboard for GLM-5.2, clocking in at 392 tokens per second. For context, Fireworks AI sits at 328 t/s, and Databricks itself was hitting around 201 t/s on NVIDIA H200 nodes only a short while ago.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhtkMFIKUh9zYHclAjh9-Q.png" /></figure><p>So what changed? Two things, mostly.</p><p>The first is hardware. They moved from H200 to NVIDIA’s Blackwell-based B300, and the jump is exactly the kind of “substantial additional boost” you’d hope for on paper but rarely see this cleanly in practice.</p><p>The second, and probably more impressive piece, is the software work. The team rebuilt a lot of the inference stack around the model’s Mixture-of-Experts (MoE) architecture — DeepGEMM’s FP8 grouped GEMM kernels, DeepEP for expert parallelism, custom kernels, speculative decoding tuned specifically for the way GLM-5.2 generates. None of these individually sound revolutionary. Stacked together, they’re the reason the number nearly doubled.</p><p>Matei Zaharia summed it up in his usual understated way:</p><blockquote><em>“The Databricks inference team is 🔥”</em></blockquote><p>The reason this matters more than a leaderboard slot is what it unlocks. Run GLM-5.2 in “High” reasoning mode — lighter chain-of-thought, faster outputs — and at ~400 t/s you finally have enough headroom for the multi-step agentic workflows everyone has been demoing but few have shipped. The agent gets to think, plan, retry, and act inside the same response budget that used to barely cover a single completion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dfkrDFPkyEgjEB0TmO5KiQ.png" /></figure><h3>The cost story is, frankly, the more disruptive one</h3><p>Speed is the headline. Cost is what actually changes purchasing decisions.</p><p>Databricks is serving GLM-5.2 at roughly <strong>85% below</strong> what you’d pay for proprietary frontier models for comparable work, especially in coding. The Artificial Analysis numbers tell the story pretty plainly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EW-sghCb00GCQ54ZRe4sbg.png" /></figure><p>If you’ve ever had to explain a six-figure monthly AI bill to a CFO, you already know why this matters. “Near-Opus reasoning at one-eighth the cost” is the kind of line that turns AI from a budget item people fight over into something teams can actually deploy at scale without flinching.</p><h3>A 1-million token context window (and what it actually costs to run)</h3><p>Speed handles <em>how fast</em>. Context handles <em>how much</em>.</p><p>GLM-5.2 jumped from a 200K-token window in v5.1 to <strong>1M tokens</strong>. That’s not an incremental bump; it’s a different way of working. Instead of retrieving snippets and praying the right ones made it into the prompt, you can hand the model the entire codebase and let it figure out the relevant parts itself.</p><p>In practice, this nudges agentic coding away from RAG and toward something closer to “just give it everything.” RAG isn’t going away, but a lot of the gymnastics around chunking, re-ranking, and hybrid search start to look optional rather than mandatory.</p><p>The catch nobody talks about is the KV cache. At 1M tokens, an FP16 KV cache balloons to around <strong>160 GB</strong> — which is more than a single GPU node can comfortably hold. Databricks gets around this by running the KV cache in FP8, which cuts the footprint roughly in half to ~80 GB. That fits cleanly on 8x H200 or 8x B200 nodes, with enough room left over for the model weights themselves. Less elegant than it sounds in a blog post, but it’s the kind of detail that decides whether the feature actually works in production or just lives on the spec sheet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FKNpd_CegcX52NKAVyXiA.png" /></figure><h3>“744 billion parameters” is not the number you should be looking at</h3><p>There’s a common reflex when people hear “MoE model with 744B parameters” — they assume it must be slow. It isn’t, and the reason is worth understanding.</p><p>GLM-5.2 has 744B <em>total</em> parameters but only about <strong>40B active</strong> per forward pass. The router picks which experts to consult on a per-token basis. You get the broad knowledge of a near-trillion-parameter model with the latency profile of a much smaller one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RchJ_QxzGw16Q5Q60rPlgQ.png" /></figure><p>DeepEP handles the expert routing across GPUs efficiently enough that the parallelism overhead doesn’t eat the gains. The net effect: deep knowledge, fast outputs, and a serving cost that doesn’t scale with the total parameter count.</p><h3>The quieter strategic move: Mosaic AI Gateway</h3><p>Benchmarks get the attention. The Gateway is the part I’d watch.</p><p>The Mosaic AI Gateway is an abstraction layer that lets you swap underlying models without touching application code. Today it’s GLM-5.2; tomorrow it’s whatever beats it. The application doesn’t care.</p><p>That’s a pretty pointed move at Snowflake, and it positions Databricks for the enterprise platform market rather than just the data-engineering crowd it grew up serving. It also gives buyers three things they actually need:</p><ul><li><strong>Governance</strong> — every call tracked and audited through Unity Catalog.</li><li><strong>Rate limiting</strong> — real per-team controls on spend and token usage, instead of the usual “we’ll figure it out in next quarter’s invoice” approach.</li><li><strong>PII filtering</strong> — automated scrubbing of sensitive data before anything leaves the secure perimeter.</li></ul><p>The strategy here is to commoditize the model layer. If models become interchangeable, the platform underneath them — the governance, the security, the optimization — is what you’re really buying. That’s a comfortable position for Databricks to be in.</p><h3>So where does this leave us?</h3><p>The AI race is shifting under our feet. The “biggest brain” framing is starting to feel dated. The interesting competition now is about <em>intelligence at speed and scale, served cheaply and governed well.</em></p><p>GLM-5.2 on Databricks isn’t perfect, and the leaderboard will move again — probably before this post is a month old. But the combination of 392 t/s, a 1M-token window, and an 85% cost advantage over the incumbents is hard to argue with. It’s the kind of stack that lets a team actually ship the agent they’ve been prototyping for the last year.</p><p>Which leaves the real question: does the future of AI belong to the labs building the biggest brains, or to the platforms building the fastest nerves?</p><p>In production, I think we already know the answer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d84285b7e6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data to AI Agents: What Databricks Agent Bricks Actually Gives You]]></title>
            <link>https://dbxdev.medium.com/from-data-to-ai-agents-what-databricks-agent-bricks-actually-gives-you-ca0a6dd00876?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca0a6dd00876</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 04:28:58 GMT</pubDate>
            <atom:updated>2026-02-23T04:28:58.374Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Agent Bricks exists</h3><p>Most companies can call an LLM, but struggle to ship agents that are accurate, affordable, and safe enough for real production use. Projects often stall because they are built as one‑off prototypes instead of as systems that integrate data, tools, evaluation, and governance end‑to‑end.</p><p>Agent Bricks tackles this by focusing on <strong>production AI agents optimized on your own data</strong>, with automation around data generation, benchmarking, and tuning so teams spend less time on trial-and-error and more time defining what the agent should do</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNQ-rtjfbqlMjYKowJGwIg.png" /></figure><h3>What Agent Bricks actually does</h3><p>At its core, Agent Bricks lets you describe an agent’s job in natural language, connect it to governed enterprise data, and then automatically builds and tunes that agent for quality and cost. It generates synthetic examples and custom evaluation suites tailored to your domain, then searches across models, prompts, and configurations to find strong defaults for your use case.</p><p>Because it’s part of the Databricks Data Intelligence Platform, the same governance layer (Unity Catalog) that controls access to tables and files also governs agents, tools, and external calls, while AI Gateway logs and controls all model traffic. This combination means you can improve an agent over time without rebuilding it from scratch, and still keep a full audit trail of what it did and why</p><h3>The main agent patterns you can build</h3><p>Agent Bricks ships with opinionated patterns for common enterprise needs, so you can start from a template rather than a blank page.</p><p><strong>Information extraction agents</strong> — Pull structured fields (names, IDs, dates, key entities) out of PDFs, emails, or forms and push them into downstream systems, while tracking field‑level accuracy.</p><p><strong>Knowledge and assistance agents</strong> — Turn document collections (policies, manuals, internal docs) into grounded Q&amp;A or helpdesk experiences that answer questions with citations back to the source</p><p><strong>Custom text and content agents</strong> — Summarize long reports, draft domain‑specific content, or transform text into your organization’s voice using models trained or configured on your own data.</p><p><strong>Supervisor / multi‑agent systems</strong> — Use Supervisor Agent to coordinate multiple specialized agents and tools (Genie Spaces, Knowledge Assistants, Unity Catalog functions, MCP servers) so complex tasks are broken into smaller steps and executed by the right “specialist.”</p><h3>How the lifecycle looks in practice</h3><p>Under the hood, Agent Bricks aligns to a simple but powerful lifecycle for agent systems:</p><p><strong>Prepare data</strong> — Clean, index, and expose enterprise data (structured and unstructured) so agents can query it without copying it around.</p><p><strong>Build the agent</strong> — Define the agent’s role, reasoning behavior, memory, and tools (APIs, functions, retrieval, external services) using low‑code or code‑first approaches.</p><p><strong>Deploy</strong> — Expose the agent behind a governed endpoint that applications, chat UIs, or workflows can call.</p><p><strong>Evaluate and optimize</strong> — Use automatically generated benchmarks, AI judges, and human feedback to measure quality, then let Agent Bricks adjust prompts, models, and settings for better cost–quality tradeoffs</p><p><strong>Govern</strong> — Enforce access controls, audit interactions, and apply safety policies across data, models, tools, and external calls through Unity Catalog and AI Gateway.</p><p>Agent Bricks is less about “yet another chatbot” and more about giving you a repeatable way to turn your data, models, and tools into <strong>measurable, governable AI agents</strong> that can actually run in production.</p><p>#Databricks #AI #Agents</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca0a6dd00876" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loading Databricks Assistant with the AI Dev Kit Skills]]></title>
            <link>https://dbxdev.medium.com/loading-databricks-assistant-with-the-ai-dev-kit-skills-9b97227faaff?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b97227faaff</guid>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 06:38:29 GMT</pubDate>
            <atom:updated>2026-02-15T06:40:37.203Z</atom:updated>
            <content:encoded><![CDATA[<p>By loading the <strong>Databricks AI Dev Kit</strong>, you can give your Assistant specific domain knowledge — like how to build production-ready pipelines using Spark Declarative Pipelines (SDP).</p><p>Here is a step-by-step guide to expanding your Assistant’s powers with AI Dev KIT.</p><h3>1. Clone the AI Dev Kit</h3><p>First, you need the skill definitions on your workspace.</p><p>Clone the repo in a workspace using git repos . Repo — https://github.com/databricks-solutions/ai-dev-kit.git</p><p>Locate the skills folder in ai-dev-kit and copy them to user home workspace under .assistant folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NMvbpJ_ow0OG8FP4NUotQ.png" /></figure><h3>2. Configure the Assistant’s Skill Registry</h3><p>The Assistant looks for a specific file to know which external skills are available. You need to create a hidden directory and a markdown file in your home folder. Create the file at ~/.assistant/SKILLS.md.</p><h3>3. Map the Skill Paths</h3><p>Now, you need to tell the Assistant exactly where the skills live.</p><p>Copy the full workspace paths for the skills you want to use and paste them into .assistant folder and update your SKILLS.md file (one path per line) with full path for SKILLS.md file for each skill. It should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BuAzp5Wl2A7lDwwI_XAzw.png" /></figure><h3><strong>4. Initialize and Verify</strong></h3><p>Once the paths are saved, it’s time to wake up the Assistant. Action: Open the Assistant pane and type: “Load and list all available skills.”</p><p>If configured correctly, the Assistant will confirm it has indexed the new capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MEVdk45iTMDDuk604GelvA.png" /></figure><h3>5. Put the Skills to Work</h3><p>Now that your Assistant is “trained,” let’s use the <strong>Spark Declarative Pipelines (SDP)</strong> skill to bootstrap a project.</p><ol><li><strong>Create a Project Folder:</strong> Set up a new directory in your workspace for your work.</li><li><strong>Enter Agent Mode:</strong> In the Assistant chat, ensure you are using Agent mode for complex task execution.</li><li><strong>Generate the Scaffold:</strong> Ask the Assistant:<br><em>“Create a starter SDP project repo using the spark-declarative-pipelines skill.”</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AmoRDNbxtWqEnKOQ3jd4IQ.png" /></figure><h3>6. Refine and Expand</h3><p>The Assistant won’t just give you a template; it can help you build the entire logic. Once the starter files are generated, follow up with:</p><p><em>“Fill in more details for a bronze-to-silver transformation using this schema…”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b97227faaff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning Databricks into an AI pair‑programmer with Claude‑powered coding agents.]]></title>
            <link>https://dbxdev.medium.com/turning-databricks-into-an-ai-pair-programmer-with-claude-powered-coding-agents-1665ad0bb43f?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1665ad0bb43f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 12:11:28 GMT</pubDate>
            <atom:updated>2026-01-29T12:11:28.745Z</atom:updated>
            <content:encoded><![CDATA[<p>This guide walks through a practical, end‑to‑end setup: installing Claude Code, wiring it to Anthropic models served from Databricks, and configuring authentication so everything “just works” from your terminal and editor. You’ll generate model‑scoped API keys, plug them into Claude’s settings, bypass the default login flow for a smoother developer experience, and finish by testing the workflow directly from the CLI and VS Code.Here are the steps</p><h3>Install Claude Code</h3><p><a href="https://code.claude.com/docs/en/setup#installation">Set up Claude Code - Claude Code Docs</a></p><h3>Generate API Keys Anthropic Model</h3><p>Login into your databricks workspace.Click on Serving under AI/ML section( Bottom left corner)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*k3a5Ok6bfIB0XJsTVzldeg.png" /><figcaption>Serving</figcaption></figure><p>Click on Integrate coding agents(<strong>Get Started</strong>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*cefAahF38laGvN4oYaPyEg.png" /><figcaption>Serving endpoints</figcaption></figure><p>Select Other integrations</p><p>Select a coding integration(Claude Code)</p><p>Select Default Anthropic Model and other options</p><p>Generate API Keys</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*LeaeC9eSeOJNSmCgYjJ1gA.png" /></figure><h3>Update keys for claude settings</h3><p>Go to ~/.claude/settings.json</p><p>Update / Replace the keys generated from previous step</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*eGOZXPKTIFVjirHWF_iCMQ.png" /></figure><h3>Override the login method</h3><p>Update/Create ~/.claude/settings.json with following value</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*njPMErFU5wbjpxJXwCYHkg.png" /></figure><p>This prevents Claude from prompting you to select a login method</p><h3>Install Claude Code for VS Code Plugin</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjZ4iUPAhYRNM7UKtf5s8Q.png" /></figure><h3>Test Drive claude</h3><p>We will ask claude code to create a sample datastage.xml job that would mimic the export of a datastage job . Then we will ask it to convert this job to databricks</p><p>Type <strong>claude</strong> . Press enter .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-kB03U57y-vwa7gLcXnVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6WJFT_USythitfTgmkHZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aG3c8F-D1lNImGwBPEwk0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QitNyUZexr1J-_lVhOOjDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rabxlAubtV1ULbln9TZ-yg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJPvJDJN7HPYaj0HTTB_LQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1665ad0bb43f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Powering Databricks Assistant with Skills]]></title>
            <link>https://dbxdev.medium.com/powering-databricks-assistant-with-skills-48ba5779768f?source=rss-04869b77d4b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/48ba5779768f</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[skills]]></category>
            <category><![CDATA[databricks-assistant]]></category>
            <dc:creator><![CDATA[Pradeep Singh]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 12:01:59 GMT</pubDate>
            <atom:updated>2026-01-19T12:01:59.191Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Gap Between Helpful and Reliable AI</h3><p>Databricks Assistant is already a powerful productivity tool. It can explain code, generate SQL, and help troubleshoot issues interactively.</p><p>However, like most general-purpose AI assistants, it has one limitation: <strong>inconsistent behavior across repeated or domain-specific tasks</strong>.</p><p>For example:</p><ul><li>Sometimes it optimizes a query correctly, sometimes it doesn’t</li><li>Sometimes it follows internal best practices, sometimes it misses them</li><li>Sometimes it explains <em>what</em> to do, but not <em>how your team actually does it</em></li></ul><p>This is where <strong>Skills</strong> come in.</p><h3>What Are Databricks Assistant Skills?</h3><p>A <strong>skill</strong> is a reusable capability that extends Databricks Assistant with <strong>explicit instructions, constraints, and execution logic</strong>.</p><p>Instead of relying on the assistant to “figure things out,” a skill:</p><ul><li>Defines <strong>what problem it solves</strong></li><li>Specifies <strong>how it should solve it</strong></li><li>Produces <strong>deterministic and repeatable results</strong></li></ul><p>Think of skills as:</p><ul><li>Guardrails for AI behavior</li><li>Codified best practices</li><li>Reusable operational knowledge</li></ul><p>This is especially important in data platforms, where <strong>small inconsistencies can lead to performance regressions, cost increases, or governance issues</strong>.</p><h3>Why Skills Matter: Consistency Over Cleverness</h3><p>Without skills, Databricks Assistant operates in a <strong>best-effort mode</strong>:</p><ul><li>It may suggest optimizations but not apply them consistently</li><li>It may overlook platform-specific nuances</li><li>It may generate SQL that “works” but isn’t optimal</li></ul><p>With skills:</p><ul><li>The assistant follows <strong>your rules</strong></li><li>The output becomes <strong>predictable</strong></li><li>The behavior scales across teams</li></ul><p>This shifts the assistant from:</p><p>“A helpful chatbot”<br> to<br> “A platform-aware engineering assistant”</p><p><strong>Demo Scenario: Optimizing Column Order for Statistics Collection</strong></p><p>In Databricks (Spark/Delta), column statistics are collected only for the first 32 columns by default.If frequently filtered or joined columns appear after the first 32 positions, the optimizer may lack the information it needs — leading to inefficient queries.</p><p>In real projects, this happens often:</p><ul><li>Wide tables</li><li>Schema evolution over time</li><li>Columns added without considering physical order</li></ul><p>Manually fixing this is:</p><ul><li>Tedious</li><li>Error-prone</li><li>Inconsistent across teams</li></ul><h3>Demo :Building and testing the Skill</h3><p>Instead of relying on the assistant to <em>remember</em> this nuance, we encode it into a skill.</p><p>The skill:</p><ul><li>Analyzes a SQL file<br>Identifies columns used in WHERE and JOIN conditions</li><li>Reorders the final SELECT list so these columns appear within the first 32 positions</li><li>Preserves query semantics and column definitions</li></ul><p><strong>SKILLs file</strong></p><ul><li>Create a .assistant folder in user workspace location if not created already</li><li>Upload or create a <a href="https://raw.githubusercontent.com/pradeepsingh87/databricks_assistant_skill/refs/heads/main/SKILLS/REORDER_COLS_FOR_STATS.md">SKILL.md</a> file with appropriate name (Example REORDER_COLS_FOR_STATS.md)</li><li>Test the skill on a <a href="https://raw.githubusercontent.com/pradeepsingh87/databricks_assistant_skill/refs/heads/main/sample_scripts/demo_sql_file_for_stats_reorder.sql">script</a> using assistant using slash command with name of skill</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFjR5t8yOoNU7RIVXvvktg.png" /><figcaption>SKILLs.md file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aquNVZpIutbn2hBcDbWrDw.png" /></figure><h3>Further Reads</h3><ul><li><a href="https://docs.databricks.com/aws/en/assistant/skills">Extend the Assistant with agent skills | Databricks on AWS</a></li><li><a href="https://agentskills.io/specification">Specification - Agent Skills</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48ba5779768f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>