<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudarshan Koirala on Medium]]></title>
        <description><![CDATA[Stories by Sudarshan Koirala on Medium]]></description>
        <link>https://medium.com/@sudarshan-koirala?source=rss-55cd50a9c816------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V4Rwzv7giTWZZPg29ewARA.jpeg</url>
            <title>Stories by Sudarshan Koirala on Medium</title>
            <link>https://medium.com/@sudarshan-koirala?source=rss-55cd50a9c816------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:14:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sudarshan-koirala/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multi-Agent Systems in Databricks with Genie Agents and the Agent Bricks CLI]]></title>
            <link>https://medium.com/@sudarshan-koirala/multi-agent-systems-in-databricks-with-genie-agents-and-the-agent-bricks-cli-bd7ae3c15c50?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd7ae3c15c50</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[genie-agents-databricks]]></category>
            <category><![CDATA[agentbricks-cli]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 12:01:01 GMT</pubDate>
            <atom:updated>2026-11-09T12:01:01.732Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Supervisor Agent is now legacy. Here is the new way to build a team of AI agents in Databricks.</strong></p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gnMF2Qgz8NLNfeEtM4zM1g.png" /></figure><p>This post is about building a <strong>multi-agent system in Databricks</strong> the way Databricks recommends today. The Supervisor Agent is now a legacy offering. For low-code agents, you use <strong>Genie Agents</strong>. To make many agents work together in code, you use the <strong>Agent Bricks CLI</strong>.</p><p>In the video, I build a small team: two Genie Agents and one sandbox, with a lead agent that runs locally (LangGraph under the hood). I ask one question, the lead calls three tools, and I get one combined answer. Every step is logged in MLflow traces.</p><h3>What changed?</h3><p>Databricks now lists the <strong>Supervisor Agent as legacy</strong> and does not recommend it for new systems. The replacement:</p><ul><li><strong>Genie Agents</strong> (formerly Genie Spaces) -&gt; low-code agents on top of your tables.</li><li><strong>Agent Bricks CLI</strong> -&gt; a code-first tool to build, run locally (agentbricks dev) and deploy agents. You give it Genie Agents as tools with agentbricks tools add genie-agent.</li></ul><p>If you built a Supervisor Agent before, it still works. But for new work, go with Genie Agents + the Agent Bricks CLI.</p><h3>Why a team of agents and not one big agent?</h3><p>You can build one agent that sees all your data. But the instructions get long and confusing, it is hard to find what went wrong, and one set of permissions must cover every tool.</p><p>With a <strong>boss + specialists</strong> setup, each agent has one short, focused job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HIlvY_uYpntDN_IoCS4Pjw.png" /></figure><p>Think of a good team lead. You do not do every task yourself. You give each task to the right person, collect the answers, and put them together. Same idea here.</p><h3>How it works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZrP6TUgi_NtKGp34iKqhg.png" /></figure><ol><li>You ask one question: <em>“Why are we losing APAC customers? Check the accounts, the support tickets and our SLA policy, and tell me what to fix first.”</em></li><li>The <strong>lead agent</strong> (built with the Agent Bricks CLI) decides which specialists to call.</li><li><strong>accounts_analyst</strong> and <strong>support_analyst</strong> are Genie Agents. Each one reads only its own table.</li><li>The <strong>sandbox</strong> runs code and reads the SLA policy from a Unity Catalog volume. When the work is done, the sandbox is deleted.</li><li>The lead combines all answers into one reply.</li><li>Every tool call, latency, token count and error goes to <strong>MLflow traces</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*DxduW8-NvTQo5bAvkgJs7w.gif" /></figure><p>A nice bonus: the MLflow UI has an <strong>AI Assistant</strong> that uses your local Claude Code. I asked it “Which tools are slowest?” and it read my traces and answered. No digging by hand.</p><h3>Build it yourself in 6 steps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GuEePWPwRUNqB-YWzSciZQ.png" /></figure><ol><li><strong>Upload the data</strong> -&gt; two CSV tables into a schema, and the SLA policy (PDF or Markdown) into a Unity Catalog volume.</li><li><strong>Create two Genie Agents</strong> -&gt; accounts_analyst and support_analyst, one per table. Copy each agent ID from the Configure tab.</li><li><strong>Install the CLI</strong> -&gt; you need uv and the Databricks CLI.</li></ol><pre>uv tool install databricks-agentbricks<br>databricks auth login --host https://&lt;your-workspace-url&gt; --profile my-workspace<br>agentbricks login --profile my-workspace</pre><ol><li><strong>Add the tools</strong> -&gt; run agentbricks tools add genie-agent for each analyst, and agentbricks tools add sandbox for the policy. The tools land in agent.toml.</li><li><strong>Write the system prompt and pick a model</strong> -&gt; add the prompt in agent.py and pick a model from AI Gateway (I used Qwen3 Next Instruct from the Free Edition list).</li><li><strong>Run it locally</strong> -&gt; agentbricks dev starts the chat app on localhost:8000 and sends traces to MLflow.</li></ol><p>All the working code and the exact commands are in my GitHub repo <strong>meridian-ops</strong> (link in the video description).</p><h3>A note on the Databricks Free Edition</h3><ul><li><strong>Build and test locally</strong> -&gt; works fine on the Free Edition.</li><li><strong>Deploy</strong> -&gt; does not work on the Free Edition (the app runtime needs features the Free Edition does not allow). Use a trial or a paid workspace.</li></ul><p>Also, these features change fast. The docs I used were updated in October 2026. Always check the latest Databricks documentation before you start.</p><p>I have made this post for all readers. Not under the paid wall.</p><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://docs.databricks.com/aws/en/agents/legacy/">Databricks: Legacy agent offerings and their replacements</a></li><li><a href="https://docs.databricks.com/aws/en/agents/custom-agents/agent-bricks-cli">Databricks: Agent Bricks CLI</a></li><li><a href="https://docs.databricks.com/generative-ai/agent-bricks/multi-agent-supervisor">Databricks: Supervisor Agent (legacy)</a></li><li><a href="https://docs.databricks.com/aws/en/genie-agents/set-up">Databricks: Create and manage a Genie Agent</a></li><li><a href="https://docs.databricks.com/gcp/en/genie-agents/conversation-api">Databricks: Use the Genie Agents API</a></li><li><a href="https://pypi.org/project/databricks-agentbricks/">PyPI: databricks-agentbricks</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/agents/legacy/">Azure Databricks: Legacy agent offerings</a></li><li><a href="https://mlflow.org/docs/latest/genai/getting-started/try-assistant/">MLflow: AI Assistant</a></li><li><a href="https://mlflow.org/blog/mlflow-claude-code">MLflow Blog: MLflow and Claude Code</a></li></ol><p>No more text here -&gt; this is the content covered in the video 👇</p><p>Link to video 👇 Happy learning 😎</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjsZlcUb0wV4&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjsZlcUb0wV4" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a49f1b5f276ff6ee4c3f6eab4985c189/href">https://medium.com/media/a49f1b5f276ff6ee4c3f6eab4985c189/href</a></iframe><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd7ae3c15c50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agent Bricks CLI: From Empty Folder to Live Databricks Agent]]></title>
            <link>https://medium.com/@sudarshan-koirala/agent-bricks-cli-from-empty-folder-to-live-databricks-agent-11d80deefa6f?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/11d80deefa6f</guid>
            <category><![CDATA[agent-bricks]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[lakebase]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 11:06:53 GMT</pubDate>
            <atom:updated>2026-10-16T11:06:53.923Z</atom:updated>
            <content:encoded><![CDATA[<p>Build an AI agent on your laptop, then ship it to Databricks with memory, sessions and traces. A few commands.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8NKy-s08H3eyenkHhHknA.png" /></figure><p>This post is about the <strong>Agent Bricks CLI</strong>, a new command line tool from Databricks (still in Beta) for developers who build and deploy custom agents in code.</p><p>Many people are confused about Agent Bricks, because you do not find an “Agent Bricks” button in the Databricks UI. Now you can just type agentbricks in your terminal and do everything from there.</p><h3>What is Agent Bricks CLI?</h3><p>Think of it like a starter kit plus a delivery truck for AI agents.</p><ul><li>The <strong>starter kit</strong>: agentbricks init creates a full agent project for you (LangGraph or OpenAI framework) based on Databricks best practices. You get a chat app, sample tools, human approval and a config file called agent.toml.</li><li>The <strong>delivery truck</strong>: agentbricks deploy takes the same project and runs it on Databricks as an app, with sessions and memory stored in <strong>Lakebase</strong> and traces in <strong>MLflow</strong>.</li></ul><p>You do not manage model keys or endpoints. The agent uses the models from your Databricks workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-cNiifZnqlVs1b_ljILXFQ.png" /></figure><h3>The Commands at a Glance</h3><p>In the video I run everything with uv run in front. Version used: 0.5.0. It is Beta, so commands can change.</p><pre># 1. Connect to your workspace<br>databricks auth login --host &lt;your-workspace-url&gt; --profile &lt;profile&gt;<br>agentbricks login --profile &lt;profile&gt;</pre><pre># 2. Create the project<br>agentbricks init --framework langgraph &lt;project-name&gt;</pre><pre># 3. Run it locally (chat app on localhost:8000)<br>agentbricks dev</pre><pre># 4. Attach long-term stores (Lakebase)<br>agentbricks sessions bind &lt;session-store&gt;<br>agentbricks memory bind &lt;memory-store&gt;</pre><pre># 5. Ship it<br>agentbricks deploy &lt;short-name&gt;</pre><pre># 6. Stop the app<br>agentbricks deployments stop &lt;name&gt;</pre><p>Tip: keep the deploy name short. The CLI adds an agent-bricks- prefix and long names fail.</p><h3>Laptop vs Databricks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyUlKYPj9TYaYKFq5yEYog.png" /></figure><p>Two things I hit in the video, so you do not have to:</p><ul><li><strong>The default model may not exist in your workspace.</strong> The template uses Claude Sonnet 4.5. On Free Edition, pick another model from the dropdown (I used GPT OSS 20B) and it works.</li><li><strong>Free Edition deploy failed for me</strong> with “scale to zero must be enabled for this workspace tier”, when memory and sessions were bound. I switched to a trial workspace and the deploy worked. A <a href="https://medium.com/lab-notes/agent-bricks-cli-beta-on-databricks-free-edition-a-live-agent-in-188-seconds-72839cbca91b">community write-up</a> reports a Free Edition deploy with the stores unbound.</li></ul><h3>The Best Part: Memory</h3><p>I told the agent: “Remember, I own the APAC region. Always show me APAC numbers first.” It called a remember tool and saved that fact in Lakebase.</p><p>Then I opened a <strong>new session</strong> and asked for a quick customer health update. No word about APAC. The answer started with APAC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7Mw2OmLMehcwT7S1Hhbcw.png" /></figure><p>One more thing: stopping the app does <strong>not</strong> delete your sessions and memory. They live in Lakebase, separate from the app. Delete the stores yourself when you clean up.</p><p>I have made this post for all readers. Not under the paid wall.</p><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><p>No more text here -&gt; this is the content covered in the video 👇</p><p>All commands and the demo CSV tool (region_health.py) are in the GitHub repo linked in the video description.</p><p><a href="https://github.com/sudarshan-koirala/agent-bricks-cli-demo">GitHub - sudarshan-koirala/agent-bricks-cli-demo: Companion files for the Agent Bricks CLI tutorial - Meridian Labs demo</a></p><p>Link to video 👇 Happy learning 😎</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9KfsoB_-P-s&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9KfsoB_-P-s&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F9KfsoB_-P-s%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2762658497f2a979eab53598b01d82e4/href">https://medium.com/media/2762658497f2a979eab53598b01d82e4/href</a></iframe><h3>Sources</h3><ol><li><a href="https://docs.databricks.com/aws/en/agents/custom-agents/agent-bricks-cli">Databricks Docs: Agent Bricks CLI</a></li><li><a href="https://docs.databricks.com/aws/en/agents/deploy/agent-server">Databricks Docs: Agent servers (DurableAgentServer)</a></li><li><a href="https://learn.microsoft.com/en-us/azure/Databricks/agents/agent-memory/managed-memory">Azure Databricks Docs: Managed agent memory</a></li><li><a href="https://docs.databricks.com/aws/en/agents/agent-memory/managed-sessions">Databricks Docs: Managed agent sessions</a></li><li><a href="https://github.com/databricks/databricks-ai-bridge/tree/main/integrations/agentbricks">GitHub: databricks-ai-bridge, agentbricks integration</a></li><li><a href="https://docs.databricks.com/aws/en/dev-tools/cli/authentication">Databricks Docs: Databricks CLI authentication</a></li><li><a href="https://docs.databricks.com/aws/en/mlflow3/genai/tracing/">Databricks Docs: MLflow Tracing for GenAI</a></li><li><a href="https://docs.databricks.com/aws/en/getting-started/free-edition">Databricks Docs: Free Edition</a></li><li><a href="https://medium.com/lab-notes/agent-bricks-cli-beta-on-databricks-free-edition-a-live-agent-in-188-seconds-72839cbca91b">Lab Notes: Agent Bricks CLI Beta on Databricks Free Edition</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11d80deefa6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Claude Haiku 5.5 vs Sonnet 5.5 vs Opus 5.5: Which One Should You Use?]]></title>
            <link>https://medium.com/@sudarshan-koirala/claude-haiku-5-5-vs-sonnet-5-5-vs-opus-5-5-which-one-should-you-use-8851af8a06af?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/8851af8a06af</guid>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[claude-opus]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 11:01:10 GMT</pubDate>
            <atom:updated>2026-10-08T11:01:10.851Z</atom:updated>
            <content:encoded><![CDATA[<p>Three models, one family, a 40x price gap. Here is how to pick the right one.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wju1FafOsxXUm1fo8EglGQ.png" /></figure><p>Anthropic just released <strong>Claude Haiku 5.5</strong> (October 7, 2026). With it, the Claude 5.5 family is complete: <strong>Opus 5.5</strong>, <strong>Sonnet 5.5</strong> and <strong>Haiku 5.5</strong>.</p><p>Short answer first:</p><ul><li><strong>Haiku 5.5</strong> -&gt; fast, cheap, repetitive work (classify, extract, route, summarize, subagents).</li><li><strong>Sonnet 5.5</strong> -&gt; your everyday default for coding, documents and agents.</li><li><strong>Opus 5.5</strong> -&gt; long, complex work where judgment matters most.</li></ul><p>The rest of this post explains why, with real numbers and a simple way to choose.</p><h3>The Claude 5.5 Family at a Glance</h3><p>All three models share the same base: a <strong>1M token context window</strong>, <strong>128K max output</strong>, <strong>adaptive thinking</strong>, image input, and a <strong>June 2026 knowledge cutoff</strong>. The difference is speed, price and how deep they think.</p><p><strong>Claude Opus 5.5</strong> (claude-opus-5-5), released September 22, 2026</p><ul><li>Price: $4 input / $20 output per million tokens</li><li>Speed: moderate</li><li>Default effort: medium (thinking is always on)</li><li>Built for: long-running agentic coding, migrations, research and complex knowledge work</li></ul><p><strong>Claude Sonnet 5.5</strong> (claude-sonnet-5-5), released September 28, 2026</p><ul><li>Price: $2 input / $10 output per million tokens</li><li>Speed: fast</li><li>Default effort: high</li><li>Built for: everyday coding, bug fixes, documents, slides, spreadsheets, agent tool use</li></ul><p><strong>Claude Haiku 5.5</strong> (claude-haiku-5-5), released October 7, 2026</p><ul><li>Price: $0.10 input / $0.50 output per million tokens (prompts up to 100K tokens)</li><li>Price: $0.50 input / $2.50 output per million tokens (prompts over 100K tokens)</li><li>Speed: fastest</li><li>Default effort: medium</li><li>Built for: classification, extraction, routing, summaries, subagents, live support, browser use</li></ul><p>Think of it like a restaurant kitchen. <strong>Opus</strong> is the head chef who designs the menu. <strong>Sonnet</strong> is the line cook who makes most of the dishes. <strong>Haiku</strong> is the prep team that chops a thousand onions without complaining.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9003ZRu57ME4MvMbiy2Sw.png" /></figure><h3>What is New in Haiku 5.5?</h3><p>Haiku 5.5 is the big news this week. A few things stand out:</p><ul><li><strong>Price drop.</strong> It is about 90% cheaper than Haiku 4.5 for prompts up to 100K tokens. Anthropic says it costs around 75% less to run on average.</li><li><strong>1M context.</strong> Up from 200K on Haiku 4.5. Max output goes from 64K to 128K.</li><li><strong>Effort control.</strong> It is the first Haiku with an adjustable effort setting, so you can trade cost against intelligence.</li><li><strong>Huge jump on hard tasks.</strong> On Terminal-Bench 4.0 (agentic coding in a terminal), Haiku 4.5 scored 0.0%. Haiku 5.5 scores 39.2%.</li><li><strong>Browser and computer use.</strong> Anthropic positions it as a strong fit for browser use because it is fast and cheap.</li></ul><p>One thing to watch: Haiku 5.5 uses the newer tokenizer, so the same text counts as roughly <strong>30% more tokens</strong> than on Haiku 4.5. Recount your prompts before you estimate costs.</p><h3>The Numbers That Matter</h3><p>All scores below are <strong>vendor-reported by Anthropic</strong> in the launch posts. Treat them as a starting point, not the final word. Your own evals matter more.</p><p><strong>Terminal-Bench 4.0 (agentic coding)</strong></p><ul><li>Haiku 5.5: 39.2%</li><li>Sonnet 5.5: 70.6%</li><li>Opus 5.5: 66.4% (at xhigh effort)</li></ul><p><strong>GDPval-AA v2.1 (real-world work across 44 occupations, Elo)</strong></p><ul><li>Haiku 5.5: 1620</li><li>Sonnet 5.5: 1844</li><li>Opus 5.5: 1846</li></ul><p><strong>Humanity’s Last Exam (with tools)</strong></p><ul><li>Haiku 5.5: 57.4%</li><li>Sonnet 5.5: 64.5%</li><li>Opus 5.5: 67.7%</li></ul><p>Surprised that Sonnet beats Opus on Terminal-Bench? So was I. But Anthropic says clearly that <strong>Opus 5.5 remains stronger at complex, open-ended work that needs sustained judgment</strong>. Benchmarks measure one slice. A multi-hour migration across six repos is a different game.</p><h3>The Price Gap is the Real Story</h3><p>Let us say your app sends <strong>1 million input tokens and 200K output tokens</strong> per day (a small support bot, for example), with prompts under 100K tokens:</p><ul><li><strong>Opus 5.5:</strong> $4 + $4 = <strong>$8 per day</strong></li><li><strong>Sonnet 5.5:</strong> $2 + $2 = <strong>$4 per day</strong></li><li><strong>Haiku 5.5:</strong> $0.10 + $0.10 = <strong>$0.20 per day</strong></li></ul><p>Same workload. Opus costs <strong>40x more</strong> than Haiku per token. If Haiku is good enough for the task, that is money you keep.</p><p>Two more price notes from this week:</p><ul><li>Anthropic <strong>halved Sonnet 5.5 cache reads</strong> to $0.10 per million tokens. They say this makes Sonnet around 20% cheaper on most agentic work.</li><li>Haiku 5.5 cache reads are just <strong>$0.01</strong> per million tokens (under 100K prompts).</li></ul><h3>How to Choose: 4 Simple Questions</h3><p>Ask these in order. Stop at the first “yes”.</p><ol><li><strong>Is the task short, repetitive and high volume?</strong> (tagging tickets, pulling fields from documents, routing requests, summarizing chats) -&gt; <strong>Haiku 5.5</strong></li><li><strong>Does it need to respond in real time?</strong> (live support, browser automation, quick lookups) -&gt; <strong>Haiku 5.5</strong></li><li><strong>Is it everyday work?</strong> (write a feature, fix a bug, create a report, build slides) -&gt; <strong>Sonnet 5.5</strong></li><li><strong>Is it long, messy or high stakes?</strong> (codebase-wide refactors, multi-hour agents, deep research, financial analysis) -&gt; <strong>Opus 5.5</strong></li></ol><p>Still not sure? Anthropic’s own guide gives two starting points:</p><ul><li><strong>Efficiency-first:</strong> start with Haiku 5.5, test, and upgrade only if it falls short.</li><li><strong>Capability-first:</strong> start with Opus 5.5, then lower the effort or move down a model once the workflow is stable.</li></ul><p>And if even Opus at xhigh or max effort is not enough, Anthropic points you to <strong>Claude Fable 5.1</strong>, which sits above the 5.5 family at $10 / $50 per million tokens.</p><h3>Better Together: Use All Three</h3><p>The smartest setup is often not one model. It is a team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJ-mKR3ZGgpn-rfvuIFJpQ.png" /></figure><ul><li><strong>Opus 5.5 plans.</strong> It breaks a big job into steps.</li><li><strong>Sonnet 5.5 builds.</strong> It writes the code or the document.</li><li><strong>Haiku 5.5 does the small jobs.</strong> Lookups, summaries, extraction, checks.</li></ul><p>This is not just theory. In the Haiku 5.5 launch post, Rogo describes a bigger model building a slide deck while a Haiku 5.5 subagent pulls the exact revenue line from a 10-K. Cognition reports using Haiku 5.5 as a sidekick to Opus 5.5 in Devin to cut cost and latency.</p><h3>Switching Models in Code</h3><p>The model ID is the only thing you change. You can also set the effort level per request.</p><pre>import anthropic</pre><pre>client = anthropic.Anthropic()</pre><pre>def ask(prompt: str, task: str = &quot;everyday&quot;):<br>    model = {<br>        &quot;simple&quot;: &quot;claude-haiku-5-5&quot;,    # classify, extract, route<br>        &quot;everyday&quot;: &quot;claude-sonnet-5-5&quot;,  # code, docs, agents<br>        &quot;complex&quot;: &quot;claude-opus-5-5&quot;,     # long, high-stakes work<br>    }[task]</pre><pre>    response = client.messages.create(<br>        model=model,<br>        max_tokens=4096,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        output_config={&quot;effort&quot;: &quot;medium&quot;},<br>    )<br>    return next(b.text for b in response.content if b.type == &quot;text&quot;)</pre><pre>print(ask(&quot;Classify this ticket: &#39;My invoice is wrong&#39;&quot;, task=&quot;simple&quot;))</pre><p>A few gotchas when you migrate to Haiku 5.5 from Haiku 4.5:</p><ul><li><strong>Remove </strong><strong>temperature, </strong><strong>top_p and </strong><strong>top_k.</strong> Non-default values now return a 400 error.</li><li><strong>No assistant prefill.</strong> End your messages with a user turn.</li><li><strong>Read content blocks by </strong><strong>type.</strong> A response can start with a thinking block, so do not assume content[0] is the answer.</li><li><strong>Replace </strong><strong>budget_tokens</strong> with adaptive thinking and the effort setting.</li></ul><p>In Claude Code, switching is one flag:</p><pre>claude --model haiku<br>claude --model sonnet<br>claude --model opus</pre><h3>My Take</h3><p>For most builders, the new default looks like this: <strong>Sonnet 5.5 for the main work, Haiku 5.5 for everything small and repetitive, Opus 5.5 when the task is long or the stakes are high.</strong></p><p>Haiku 5.5 is the one that changes the math. Jobs that were too expensive to run at scale last month are now cheap enough to just do.</p><p>Run your own evals. Benchmarks help you shortlist. Your data decides.</p><h4>I Have Made This Post for All Readers. Not Under the Paid Wall.</h4><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://www.anthropic.com/claude-haiku-5-5">Introducing Claude Haiku 5.5 (Anthropic)</a></li><li><a href="https://www.anthropic.com/claude-sonnet-5-5">Introducing Claude Sonnet 5.5 (Anthropic)</a></li><li><a href="https://www.anthropic.com/claude-opus-5-5">Introducing Claude Opus 5.5 (Anthropic)</a></li><li><a href="https://platform.claude.com/docs/en/models/overview">Claude Models Overview (Claude Platform Docs)</a></li><li><a href="https://platform.claude.com/docs/en/about-claude/models/choosing-a-model">Choosing the Right Model (Claude Platform Docs)</a></li><li><a href="https://platform.claude.com/docs/en/models/haiku-5-5/overview">Claude Haiku 5.5 Overview and Pricing (Claude Platform Docs)</a></li><li><a href="https://platform.claude.com/docs/en/models/haiku-5-5/whats-new-haiku-5-5">What’s New in Claude Haiku 5.5 (Claude Platform Docs)</a></li><li><a href="https://platform.claude.com/docs/en/build-with-claude/effort">Effort Parameter (Claude Platform Docs)</a></li><li><a href="https://platform.claude.com/docs/en/about-claude/pricing">Claude API Pricing (Claude Platform Docs)</a></li><li><a href="https://codersera.com/blog/claude-sonnet-5-5-vs-opus-5-5-2026/amp/">Codersera: Claude Sonnet 5.5 vs Opus 5.5</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8851af8a06af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your First AI Agent in Claude (No Code)]]></title>
            <link>https://medium.com/@sudarshan-koirala/build-your-first-ai-agent-in-claude-no-code-176e3ece9584?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/176e3ece9584</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[no-code]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[small-business]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 11:06:01 GMT</pubDate>
            <atom:updated>2026-10-06T11:06:01.609Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>A morning manager for a small bakery: it reads your files, finds patterns and drafts emails while you sleep</strong></p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UBgPeiPdD611Q39jTT2KfA.png" /></figure><p>This post takes you to the video where I build a real AI agent inside Claude, from zero. No code. No servers. Only the Claude website, a Google Drive folder and a Gmail account.</p><p>The example is <strong>Sunrise Bakery</strong>, a small (fictional) bakery. At 6:30 AM, the bakery is still asleep. The agent wakes up, reads the sales and the stock, finds problems and writes draft emails to suppliers. Maya, the owner, reads everything with her morning coffee.</p><h3>Chatbot vs Agent: What Is the Difference?</h3><p>A <strong>chatbot</strong> answers when you ask. Then it stops. Tomorrow, you must ask again.</p><p>An <strong>agent</strong> does work for you. It runs on a schedule, reads your own data, uses tools and gives you a result to check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1UKSYjBe1wRfN74tW4RewA.png" /></figure><h3>The 7 Ingredients of the Agent</h3><p>In Claude, you do not need a framework. You combine features that already exist:</p><ul><li><strong>Brain</strong> -&gt; the Claude model (Opus or Sonnet)</li><li><strong>Job description</strong> -&gt; the project instructions</li><li><strong>Tools</strong> -&gt; the Google Drive and Gmail connectors</li><li><strong>Knowledge</strong> -&gt; 4 bakery files in Drive: sales log, inventory, this week, handbook</li><li><strong>Skill</strong> -&gt; a bakery-morning-report skill that tells the agent the steps</li><li><strong>Trigger</strong> -&gt; a scheduled task that runs every morning</li><li><strong>Guardrails</strong> -&gt; “Drafts only. Never send email.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nx8ZJYFzHmCFr2WPGD8yyA.png" /></figure><p>Six of the seven ingredients work on the Free plan. Only the <strong>trigger</strong> (scheduled tasks) needs a paid plan such as Pro.</p><h3>What the Agent Found on Its First Run</h3><p>I gave the agent one prompt: <em>“Run the morning report for today. If anything needs reordering, create a Gmail draft to the supplier.”</em></p><p>It loaded the skill by itself, read the 4 files and came back with:</p><ul><li><strong>Waste alert:</strong> bake about 24 sourdough loaves on Monday, not 40. On the last two Mondays, 18 of 40 went to waste.</li><li><strong>Stock alert:</strong> almond flour is at 0 kg (minimum is 5 kg). It wrote a Gmail draft to the supplier.</li><li><strong>Special order:</strong> 80 extra butter croissants for a Saturday wedding. Count the butter first.</li><li><strong>Smart check:</strong> on the second run, it saw the draft was already in Gmail and did not make a copy.</li></ul><p>It <strong>did not send</strong> anything. That is the guardrail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UzYsIm0GVbVb2-wjMwNdNw.png" /></figure><h3>What I Demonstrated in the Video</h3><ol><li>Create a <strong>Project</strong> and write the job description (instructions)</li><li>Put the 4 files in a <strong>Sunrise Bakery</strong> Google Drive folder</li><li>Connect the <strong>Google Drive</strong> and <strong>Gmail</strong> connectors</li><li>Upload a custom <strong>skill</strong> as a ZIP file (a folder with a SKILL.md)</li><li>Run the morning report by hand and check the Gmail draft</li><li>Create a <strong>scheduled task</strong> for every morning and test it with <strong>Run now</strong></li></ol><p>The sample files and the skill ZIP are in the GitHub repository. The link is in the video description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cz-SYzOFl_KbQXRb89ODzQ.gif" /></figure><h3>A Few Honest Notes</h3><ul><li><strong>Keep the files online.</strong> The agent reads the latest version from Drive every time. Local files need your computer to be on.</li><li><strong>Scheduled times can shift a few minutes.</strong> The server spreads the load, so 6:30 can become 6:38.</li><li><strong>Start with drafts.</strong> AI can make mistakes. When money and orders are involved, review first. Send automatically only when you trust the results.</li><li><strong>Write good instructions.</strong> Iterate on them. The job description is the most important part.</li></ul><p>This is not only for bakeries. Any small business with a daily routine (stock, orders, bookings, reports) can use the same 7 ingredients.</p><p>I have made this post for all readers. Not under the paid wall.</p><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><p>No more text here -&gt; this is the content covered in the video 👇</p><p>Link to video 👇 Happy learning 😎</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FS7Z6b8iLNs8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DS7Z6b8iLNs8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FS7Z6b8iLNs8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/072722df856926515023182c85cbafba/href">https://medium.com/media/072722df856926515023182c85cbafba/href</a></iframe><h3>Sources</h3><ol><li><a href="https://support.claude.com/en/articles/9519177-how-can-i-create-and-manage-projects">Claude Help Center: How to create and manage projects</a></li><li><a href="https://support.claude.com/en/articles/10185728-understanding-claude-s-personalization-features">Claude Help Center: Understanding Claude’s personalization features</a></li><li><a href="https://support.claude.com/en/articles/11176164-use-connectors-to-extend-claude-s-capabilities">Claude Help Center: Use connectors to extend Claude’s capabilities</a></li><li><a href="https://support.claude.com/en/articles/10166901-use-google-workspace-connectors">Claude Help Center: Use Google Workspace connectors</a></li><li><a href="https://claude.com/docs/connectors/google/drive">Claude Docs: Google Drive connector</a></li><li><a href="https://support.claude.com/en/articles/12512180-use-skills-in-claude">Claude Help Center: Use skills in Claude</a></li><li><a href="https://support.claude.com/en/articles/13854387-schedule-recurring-tasks-in-claude-cowork">Claude Help Center: Schedule recurring tasks</a></li><li><a href="https://support.claude.com/en/articles/13345190-get-started-with-claude-cowork">Claude Help Center: Get started with Claude Cowork</a></li><li><a href="https://academy.claude.com/courses/claude-101/introduction-to-projects">Claude Academy: Introduction to projects</a></li><li><a href="https://academy.claude.com/courses/introduction-to-claude-cowork/scheduled-tasks">Claude Academy: Scheduled tasks</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=176e3ece9584" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LlamaParse Deal Checker: Find Red Flags in Contracts with Parse, Extract and Python Rules]]></title>
            <link>https://medium.com/@sudarshan-koirala/llamaparse-deal-checker-find-red-flags-in-contracts-with-parse-extract-and-python-rules-ab238f0092aa?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab238f0092aa</guid>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[parse-contracts]]></category>
            <category><![CDATA[llamaparse]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Sun, 27 Sep 2026 18:56:43 GMT</pubDate>
            <atom:updated>2026-09-27T18:56:43.593Z</atom:updated>
            <content:encoded><![CDATA[<p>Drop contracts in a folder. Run one command. Get the red flags and the page they come from.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BS_mzC9LLGIeqVrc6-scXQ.png" /></figure><p>Let’s say you are a content creator. Brands send you sponsorship deals. Each deal comes with a contract, and each contract has terms you must read before you say yes: payment dates, exclusivity, revision rounds. Most people do not read them properly.</p><p><strong>Deal Checker</strong> is a small tool that does this for you. You put the contracts in a folder, you run one command, and it tells you which terms to push back on, with the exact page where it found them.</p><p><a href="https://github.com/sudarshan-koirala/deal-checker">GitHub - sudarshan-koirala/deal-checker</a></p><p>It uses two parts of <strong>LlamaParse</strong>: <strong>Parse</strong> to read the documents and <strong>Extract</strong> to pull out the deal terms. Then plain Python rules decide what is a red flag. The same pattern works for invoices, leases and almost any other document.</p><p>If you prefer, link to video 👇 Happy Learning 😎</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkwxkfkL3tHQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkwxkfkL3tHQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FkwxkfkL3tHQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d90f3c6a25728181b75873877f94529/href">https://medium.com/media/2d90f3c6a25728181b75873877f94529/href</a></iframe><h3>How It Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAdedAo2voXcfBQON1wFTQ.png" /></figure><p>The flow has five steps:</p><ol><li><strong>Contracts</strong>: PDFs, scans or photos go into sample_contracts/.</li><li><strong>Parse</strong>: LlamaParse turns each file into text and keeps the page numbers.</li><li><strong>Extract</strong>: LlamaParse Extract reads the parsed text and fills a Pydantic schema with the deal terms.</li><li><strong>Rules</strong>: Plain Python checks the terms against your limits.</li><li><strong>Report</strong>: You get deal_report.md and deals.csv in the reports/ folder.</li></ol><p>The key idea: <strong>AI reads the contract. Your rules decide the red flags.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kEphtQMOByS3aGOQesRHzg.gif" /></figure><h3>Setup in 3 Minutes</h3><p>You need <a href="https://docs.astral.sh/uv/">uv</a> installed. Then clone the repo and run:</p><pre>uv sync<br>cp .env.example .env</pre><p>Next, create a free account on LlamaParse (you can choose the global or the EU region). Go to <strong>API Keys</strong>, create a key and paste it into .env. The free plan gives you 10,000 credits per month. If you use the EU region, uncomment the EU base URL line in .env too.</p><p>That is all the setup.</p><h3>schema.py: Tell Extract What to Find</h3><p>This file is the most important one. You describe the fields you want, and <strong>every description is an instruction to the extraction model</strong>. Clear descriptions give better results.</p><p>Here is a simplified version of the idea:</p><pre>from pydantic import BaseModel, Field</pre><pre>class DealTerms(BaseModel):<br>    brand_name: str = Field(description=&quot;Name of the brand that sponsors the deal&quot;)<br>    fee_usd: float | None = Field(None, description=&quot;Total fee in USD&quot;)<br>    payment_days: int | None = Field(None, description=&quot;Days after delivery until payment is due&quot;)<br>    exclusivity_days: int | None = Field(None, description=&quot;Days the creator cannot work with competitors&quot;)<br>    revision_rounds: int | None = Field(None, description=&quot;Number of revision rounds. Use -1 if unlimited&quot;)</pre><p>Want to check a different document type? Change the fields. The rest of the pipeline stays the same.</p><h3>rules.py: Keep the Decision in Plain Python</h3><p>AI can give different answers on different runs. Rules do not. So the red flag logic lives in normal Python:</p><pre>MAX_PAYMENT_DAYS = 60<br>MAX_EXCLUSIVITY_DAYS = 90<br>MAX_REVISION_ROUNDS = 2</pre><pre>def check(terms):<br>    flags = []<br>    if terms.payment_days and terms.payment_days &gt; MAX_PAYMENT_DAYS:<br>        flags.append(&quot;Payment takes too long&quot;)<br>    if terms.exclusivity_days and terms.exclusivity_days &gt; MAX_EXCLUSIVITY_DAYS:<br>        flags.append(&quot;Exclusivity is too long&quot;)<br>    if terms.revision_rounds == -1 or (terms.revision_rounds or 0) &gt; MAX_REVISION_ROUNDS:<br>        flags.append(&quot;Too many revision rounds&quot;)<br>    return flags</pre><p>Change the limits to match your own deals. In the real project, each flag also carries the page citation from Extract.</p><h3>Pick a Tier and Run It</h3><p>LlamaParse Parse has four tiers. Parse-only credits per page:</p><ul><li><strong>Fast</strong>: 1 credit (spatial text only, no markdown)</li><li><strong>Cost-effective</strong>: 3 credits</li><li><strong>Agentic</strong>: 10 credits (the default in this project)</li><li><strong>Agentic Plus</strong>: 45 credits (for the hardest documents)</li></ul><p>Extract is billed on top of Parse. The CLI in this project shows a combined estimate before it runs: about 8 credits per page for cost-effective, 25 for agentic and 60 for agentic plus. Check the official pricing page before large runs, because prices can change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bf991VY2eXpzeewOahGfIw.png" /></figure><p>Run the checker:</p><pre>uv run deal-checker</pre><p>In the video, I ran three sample contracts (1, 2 and 32 pages, about 35 pages in total) on Agentic Plus. The CLI estimated 2,100 credits. It parsed and extracted each file, then printed the red flags in the terminal with page numbers such as page 13 and page 31. One contract had no red flags at all.</p><p>The report shows the brand, fee, pages, payment terms and red flags for each contract, plus an explanation of each flag. Now you know what to discuss with each brand before you sign.</p><p>Note: the sample contracts are fake documents made with AI for the demo.</p><h3>Bonus 1: Give Your Coding Agent LlamaParse Skills</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JQ3CPC4xO-adSUsbY5981A.png" /></figure><p>If you already use Claude Code, Cursor or another coding agent, you can skip the SDK and install the LlamaParse agent skills:</p><pre>npx skills add run-llama/llamaparse-agent-skills</pre><p>To install only one skill, add --skill llamaparse or --skill liteparse. The installer asks which skills and which agents you want, then puts them in your project.</p><ul><li><strong>llamaparse</strong>: cloud parsing, needs your API key.</li><li><strong>liteparse</strong>: local parsing, no API key.</li><li><strong>LlamaCloud Index</strong>: search documents in a LlamaCloud index.</li></ul><p>Then just ask your agent, for example: <em>“Parse brew_scan.pdf with LiteParse.”</em> The agent reads the skill, runs OCR on the scanned file and gives you the content.</p><h3>Bonus 2: Serve the Workflow as an API</h3><p>The project’s pyproject.toml lets llamactl serve the same workflow as an API:</p><pre>uvx llamactl serve</pre><p>This starts a local server with a FastAPI Swagger UI, where you can call the workflow with GET and POST endpoints. It runs on your machine and uses your LlamaParse API key.</p><p>The repo also has a GitHub Actions workflow that runs pytest, so you have an end-to-end project to start from.</p><h3>Key Takeaways</h3><ul><li><strong>Parse</strong> reads messy documents. <strong>Extract</strong> turns them into typed fields.</li><li><strong>Good field descriptions</strong> in your schema give better extraction.</li><li><strong>Plain Python rules</strong> make the final decision repeatable.</li><li><strong>Start with a cheaper tier</strong>, move up only when your documents need it.</li><li><strong>The same pattern works</strong> for invoices, leases and other documents.</li></ul><h4>I Have Made This Post for All Readers. Not Under the Paid Wall.</h4><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://developers.llamaindex.ai/llamaparse/general/pricing/">LlamaParse Pricing</a></li><li><a href="https://developers.llamaindex.ai/llamaparse/parse/guides/tiers/">LlamaParse Tiers</a></li><li><a href="https://developers.llamaindex.ai/llamaparse/general/faq/">LlamaParse FAQ</a></li><li><a href="https://developers.llamaindex.ai/llamaparse/for-agents/skills/">LlamaParse Skills and Plugins</a></li><li><a href="https://github.com/run-llama/llamaparse-agent-skills">llamaparse-agent-skills on GitHub</a></li><li><a href="https://github.com/run-llama/liteparse">LiteParse on GitHub</a></li><li><a href="https://developers.llamaindex.ai/liteparse/guides/agent-skill/">LiteParse Agent Skill Guide</a></li><li><a href="https://developers.llamaindex.ai/python/llamaagents/llamactl/getting-started/">Getting Started with llamactl</a></li><li><a href="https://developers.llamaindex.ai/python/llamaagents/llamactl/workflow-api/">Serving your Workflows with LlamaAgents</a></li><li><a href="https://www.llamaindex.ai/blog/beyond-raw-text-how-llamaparse-and-liteparse-give-agents-real-document-understanding">LlamaIndex Blog: How LlamaParse and LiteParse Give Agents Real Document Understanding</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab238f0092aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Agent Skills: Teach Your Coding Agent How Databricks Works]]></title>
            <link>https://medium.com/@sudarshan-koirala/databricks-agent-skills-teach-your-coding-agent-how-databricks-works-c6ffa320ef52?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6ffa320ef52</guid>
            <category><![CDATA[databricksaitools]]></category>
            <category><![CDATA[agent-skills]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-genie-code]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 12:01:02 GMT</pubDate>
            <atom:updated>2026-09-14T12:01:02.393Z</atom:updated>
            <content:encoded><![CDATA[<p>One CLI command, 29 skills, and your own rules on top.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFcBVllNzy1vgpV9kcS6LA.png" /></figure><p>I get asked about Databricks skills a lot. So I made a full video about it.</p><p>Databricks moved the skills out of the AI Dev Kit and put them into the Databricks CLI itself. This is now the official path. In this post I explain what a skill is, how to install the skills, how to write your own, and what skills cannot do.</p><h3>What is an agent skill?</h3><p>A skill is one Markdown file called SKILL.md. That is it. Do not make it more complicated than that.</p><p>The file has two parts. On the top there is the metadata: a name and a description. Below that you write, in plain English, what you want your AI coding assistant to do.</p><p>A skill is knowledge, not action. It does not do the task. It tells your coding assistant how to do the task the way your team does it.</p><p>The name and the description are the most important lines in the file. The agent reads only the names and the descriptions of all installed skills. When your request matches one description, the agent opens that one file. The other skills stay closed, so your context window stays small.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4tyjel27MQWTwrShyMuEMg.png" /></figure><h3>Install the skills</h3><p>First, install the Databricks CLI. You need version 1.0.0 or later, because that version ships the aitools command group. On macOS you may also need sudo for the curl install.</p><p>Then log in to your workspace:</p><pre>databricks auth login --host https://your-workspace-url --profile my-profile</pre><p>Now install the skills:</p><pre>databricks aitools install</pre><p>The CLI asks where to install, global or project. Then it detects the coding agents on your machine and shows you a list. Supported agents today are Claude Code, Cursor, Codex CLI, OpenCode, GitHub Copilot, Antigravity, Pi, and Gemini CLI.</p><p>The install path is not the same for every agent. Claude Code, Codex CLI, and GitHub Copilot get the Databricks plugin. The others get raw skill files. Add --skills-only if you want raw skill files everywhere.</p><p>You can also skip the prompts:</p><pre># only one agent<br>databricks aitools install --agents claude-code</pre><pre># install into the current project, not globally<br>databricks aitools install --scope project</pre><pre># only the skills you want<br>databricks aitools install --skills bundles,sql</pre><p>In my video the install gave me 29 stable skills. That number will grow, so check databricks aitools list for your own count.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*1P3WqzT1R1U5hDIkqXMSeg.gif" /></figure><h3>What I showed in the video</h3><ol><li>What a skill is, with a look inside a real SKILL.md file</li><li>Install of the Databricks CLI and the skills, with the agent detection step</li><li>The same install done from inside Claude Code with the plugin marketplace</li><li>A gold table request without a custom skill, so you see the agent guess</li><li>The same request with my own skill, so you see it follow my rules</li><li>The same skill used inside Databricks Genie Code</li><li>The commands that keep you in control</li></ol><h3>Write your own skill</h3><p>The built-in skills know Databricks. They do not know your naming rules. That is your job.</p><p>Make a folder, add one file, and keep the file name as SKILL.md:</p><pre>mkdir -p ~/.claude/skills/meridian-standards<br>cd ~/.claude/skills/meridian-standards</pre><p>Here is the shape of the file:</p><pre>---<br>name: meridian-standards<br>description: Naming, comment and tag standards for Meridian data. Use when<br>  creating or changing catalogs, schemas or tables in Databricks.<br>parent: databricks-core<br>---</pre><pre># Meridian data standards</pre><pre>## Catalogs and schemas<br>- Dev catalog is meridian_dev, prod catalog is meridian_prod<br>- bronze is raw data, silver is cleaned data, gold is business ready data</pre><pre>## Tables<br>- Use snake_case names, for example churn_by_region<br>- Every gold table needs a comment that says what one row is<br>- Every gold table needs a domain tag and an owner tag</pre><p>The parent line points to a bigger skill. My skill points to the core Databricks skill, so authentication and CLI basics come for free.</p><p>Then I asked Claude Code the same question twice: <em>“Create a gold table that shows the churn rate by region.”</em></p><p>Without my skill, the agent used a discovery skill, found the tables, and made its own choices. The result was fine, but it was not my standard.</p><p>With my skill, it used my catalog, my schema, my table name, my comment, and my tags. Same prompt, better answer, every time.</p><p>One tip from the video: write the descriptions with real words your team uses. That is how the agent finds the right skill.</p><p>SKILL.md and data in this github repo: <a href="https://github.com/sudarshan-koirala/databricks-aitools-youtube">https://github.com/sudarshan-koirala/databricks-aitools-youtube</a></p><h3>The same skill inside Genie Code</h3><p>You do not need a local editor for this. Genie Code in the Databricks workspace reads the same file format.</p><p>Put your skill folder here:</p><ul><li>For you only: /Users/{your-email}/.assistant/skills/</li><li>For everybody in the workspace: Workspace/.assistant/skills/</li></ul><p>Each skill needs its own folder with a SKILL.md inside. Genie Code picks it up in your next chat. You can also call a skill directly with an @ mention. If you edit a skill, start a new chat, because edits do not apply to an open chat.</p><p>In my test, Genie Code read my skill, created the same gold table, and then wrote an even better table comment when I asked for one.</p><h3>What skills cannot do</h3><p>This part matters.</p><p>Skills give knowledge. MCP servers give tools. A skill tells the agent how to work, an MCP server lets the agent do the work against your data. Installing the skills does not set up any MCP server for you.</p><p>So if you want your agent to write correct Databricks code <em>and</em> read live workspace data, you need both.</p><h3>Commands to stay in control</h3><pre># what is installed right now<br>databricks aitools list</pre><pre># preview updates without downloading<br>databricks aitools update --check</pre><pre># update, and pick up new skills from the manifest<br>databricks aitools update</pre><pre># add the experimental skills<br>databricks aitools install --experimental</pre><pre># skill version, and clean removal<br>databricks aitools version<br>databricks aitools uninstall</pre><p>One more thing. If you used the AI Dev Kit before, uninstall those skills first. The Dev Kit skills are now deprecated and the same skills come from the official repository. Two copies will fight each other.</p><h4>I Have Made This Post for All Readers. Not Under the Paid Wall.</h4><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://docs.databricks.com/aws/en/agent-skills/">Agent skills for AI coding assistants, Databricks docs</a></li><li><a href="https://docs.databricks.com/aws/en/dev-tools/cli/reference/aitools-commands">aitools command group, Databricks CLI reference</a></li><li><a href="https://docs.databricks.com/aws/en/genie-code/skills">Extend Genie Code with agent skills</a></li><li><a href="https://github.com/databricks/databricks-agent-skills">databricks/databricks-agent-skills on GitHub</a></li><li><a href="https://github.com/databricks-solutions/ai-dev-kit">Databricks AI Dev Kit on GitHub</a></li><li><a href="https://docs.databricks.com/aws/en/dev-tools/cli">Databricks CLI documentation</a></li><li><a href="https://agentskills.io/">Agent Skills open standard</a></li><li><a href="https://agentskills.io/specification">Agent Skills specification</a></li><li><a href="https://developers.databricks.com/docs/tools/ai-tools/agent-skills">Agent skills, Databricks Developers</a></li><li><a href="https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices">Anthropic Agent Skills best practices</a></li></ol><p>No more text here -&gt; this is the content covered in the video 👇</p><p>Link to video 👇 Happy learning 😎</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMW04lIKPHCM&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMW04lIKPHCM" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/35ca5d01f5b9393fd7da6c5d52897e7a/href">https://medium.com/media/35ca5d01f5b9393fd7da6c5d52897e7a/href</a></iframe><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6ffa320ef52" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PDD, SDD and OID for Agentic AI: The Four Documents That Get an Agent Into Production]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@sudarshan-koirala/pdd-sdd-and-oid-for-agentic-ai-the-four-documents-that-get-an-agent-into-production-7176eedcea58?source=rss-55cd50a9c816------2"><img src="https://cdn-images-1.medium.com/max/1672/1*ZnzRCtUTuLrRi-rpI9YejA.png" width="1672"></a></p><p class="medium-feed-snippet">The RPA document chain still works for AI agents. What goes inside each document changes a lot.</p><p class="medium-feed-link"><a href="https://medium.com/@sudarshan-koirala/pdd-sdd-and-oid-for-agentic-ai-the-four-documents-that-get-an-agent-into-production-7176eedcea58?source=rss-55cd50a9c816------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sudarshan-koirala/pdd-sdd-and-oid-for-agentic-ai-the-four-documents-that-get-an-agent-into-production-7176eedcea58?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/7176eedcea58</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[oid]]></category>
            <category><![CDATA[enterprise-ai]]></category>
            <category><![CDATA[process-design-document]]></category>
            <category><![CDATA[solution-design-document]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 07:30:31 GMT</pubDate>
            <atom:updated>2026-08-29T07:30:31.294Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Graph Engineering for AI Coding Agents: Nodes, Edges, and Stop Conditions]]></title>
            <link>https://medium.com/@sudarshan-koirala/graph-engineering-for-ai-coding-agents-nodes-edges-and-stop-conditions-e605634cb029?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/e605634cb029</guid>
            <category><![CDATA[langgrpah]]></category>
            <category><![CDATA[graph-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[agentic-coding]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Fri, 21 Aug 2026 06:01:01 GMT</pubDate>
            <atom:updated>2026-08-21T06:01:01.478Z</atom:updated>
            <content:encoded><![CDATA[<p>Your agent doesn’t need a longer prompt. It needs a map.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqp36hl9Dq7pkMldjJPAeg.png" /></figure><p>If you’ve spent real time running coding agents on non-trivial tasks, you already know the failure mode. You give Claude Code or Codex a meaty task, it runs for a while, and somewhere in the middle it quietly loses the plot. It declares victory before actually checking its work. It “fixes” a failing test by deleting it. It forgets a constraint you mentioned twenty minutes ago. None of that happens because the model got worse mid-session. It happens because the plan, the checks, and the decision about what to do next were all hidden inside one long, ever-growing context window, with nothing forcing them to stay explicit.</p><p>2026 has been the year the industry stopped blaming the model for this and started fixing the system around it. First came loop engineering: designing the repeated plan, act, verify cycle an agent runs instead of firing off one-shot prompts. Then harness engineering: the permissions, progress files, and git checkpoints that keep an agent oriented across long sessions. The newest layer on top of both is graph engineering, and it answers a question neither of the earlier two fully solves on its own: when a task has more than one moving part, who decides what happens next, and how do you prove it happened correctly?</p><p>AI agents are moving fast. ⚡</p><p>I’m tracking what actually matters in <strong>The Agent Brief, </strong>a weekly newsletter in Linkedin.</p><p>Practical updates. Key trends. Less noise.</p><p>Subscribe 👇<br><a href="https://www.linkedin.com/newsletters/the-agent-brief-7482745760560697344/">https://www.linkedin.com/newsletters/the-agent-brief-7482745760560697344/</a></p><h3>What Is Graph Engineering?</h3><p>Graph engineering is the practice of representing an AI workflow as an explicit graph instead of a single agent looping on its own judgment. Every unit of work becomes a <strong>node</strong> with one job. Every transition between units becomes an <strong>edge</strong> with an explicit rule attached to it. Information that needs to survive from one node to the next travels as <strong>shared state</strong>, instead of relying on the model to remember it correctly. And before the workflow is allowed to call itself finished, it usually has to pass through a <strong>gate</strong>: a test suite, a validator, or a human.</p><p>The shift is subtle but important. In a single-agent loop, the model itself decides when it has searched enough, when the evidence is sufficient, and when the task is “done.” Those decisions live inside the model’s own reasoning, which makes them invisible and unverifiable from the outside. In a graph-engineered system, those same decisions get pulled out into the structure of the graph itself. The agent inside each node can still reason freely about its own bounded job. It just doesn’t get to unilaterally decide what the whole system does next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V3Ur3FFGh8ZdPxsSu44J6g.png" /></figure><h3>Where It Fits: Prompt, Loop, Harness, Graph</h3><p>Think of these as four layers that stack, not four competing trends:</p><ul><li><strong>Prompt engineering</strong> shapes what you say to the model in a single turn.</li><li><strong>Loop engineering</strong> shapes how the model iterates: plan, act, verify, retry, stop. The term spread in mid-2026 after Anthropic’s Boris Cherny and OpenClaw’s Peter Steinberger both started describing how they design the loop around an agent instead of typing it instructions turn by turn, with Google’s Addy Osmani giving the practice its name.</li><li><strong>Harness engineering</strong> shapes the environment the loop runs inside: permissions, context management, progress files, git checkpoints. It’s what lets a single agent survive a multi-hour or multi-day session without forgetting where it left off.</li><li><strong>Graph engineering</strong> shapes the topology connecting multiple nodes, whether those nodes are agents, tools, validators, or humans. It’s what you reach for once a single loop isn’t enough.</li></ul><p>You don’t throw out the first three layers when you add the fourth. A well-built graph node is usually a small, well-harnessed loop in its own right.</p><p>Worth saying plainly: none of these four names are official standards handed down by a vendor. They’re labels the community converged on this year to describe patterns people were already building by hand. The label is new. The underlying idea, breaking a task into pieces you can actually check, is not.</p><h3>The Building Blocks</h3><p>A graph-engineered workflow is made of the same four pieces every time.</p><p><strong>Nodes</strong> are bounded units of work. Each one has a single responsibility: plan the change, write the code, run the tests. A node trying to do three things at once is usually a sign the graph needs to be split further.</p><p><strong>Edges</strong> decide what happens next, and they’re explicit rather than implied. “If the tests pass, move to Done. If they fail, move to Repair.” That rule lives in the graph itself, not buried in a paragraph of system prompt the model might interpret differently each time it rereads it.</p><p><strong>Shared state</strong> is what carries information across nodes: the plan, the diff, the test output, the retry count. Instead of hoping the model remembers a decision it made three steps ago, the state gets written down and handed forward.</p><p><strong>Gates</strong> are the checks a result has to clear before the graph calls it done: an automated test suite, a linter, a second agent acting as reviewer, or a human approval step for anything high-stakes.</p><h3>A Worked Example</h3><p>Here’s the shape most coding graphs end up taking. A task enters at <strong>Plan</strong>, which breaks the request into concrete steps. <strong>Implement</strong> writes the code. <strong>Test</strong> runs the suite. If the tests pass, the edge routes to <strong>Done</strong> and the change ships. If they fail, the edge routes to <strong>Repair</strong>, which fixes the specific failure and hands control back to <strong>Test</strong>, not back to the beginning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*jFQ7Gi1NLaqflgP4jAYqQQ.gif" /></figure><p>That loop-back edge is the detail that matters most. A plain agent loop that fails a test usually just tries again from wherever its context happened to leave off, which is how you end up with an agent re-explaining the whole problem to itself for the fifth time. A graph-engineered failure path is scoped: Repair only knows about the specific failure it was handed, makes the smallest fix it can, and re-enters Test. The retry is cheap because the node doing it has one narrow job.</p><p>Sketched as code, the same graph looks something like this. This is illustrative, not tied to any one library’s exact syntax, but the shape (nodes, edges, a conditional branch) is what you’ll find in most graph-orchestration tools:</p><pre>graph.add_node(&quot;plan&quot;, plan_fn)<br>graph.add_node(&quot;implement&quot;, implement_fn)<br>graph.add_node(&quot;test&quot;, test_fn)<br>graph.add_node(&quot;repair&quot;, repair_fn)<br>graph.add_node(&quot;done&quot;, done_fn)</pre><pre>graph.add_edge(&quot;plan&quot;, &quot;implement&quot;)<br>graph.add_edge(&quot;implement&quot;, &quot;test&quot;)<br>graph.add_conditional_edge(&quot;test&quot;, check_result,<br>    {&quot;pass&quot;: &quot;done&quot;, &quot;fail&quot;: &quot;repair&quot;})<br>graph.add_edge(&quot;repair&quot;, &quot;test&quot;)</pre><h3>How This Differs From “Just Add a Bigger Loop”</h3><p>It’s tempting to think you can get the same result by writing a more elaborate single-agent loop with a longer system prompt describing every case. In practice this breaks down for the same reason long, sprawling functions break down in regular software: every added branch increases the chance the model takes the wrong one, and none of those branches are independently testable. A graph forces the branches to be explicit and gives each one its own, much smaller, job to get right.</p><p>It also unlocks a few things a single loop structurally can’t do well:</p><ul><li><strong>Parallel fan-out.</strong> Independent nodes, say updating three unrelated modules, can run at the same time instead of serially, as long as each one owns its own files. Developers coordinating multiple coding agents on one repository converge on the same fix for the classic merge-conflict problem here: give each active task clear file ownership so two nodes never touch the same code at once.</li><li><strong>Verification that isn’t the same model grading its own work.</strong> A separate node, sometimes a second agent with different instructions, checks the output. That maker and checker split tends to catch mistakes a single agent misses when reviewing itself.</li><li><strong>A human checkpoint that’s a real graph node</strong>, not an interruption. High-stakes changes can route to a person before the Done edge fires, the same way they would in any other reviewed pipeline.</li></ul><h3>Where You’ll Actually See This</h3><p>The most common way people build these graphs today is with a state-graph library like LangGraph, where nodes and edges are defined explicitly and the framework handles execution and state-passing between them. You’ll also see the same ideas show up with no framework at all: developers running multiple coding agents side by side across one repo, giving each one ownership of specific files by convention so their work never collides. That’s graph engineering by discipline rather than by library.</p><p>The pattern generalizes past coding, too. A research assistant that searches, drafts, fact-checks its own draft, and only then sends a report is the same four-piece structure: node, edge, state, gate, just with a different job sitting in each node.</p><h3>When You Actually Need This</h3><p>Not every task deserves a graph. If you can describe the job in one clear instruction and a single well-harnessed loop can verify its own output, that’s simpler to build and easier to debug. Reach for graph engineering when you notice one of these:</p><ul><li>The task has genuinely independent sub-parts that could run in parallel</li><li>You need a check that’s separate from the agent that produced the work</li><li>Failures need a narrower recovery path than “start over from scratch”</li><li>A human needs a real, auditable decision point before anything ships</li></ul><p>Start with the smallest graph that satisfies those. Three or four nodes with one honest edge each will outperform a single sprawling prompt every time, and it’ll be far easier to explain to the next engineer who has to debug it at 2 a.m.</p><p>I have made this post for all readers. Not under the paid wall.</p><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://dev.to/naresh_007/graph-engineering-for-ai-coding-agents-beyond-prompt-loops-48h4">Graph Engineering for AI Coding Agents: Beyond Prompt Loops — DEV Community</a></li><li><a href="https://www.analyticsvidhya.com/blog/2026/07/graph-engineering/">Graph Engineering for AI Agents: A Complete Guide in LangGraph — Analytics Vidhya</a></li><li><a href="https://github.com/codejunkie99/graph-engineering">graph-engineering: task-graph orchestration patterns — GitHub</a></li><li><a href="https://www.eigent.ai/blog/graph-engineering-ai-agents">Graph Engineering for AI Agents — Eigent</a></li><li><a href="https://adtmag.com/articles/2026/07/01/loop-engineering-emerges-as-developers-put-ai-coding-agents-on-repeat.aspx">Loop Engineering Emerges as Developers Put AI Coding Agents on Repeat — ADTmag</a></li><li><a href="https://www.ibm.com/think/topics/loop-engineering">What Is Loop Engineering? — IBM</a></li><li><a href="https://www.verdent.ai/guides/agent/what-is-a-loop-engineer">What Is a Loop Engineer in AI Coding? — Verdent</a></li><li><a href="https://www.faros.ai/blog/harness-engineering">Harness Engineering: Making AI Coding Agents Work in 2026 — Faros AI</a></li><li><a href="https://www.nxcode.io/resources/news/what-is-harness-engineering-complete-guide-2026">What Is Harness Engineering? Complete Guide for AI Agents — NxCode</a></li><li><a href="https://www.ignorance.ai/p/the-emerging-harness-engineering">The Emerging “Harness Engineering” Playbook</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e605634cb029" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Unity Catalog Pages: Bulk Import Your Business Glossary with AI]]></title>
            <link>https://medium.com/@sudarshan-koirala/databricks-unity-catalog-pages-bulk-import-your-business-glossary-with-ai-7c222d90bb66?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c222d90bb66</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[unity-catalog-pages]]></category>
            <category><![CDATA[databricks-denie-ai]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Sat, 15 Aug 2026 08:00:43 GMT</pubDate>
            <atom:updated>2026-08-15T08:00:43.656Z</atom:updated>
            <content:encoded><![CDATA[<p>Stop writing term definitions by hand. Let Genie Code read your docs and draft them for you.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBmN6-F3VLX7kRk78KMLJQ.png" /></figure><p>Every company has its own words for things. One team calls it “MRR,” another calls it “recurring revenue,” and a third just says “the number.” When you ask an AI tool a question about your data, it has to guess which one you mean. Databricks Unity Catalog now has a fix for this called <strong>Pages</strong>, and a new feature called <strong>Bulk Import</strong> that builds your first batch of Pages for you.</p><h3>What Is a Page?</h3><p>A Page is a governed, single definition of a business term, acronym, or KPI. It lives inside Unity Catalog, under a domain (think of a domain as a folder for a business area, like “Finance” or “Marketing”).</p><p>Each Page holds eight parts:</p><ul><li><strong>Domain</strong>: which business area it belongs to</li><li><strong>Owner</strong>: the person responsible for it (you, by default)</li><li><strong>Synonyms</strong>: other names people use for the same thing</li><li><strong>Description</strong>: a short summary</li><li><strong>Page body</strong>: the full write-up, with links, images, tables, and @ tags to other assets</li><li><strong>Related assets</strong>: the tables, dashboards, or metrics tied to the term</li><li><strong>Sources</strong>: the documents the definition came from</li></ul><p>Once a Page is published, it becomes the single source of truth. When someone asks Genie One about that term, Genie One uses the Page’s definition first, instead of guessing from context.</p><h3>The Problem Pages Solve</h3><p>Without Pages, an AI agent infers meaning from whatever data it can see. That works until two teams use the same word differently, or a new hire has no idea what “the number” refers to. Pages remove the guesswork. One definition, one owner, one place to look.</p><h3>What Is Bulk Import?</h3><p>Writing Pages one by one is slow, and most teams already have this knowledge written down somewhere: a wiki, a PDF glossary, a Confluence page, an old onboarding doc. Bulk Import points Genie Code at that material and asks it to do the first draft.</p><p>You still review every term before anything goes live. Genie Code proposes, you decide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQwrcHH_6XSLAlb1VvZv4w.png" /></figure><p>Here is the flow, step by step:</p><ol><li><strong>Attach your source material.</strong> Go to Discover &gt; Pages &gt; Create page, open the Genie Code pane, and pick Bulk import pages. You can attach local files, Unity Catalog or workspace objects, MCP connections, or paste in a public link.</li><li><strong>Genie Code reads it.</strong> It scans every document you attached and pulls out the terms, acronyms, and KPIs it can find.</li><li><strong>You get proposed Pages.</strong> Genie Code returns one draft Page per term. It flags duplicates and low-confidence guesses along the way.</li><li><strong>You review and edit.</strong> Keep the good ones, fix the ones that need work, and exclude anything you do not want. Nothing is created until you approve it.</li><li><strong>Pages are created as drafts.</strong> All the terms you kept get saved at once, under the domain you picked.</li><li><strong>You publish, and Genie One cites it.</strong> Once published, the Page shows up as a source whenever Genie One answers a question that touches that term.</li></ol><p>Draft Pages stay visible only to their owner. Published Pages are visible to everyone in every Genie One conversation.</p><h3>Who Can Do What</h3><p>Page access is not set per Page. It is inherited from the domain:</p><ul><li><strong>Consumer</strong>: can create and publish Pages, and comment or suggest edits on Pages owned by others</li><li><strong>Page Owner</strong>: can edit drafts and published content, accept or reject suggestions, and change ownership</li><li><strong>Curator</strong>: has full control over every Page in the domains where they hold MANAGE DISCOVERY permission</li><li><strong>Account or Workspace Admin</strong>: is automatically a curator across every domain</li></ul><h3>Working With Published Pages</h3><p>Once a Page is live, people can leave comments, suggest edits, or react to it. Suggestions are batch edits, not inline ones, and the owner accepts or rejects the whole batch at once. Owners and curators can also skip the suggestion step and edit directly.</p><p>One thing worth knowing: accepting a suggestion clears every other pending batch on that Page, and you cannot undo it. So if two people suggest edits at the same time, only handle one batch at a time.</p><h3>Limits to Know Before You Start</h3><ul><li><strong>It is a Beta feature.</strong> An account admin has to turn it on from the Previews page first.</li><li><strong>No customer-managed key encryption.</strong> Page content may be replicated globally, so keep PII and regulated data out of it.</li><li><strong>No inline suggestions</strong>, only batch-based ones.</li><li><strong>Bulk Import needs source material.</strong> It cannot invent definitions from nothing. It works from what you feed it.</li></ul><h3>A Quick Example</h3><p>The Databricks docs use an “Estimates” Page inside a Financial Data domain as their example: a short definition, the business context around when the term applies, and a usage example underneath. That is really all a good Page needs. Clear enough that a new hire understands it in one read, structured enough that Genie One can quote it correctly.</p><p>If your team already has a glossary buried in a wiki or a PDF, Bulk Import is the fastest way to get it into Unity Catalog. Point Genie Code at it, review the drafts, publish the good ones, and your AI agent stops guessing what your business terms mean.</p><h4>I Have Made This Post for All Readers. Not Under the Paid Wall.</h4><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://docs.databricks.com/aws/en/uc-semantics/pages">Unity Catalog Semantics: Pages (Bulk Import)</a></li><li><a href="https://docs.databricks.com/aws/en/uc-semantics/">Unity Catalog Semantics Overview | Databricks on AWS</a></li><li><a href="https://www.databricks.com/product/unity-catalog/business-semantics">Unity Catalog Semantics | Databricks</a></li><li><a href="https://www.databricks.com/blog/announcing-public-preview-discover-and-domains-powered-unity-catalog">Announcing the Public Preview of Discover and Domains, powered by Unity Catalog</a></li><li><a href="https://www.databricks.com/blog/unified-data-discovery-business-context-unity-catalog">Unified data discovery with business context in Unity Catalog</a></li><li><a href="https://www.databricks.com/blog/next-generation-databricks-genie">The next generation of Databricks Genie</a></li><li><a href="https://docs.databricks.com/aws/en/genie/">Genie | Databricks on AWS</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/uc-semantics/">Unity Catalog Semantics — Azure Databricks | Microsoft Learn</a></li><li><a href="https://www.databricks.com/blog/whats-new-unity-catalog-data-ai-summit-2026">What’s new with Unity Catalog at Data + AI Summit 2026</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c222d90bb66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Technical Skills Every Data Engineer Should Have]]></title>
            <link>https://medium.com/@sudarshan-koirala/10-technical-skills-every-data-engineer-should-have-dca88750852c?source=rss-55cd50a9c816------2</link>
            <guid isPermaLink="false">https://medium.com/p/dca88750852c</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[Sudarshan Koirala]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 06:01:01 GMT</pubDate>
            <atom:updated>2026-08-10T06:01:01.452Z</atom:updated>
            <content:encoded><![CDATA[<p>The stack everybody actually uses, explained in plain English.</p><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9QI0B-WyjUkIQ6SDGfbMw.png" /></figure><p>Open any data engineering roadmap and you get a wall of fifty tool logos. It looks impossible, and most of it is noise.</p><p>In real projects the same ten things keep showing up, no matter the company, the cloud or the year. Vendors rename their products, new tools appear every quarter, but these ten stay exactly the same underneath.</p><p>Here is each one: what it is, why it matters, and what to actually learn. No fluff, just the parts you will use.</p><h3>The short version</h3><p>If you read nothing else, read this list.</p><ol><li><strong>Data modeling</strong> -&gt; the blueprint everything else is built on</li><li><strong>SQL</strong> -&gt; the language you speak every single day, in every tool</li><li><strong>Python and software engineering craft</strong> -&gt; code that survives production</li><li><strong>Distributed processing</strong> -&gt; what happens when data stops fitting on one machine</li><li><strong>Orchestration</strong> -&gt; schedules, retries and backfills you can trust at 3 AM</li><li><strong>Storage and file formats</strong> -&gt; Parquet, Delta, Iceberg and why your query is slow</li><li><strong>Cloud platform basics</strong> -&gt; object storage, permissions, compute and cost</li><li><strong>Streaming</strong> -&gt; Kafka and real time, when a nightly batch is too late</li><li><strong>Data quality, tests and observability</strong> -&gt; catching bad data before your users do</li><li><strong>Docker and CI/CD</strong> -&gt; shipping the same thing everywhere, automatically</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZnKj8j5obPPgo649ITrOA.png" /></figure><h3>1. Data modeling</h3><p><strong>Why it matters:</strong> the model is the contract. If the table is designed badly, every query, dashboard and ML feature downstream inherits the mess. You can survive a few months of loading raw data and querying it blindly. After that, nobody trusts the numbers.</p><p><strong>What to learn:</strong></p><ul><li>The grain of a table: what does one row actually mean?</li><li>Facts and dimensions (Kimball style star schema)</li><li>Slowly Changing Dimensions, especially Type 2 for history</li><li>Normalization vs denormalization, and when each one wins</li><li>Layered design: bronze (raw), silver (cleaned), gold (business ready)</li></ul><pre>-- one row = one order line, that is the grain<br>CREATE TABLE fact_order_items (<br>  order_item_id  BIGINT,<br>  order_id       BIGINT,<br>  customer_key   BIGINT,   -- FK to dim_customer<br>  product_key    BIGINT,<br>  order_date     DATE,<br>  quantity       INT,<br>  amount         DECIMAL(12,2)<br>);</pre><pre>CREATE TABLE dim_customer (<br>  customer_key   BIGINT,   -- surrogate key<br>  customer_id    STRING,   -- natural key from the source<br>  country        STRING,<br>  valid_from     TIMESTAMP,<br>  valid_to       TIMESTAMP,<br>  is_current     BOOLEAN   -- SCD Type 2<br>);</pre><p><strong>Self check:</strong> can you say the grain of your biggest table in one sentence?</p><h3>2. SQL</h3><p><strong>Why it matters:</strong> SQL shows up in every tool you will ever touch. Postgres, Snowflake, BigQuery, Databricks, DuckDB, dbt, and now AI assistants writing queries for you. You still have to read what they produce and say whether it is correct.</p><p><strong>Go past SELECT and JOIN:</strong></p><ul><li>CTEs to break a monster query into readable steps</li><li>Window functions for ranking, running totals and deduplication</li><li>NULL behaviour, join fan-out and the classic GROUP BY traps</li><li>Reading EXPLAIN output instead of guessing why something is slow</li></ul><pre>WITH ranked AS (<br>  SELECT *,<br>         ROW_NUMBER() OVER (<br>           PARTITION BY order_id<br>           ORDER BY updated_at DESC<br>         ) AS rn<br>  FROM bronze.orders_raw<br>)<br>SELECT * FROM ranked WHERE rn = 1;   -- keep the latest version of each order</pre><p><strong>Self check:</strong> run EXPLAIN on your slowest query. If the plan means nothing yet, that is your next lesson.</p><h3>3. Python and software engineering craft</h3><p><strong>Why it matters:</strong> a pipeline is software. It gets deployed, it breaks at night, someone else has to fix it. The gap between junior and senior is usually not Python syntax, it is engineering habits.</p><p><strong>The habits that matter:</strong></p><ul><li>Git and code review, even when you work alone</li><li>Tests: unit tests for transform logic, data tests for the output</li><li>Idempotency: running the same job twice gives the same result</li><li>Config outside the code, secrets never inside it</li><li>Clear logging, so failures explain themselves</li><li>Environments and packaging (uv or poetry), so it runs on the server too</li></ul><pre>def load_day(df, table: str, run_date: str):<br>    &quot;&quot;&quot;Re-running this for the same date is safe.&quot;&quot;&quot;<br>    (df.filter(df.order_date == run_date)<br>       .write<br>       .mode(&quot;overwrite&quot;)<br>       .option(&quot;replaceWhere&quot;, f&quot;order_date = &#39;{run_date}&#39;&quot;)  # only that partition<br>       .format(&quot;delta&quot;)<br>       .saveAsTable(table))</pre><p>The dbt docs make the same point about incremental models: because they carry state, they are the easiest place to accidentally break idempotence. Design for re-runs from day one.</p><h3>4. Distributed processing</h3><p><strong>Why it matters:</strong> at some point the data stops fitting on one machine and your intuition about performance stops working. This is what separates “it runs in 4 hours” from “it runs in 6 minutes”.</p><p><strong>The mental model:</strong></p><ul><li>Partitions: data is split into chunks processed in parallel</li><li>Shuffle: moving data across the network, the expensive part of any job</li><li>Skew: one key holds most of the rows, so one task holds up everything</li><li>Lazy evaluation: nothing runs until an action triggers it</li><li>Broadcast joins: send the small table to every executor instead of shuffling both</li></ul><pre>from pyspark.sql.functions import broadcast</pre><pre># small dimension, big fact -&gt; broadcast avoids a full shuffle<br>result = fact_orders.join(broadcast(dim_country), &quot;country_code&quot;)</pre><pre>result.explain()   # read the plan, do not guess</pre><p>Spark can also adapt plans at runtime using statistics, but missing or stale statistics lead to bad plans. Learning to read a query plan is the highest leverage habit here.</p><h3>5. Orchestration</h3><p><strong>Why it matters:</strong> one script is easy. Forty jobs with dependencies, different schedules, late data and partial failures is a different problem. Orchestration turns a pile of scripts into a system.</p><p><strong>What to learn:</strong></p><ul><li>DAGs: tasks plus dependencies, nothing magical</li><li>Time based schedules and event based triggers</li><li>Retries with backoff, alerting and SLAs</li><li>Backfills: rerunning history without corrupting today</li><li>Idempotent tasks, because retries will happen</li></ul><pre>from airflow.decorators import dag, task<br>import pendulum</pre><pre>@dag(schedule=&quot;0 3 * * *&quot;,<br>     start_date=pendulum.datetime(2026, 1, 1, tz=&quot;UTC&quot;),<br>     catchup=True,<br>     default_args={&quot;retries&quot;: 3})<br>def daily_sales():</pre><pre>    @task<br>    def ingest(**ctx):<br>        return load_day(ctx[&quot;ds&quot;])     # ds = the logical date, not &quot;today&quot;</pre><pre>    @task<br>    def transform(path: str):<br>        return build_gold(path)</pre><pre>    transform(ingest())</pre><pre>daily_sales()</pre><p>The detail most people miss: use the logical run date the scheduler gives you, never datetime.now(). That one habit is what makes backfills possible.</p><h3>6. Storage and file formats</h3><p><strong>Why it matters:</strong> the same data in the wrong format can cost ten times more and run ten times slower. This is the skill that quietly saves the most money.</p><p><strong>What to learn:</strong></p><ul><li>Row based (CSV, JSON, Avro) vs columnar (Parquet, ORC), and why analytics loves columns</li><li>Compression: zstd or snappy beats plain text every time</li><li>Table formats: Delta Lake, Apache Iceberg and Hudi add ACID transactions, time travel and schema evolution on top of files</li><li>Partitioning by the column you actually filter on</li><li>The small file problem: thousands of tiny files kill performance, so compact them</li></ul><pre>(df.write<br>   .mode(&quot;overwrite&quot;)<br>   .partitionBy(&quot;order_date&quot;)        # a column you filter on, not a high cardinality id<br>   .option(&quot;compression&quot;, &quot;zstd&quot;)<br>   .parquet(&quot;s3://lake/silver/orders/&quot;))</pre><p>Partitioning by customer_id with a million customers is how you create a million file disaster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*XmQru-dECfo8CisBvm1RlQ.gif" /></figure><h3>7. Cloud platform basics</h3><p><strong>Why it matters:</strong> almost every pipeline you build runs on AWS, Azure or GCP. You do not need a certification, you need to be comfortable with four things: where data sits, who can read it, what runs the compute, and what it costs.</p><p><strong>What to learn (pick one cloud, the concepts transfer):</strong></p><ul><li>Object storage: S3, ADLS or GCS, plus buckets, prefixes and lifecycle rules</li><li>Identity and permissions: roles and policies, least privilege, no keys in code</li><li>Compute options: a warehouse, a Spark cluster, a container or a serverless function</li><li>Networking basics: private endpoints, why your job cannot reach the database</li><li>Cost: storage is cheap, compute and data scanned are not</li></ul><p>Practical habits that pay off immediately:</p><ul><li>Tag resources so you can see where the money goes</li><li>Set lifecycle rules to move cold data to cheaper storage</li><li>Watch “bytes scanned” in your warehouse, that is usually the bill</li></ul><p><strong>Self check:</strong> can you explain who is allowed to read your bronze bucket, and why?</p><h3>8. Streaming and real time</h3><p><strong>Why it matters:</strong> some data cannot wait for the nightly batch. Fraud checks, live dashboards, alerting and recommendations all need events within seconds. Streaming is not a replacement for batch, it is a second gear.</p><p><strong>What to learn:</strong></p><ul><li>Kafka basics: topics, partitions, producers, consumers, offsets</li><li>Event time vs processing time, and why late data is normal</li><li>Windows and watermarks for aggregations over a moving stream</li><li>Checkpointing and exactly once semantics, so restarts do not duplicate data</li><li>When streaming is genuinely worth it, and when a 15 minute batch is fine</li></ul><pre>events = (spark.readStream<br>    .format(&quot;kafka&quot;)<br>    .option(&quot;kafka.bootstrap.servers&quot;, &quot;localhost:9092&quot;)<br>    .option(&quot;subscribe&quot;, &quot;orders&quot;)<br>    .load())</pre><pre>(events.selectExpr(&quot;CAST(value AS STRING) AS json&quot;)<br>    .writeStream<br>    .format(&quot;delta&quot;)<br>    .option(&quot;checkpointLocation&quot;, &quot;/chk/orders&quot;)   # this is what makes restarts safe<br>    .toTable(&quot;bronze.orders_stream&quot;))</pre><p>Structured Streaming lets you write streaming logic almost the same way you write batch logic, which is why it is the easiest place to start.</p><h3>9. Data quality, tests and observability</h3><p><strong>Why it matters:</strong> a pipeline that runs successfully and produces wrong numbers is worse than one that fails loudly. Trust is the actual product of a data team.</p><p><strong>What to learn:</strong></p><ul><li>Basic assertions: not null, unique, accepted values, referential integrity</li><li>Freshness and volume checks: did the data arrive, and is the row count sane?</li><li>Where to check: at the source, and again after every transformation hop</li><li>Fail vs warn: stop the pipeline for critical issues, warn for the rest</li><li>Lineage and alerting: know what broke, and what it breaks downstream</li></ul><pre># dbt: tests live next to the model<br>models:<br>  - name: orders<br>    columns:<br>      - name: order_id<br>        data_tests: [unique, not_null]<br>      - name: status<br>        data_tests:<br>          - accepted_values:<br>              arguments:<br>                values: [&#39;placed&#39;, &#39;shipped&#39;, &#39;completed&#39;, &#39;returned&#39;]</pre><p>Start with four tests on your most important table. That is already better than most production pipelines.</p><h3>10. Docker and CI/CD</h3><p><strong>Why it matters:</strong> “it works on my laptop” is not a deployment strategy. Containers make your job run the same way everywhere, and CI/CD makes shipping boring, which is exactly what you want.</p><p><strong>What to learn:</strong></p><ul><li>Images, containers, volumes and a simple Dockerfile</li><li>Docker Compose to run Postgres, Kafka or Airflow locally in one command</li><li>A pipeline that runs on every pull request: lint, tests, build</li><li>Deploy on merge, with the ability to roll back</li><li>Never bake secrets into an image</li></ul><pre>FROM python:3.12-slim<br>WORKDIR /app<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir -r requirements.txt<br>COPY . .<br>CMD [&quot;python&quot;, &quot;-m&quot;, &quot;pipelines.daily_sales&quot;]</pre><pre># .github/workflows/ci.yml<br>on: [pull_request]<br>jobs:<br>  test:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - uses: actions/checkout@v4<br>      - run: pip install -r requirements.txt<br>      - run: pytest -q</pre><p>Once this is in place, every change gets tested automatically and nobody has to remember to do it.</p><h3>How to learn these ten, in order</h3><p>Reading about skills is not learning them. Here is an order that works, roughly one step per week or two:</p><ol><li><strong>SQL first.</strong> Free Postgres or DuckDB, a public dataset, queries until window functions feel normal.</li><li><strong>Model something.</strong> Design a star schema for that dataset. One fact table, two or three dimensions.</li><li><strong>Automate it with Python.</strong> Move the load into a script, add git, add one test, make it safe to run twice.</li><li><strong>Add storage discipline.</strong> Write partitioned Parquet, then convert it to Delta or Iceberg and try time travel.</li><li><strong>Put it in the cloud.</strong> Same pipeline, but reading and writing to object storage with proper permissions.</li><li><strong>Schedule it.</strong> Airflow or Dagster with retries, then backfill 30 days of history on purpose.</li><li><strong>Add data tests.</strong> not null, unique, freshness. Break the data deliberately and watch it fail.</li><li><strong>Scale it.</strong> Load something big enough to hurt, run it on Spark, read the plan, fix one shuffle.</li><li><strong>Add a stream.</strong> Kafka locally, one topic, one streaming job writing into the same tables.</li><li><strong>Containerize and automate.</strong> Dockerfile plus a CI workflow that runs your tests on every push.</li></ol><p>At the end you have one project that touches all ten skills. That is worth far more in an interview than ten finished courses.</p><h3>Mistakes to avoid</h3><ul><li>Collecting tools instead of concepts. The tool is a detail, the concept is the skill.</li><li>Skipping data modeling because SQL feels faster. You pay for that later, with interest.</li><li>Writing pipelines that cannot be re-run safely.</li><li>Tuning by trial and error instead of reading the query plan.</li><li>Reaching for streaming when a 15 minute batch would do.</li><li>Building only in notebooks, so nothing is versioned, tested or scheduled.</li><li>Waiting until you feel ready. Build the small ugly version first, then improve it.</li></ul><p>One more thing that helps more than any tool: get a feedback loop. A senior colleague, a community, or an AI assistant prompted to review your work like an experienced data engineer would. Feedback is how you find out whether you are on the right track.</p><h3>The takeaway</h3><p>Tools will keep changing. AI will write more of the boilerplate every year. What stays valuable is the person who can say what the table means, why the query is slow, and what happens when the job fails at 3 AM.</p><p>Learn these ten, build one project that uses all of them, and you are ahead of most people collecting tool logos on their resume.</p><p>I have made this post for all readers. Not under the paid wall.</p><p><strong>I will keep posting content under the free plan, all I need is your support:</strong></p><ul><li><strong>Clap</strong> 50 times: each one helps more than you think! 👏</li><li><strong>Follow</strong> me here on Medium and subscribe for free to catch my latest posts. 🫶</li></ul><h3>Sources</h3><ol><li><a href="https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/">Kimball Group: Dimensional Modeling Techniques</a></li><li><a href="https://docs.getdbt.com/docs/build/incremental-models">dbt docs: Incremental models</a></li><li><a href="https://docs.getdbt.com/docs/build/data-tests">dbt docs: Add data tests to your DAG</a></li><li><a href="https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html">Apache Airflow: Core Concepts</a></li><li><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html">Apache Spark: SQL Performance Tuning</a></li><li><a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Apache Spark: Structured Streaming Programming Guide</a></li><li><a href="https://kafka.apache.org/quickstart/">Apache Kafka: Quickstart</a></li><li><a href="https://iceberg.apache.org/docs/latest/">Apache Iceberg Documentation</a></li><li><a href="https://docs.databricks.com/aws/en/lakehouse/medallion">Databricks: What is the medallion lakehouse architecture?</a></li><li><a href="https://docs.docker.com/get-started/">Docker: Get started</a></li></ol><p>👨🏾‍💻 <a href="https://github.com/sudarshan-koirala">GitHub</a> ⭐️ | 🐦 <a href="https://twitter.com/mesudarshan">Twitter</a> | 📹 <a href="https://www.youtube.com/@datasciencebasics">YouTube</a> | 👔 <a href="https://www.linkedin.com/in/sudarshan-koirala/">LinkedIn</a> | ☕️ <a href="http://ko-fi.com/datasciencebasics">Ko-fi</a></p><p>Thank you for your time in reading this post!</p><p>Make sure to leave your feedback and comments. See you in the next blog, stay tuned 📢</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dca88750852c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>