<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Umair Abro on Medium]]></title>
        <description><![CDATA[Stories by Umair Abro on Medium]]></description>
        <link>https://medium.com/@umairabro?source=rss-3ea379767a43------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3I5nyiSNXItHZmps.</url>
            <title>Stories by Umair Abro on Medium</title>
            <link>https://medium.com/@umairabro?source=rss-3ea379767a43------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:40:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@umairabro/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Ontology Builds Itself Now. The Source of Truth Doesn’t.]]></title>
            <link>https://medium.com/@umairabro/the-ontology-builds-itself-now-the-source-of-truth-doesnt-555abc815877?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/555abc815877</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 14:02:27 GMT</pubDate>
            <atom:updated>2026-11-08T14:03:07.031Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR</strong></p><ul><li><strong>The take:</strong> ranking decides which definition wins, not whether it’s right. Ownership is still a human job.</li><li><strong>What we did:</strong> four Genie agents in Slack, a control mapped to each OntoRank signal (four live, one open), and a parity gate that holds one agent to its dashboard (105 of 105 questions).</li><li><strong>What’s next:</strong> retire duplicate sources, give dashboards a governed home, and alert on every answer a user marks as wrong.</li></ul><h3>Nothing error-ed</h3><p>In July testing, our sales agent said 28 agencies earn more than 20% commission. The answer was 30.</p><p>The query ran. Nothing errored. The agent had read an incomplete file upload instead of the commission field in our CRM. Our first fix was a memory update, which fixed it for exactly one user: me. The real fix was a written rule, for every user, naming which source answers which question.</p><p>The same bug was waiting in our metrics. Two tables both called themselves MRR, and both were correct, for different questions. One tracked product MRR day by day; the other fed a customer MRR dashboard I’d just built. Put both in one Genie space and the same question could get two answers. Neither query would fail.</p><p>That’s not a model problem or a prompt problem. It’s an authority problem: nothing told the agent which number the company stands behind. Every company has a metric with two defensible definitions and only one that reaches the board.</p><p>Databricks now ranks authority for you. This isn’t a review of that ranker; it’s what we learned building the layer it ranks. My take: <strong>ranking settles which definition wins. It can’t tell you the winner is right, and it can’t make anyone own it.</strong></p><h3>Ranking decides authority</h3><p>The idea is old. <em>Ontologia</em> first appears in a <a href="https://en.wikipedia.org/wiki/Jacob_Lorhard">1606</a> Latin textbook.</p><p>What’s new is the automation. In June, Databricks introduced <a href="https://www.databricks.com/blog/introducing-genie-one-genie-ontology-and-genie-agents">Genie Ontology</a>, which builds an ontology from your tables, queries and dashboards. When definitions compete, <a href="https://answers.databricks.com/ontorank-genie-ontology">OntoRank</a> ranks them on five signals, and Genie uses the winner first:</p><ul><li><strong>Source authority:</strong> where the definition came from</li><li><strong>Author relevance:</strong> who created or maintains it</li><li><strong>Usage frequency:</strong> how often people rely on it</li><li><strong>Asset certification:</strong> whether it’s tagged as certified</li><li><strong>Freshness:</strong> how recent it is next to its rivals</li></ul><p>The layer is worth building. On data.world’s <a href="https://arxiv.org/abs/2311.07509">benchmark</a>, GPT-4 answered 16.7% of enterprise questions correctly against raw SQL and 54.2% against an ontology-backed knowledge graph. A <a href="https://arxiv.org/abs/2405.11706">follow-up</a> reached 72.55% by checking each query against the ontology and repairing it. Its model answered “I don’t know” about 8% of the time. Ours said 28.</p><h3>Authority isn’t correctness</h3><p>None of those five signals checks the arithmetic. <a href="https://rearc.io/blog/genie-ontology">Rearc</a>’s headline says it: Genie Ontology ranks trust, not correctness. We’ve been bitten both ways.</p><p><strong>Stale data that looked fresh.</strong> Our finance source mirrors stopped updating while the jobs reading them kept running on schedule. Judged by when the jobs last ran, the data looked fresh.</p><p><strong>Wrong data that looked trusted.</strong> A product launch broke an assumption baked into our subscription table. Simplified, the build did this:</p><pre>SELECT s.id, item.plan_name AS product_plan<br>FROM subscriptions s<br>LATERAL VIEW EXPLODE(s.line_items) t AS item<br>WHERE item.plan_name &lt;&gt; &#39;Seats&#39;  -- assumes one non-seat item</pre><p>A new reseller plan carried two or three non-seat items, so its subscriptions fanned out into 10 duplicate rows. Seats-only subscriptions lost their one row to the filter: 392 of them. No code changed; the data did. A uniqueness check caught the duplicates. Nothing caught the 392: the row count sat inside its alert band, and every check ran after the table was published.</p><p>Alarms, not gates.</p><p>Neither failure shows up in a name, a usage count or a timestamp. A ranker can’t see them. A check against the source can: when did the source last change, and does the table hold every record the source holds?</p><h3>Declare more, infer less</h3><p>Our one hard rule: where two sources compete, like our two MRR tables, a written rule decides which one answers which question. It binds every user, and it has a price, which the next section shows. Everything else goes to the ranker, so each of its signals maps to a control of ours. Four are live; one isn’t.</p><ol><li><strong>Source authority → manufacture the definitions.</strong> Our Metric View Factory generates governed metric views in six standard shapes. Each must tie out to certified figures before Genie can query it.</li><li><strong>Author relevance → give definitions a governed home.</strong> This is the open one: all 44 dashboards in our workspace still live in personal folders.</li><li><strong>Usage frequency → make the governed path the easy one.</strong> We treated self-serve as an adoption problem and moved Genie into Slack. Genie answers as the person asking, so questions fell through to a broader, less curated agent until we fixed the grants.</li><li><strong>Asset certification → earn the tag with a test.</strong> Our product agent’s trusted examples are generated from the dashboard’s own queries, and no change ships until the parity suite passes: 105 of 105 on Oct 15. Parity proves the agent agrees with the dashboard, not that the dashboard is right.</li><li><strong>Freshness → measure it at the source.</strong> For finance data, we check when the source last changed, not when the job last ran.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*58vHIrkdo_7MtXhgHL1RHQ.png" /><figcaption>OntoRank can only rank what the data team makes trustworthy</figcaption></figure><p>The tinted band is the data team’s job. The more we declare there, the less OntoRank has to infer.</p><h3>A rule in prose is a suggestion</h3><p>In September, a team building its Q4 plan logged 13 issues against one of our agents. The plan took about 30 Genie queries over three days, most of them rework. Several wrong figures reached a draft, and two were caught only because someone already knew the answer.</p><p>The clearest issue, postmortem-style:</p><ul><li><strong>Symptom:</strong> a count came back as 1. The answer was 64.</li><li><strong>Cause:</strong> a product line had been renamed, and the agent still filtered on the old name.</li><li><strong>Fix:</strong> a mapping from every product name to its current group.</li><li><strong>Result:</strong> after this fix and the others, the agent’s benchmark went from 65% to 98% (47 of 48 runs).</li></ul><p>We also told the agent to open every answer with its scope: record types · date range · product scope · ACV method · data source. It does so on about 55% of the answers that report a figure. Meanwhile, its instructions grew from about 28,000 characters in September to 35,373 on October 5.</p><p>A rule in prose is a suggestion, including the one I called the real fix in July. Our next move is SQL, not more prose.</p><h3>Next: fewer sources, more tests</h3><ul><li><strong>Retire the duplicates:</strong> the last pipelines reading defunct CRM copies, and a legacy table one agent still queries.</li><li><strong>Give dashboards a governed home:</strong> out of personal folders, with a real access model.</li><li><strong>Move rules from prose into pinned SQL examples.</strong></li><li><strong>Bring back our answer watchdog,</strong> alerting only when a user marks an answer as wrong. None of OntoRank’s five signals is “a user said this was wrong.”</li><li><strong>Test the next agent before it ships:</strong> our product-analytics agent gets checked against the reports its users already trust.</li></ul><p>One more: our audit log counted 605 Genie chat responses in Q3. We’re recounting under a written definition, because even our success metric needed an ontology.</p><h3>Where I’ve landed</h3><ul><li><strong>Ontology pays the moment two systems give one metric two answers.</strong></li><li><strong>Lock the definitions that matter, and make every surface say which one it uses.</strong> Declare the head; let the ranker handle the long tail.</li><li><strong>Someone has to own the truth.</strong> The data team owns definitions and their contract plus the agents that consume them.</li></ul><p>We used to answer questions. Now the job is making sure the answer that wins is right. The next wrong one won’t throw an error either.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=555abc815877" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Bottleneck Moved. Most Engineering Leaders Didn’t.]]></title>
            <link>https://medium.com/@umairabro/the-bottleneck-moved-most-engineering-leaders-didnt-99d409fc9129?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/99d409fc9129</guid>
            <category><![CDATA[dataops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[engineering-leadership]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 10:19:34 GMT</pubDate>
            <atom:updated>2026-11-02T10:19:34.949Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Engineering leadership when wrong answers pass every test</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etseKVHteM5flCBtUFbiHQ.png" /></figure><p><strong>TL;DR</strong></p><ul><li>Agents made writing code cheap. Trusting it is now the expensive part.</li><li>In data and AI it’s worse: a wrong number can pass every test.</li><li>Leaders have to move with it: decide where AI may be wrong, give every number an owner, and spend the time agents save on verification.</li></ul><h3>Two failures, zero errors</h3><p>In October, a partner commission total in one of our finance pipelines moved by 2.6% across three daily runs. Nothing crashed.</p><p>The cause was an LLM step that read commission rates out of invoice text. Run it twice on the same invoice and you could get two different rates. In most AI features, that’s a quirk. In a number people get paid on, it’s a defect.So we replaced the LLM with regex and arithmetic checks. Deterministic. Boring. Repeatable.</p><p>The first version of that fix came with 19 tests. They checked a Python copy of the logic, not the SQL that actually ran. In review, we broke that SQL seven different ways. Every test still passed. Review caught it. The tests never would have.</p><p>Two failures in one month. Neither threw an error.</p><h3>The bottleneck moved</h3><p>It isn’t just my team. Different studies, same direction:</p><ul><li><strong>More than 40%</strong> of pull requests on GitHub now start with an agent’s first draft, up from 17% in March (<a href="https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/">GitHub Octoverse</a>).</li><li>Organizations writing the most AI code <strong>wait 105% longer</strong> for a first review (<a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/">DX</a>).</li><li>Pull requests merged with <strong>no review at all are up 76.3%</strong> (<a href="https://www.faros.ai/blog/ai-speed-trap-takeaways">Faros AI</a>).</li></ul><p>Writing code got cheap. Reviewing it, verifying it and deciding to trust it did not.</p><p>Goldratt’s <em>The Goal</em> made the point more than forty years ago: time saved anywhere except the bottleneck doesn’t make the system faster. Kent Beck has the 2026 version: <strong>“Teams accumulate code faster than they accumulate trust”</strong> (<a href="https://newsletter.kentbeck.com/p/trust-is-the-binding-constraint">Kent Beck</a>).</p><h3>Why data and AI feel it first</h3><p>Application code tends to fail loudly: a crash, a 500, a red test. Data and AI fail quietly. The pipeline is green, the number is plausible, and it’s wrong.</p><p>I see three kinds of quiet failure:</p><ul><li><strong>Checks that can’t fail</strong>, like those 19 tests.</li><li><strong>Definitions nobody wrote down.</strong> One of our billing-quality metrics is more than 20 times higher under one definition than the other, depending on whether unchanged reissued invoices count as errors. Both queries are correct. We publish both until the metric’s business owner decides. Ask an agent for “the error rate” and it will often pick one without asking.</li><li><strong>Confident AI answers.</strong> Our finance AI analyst passed 308 of 310 questions in its test set. A great score. In finance, the two misses are still the story.</li></ul><p>Anthropic’s study of about 400,000 Claude Code sessions found that “domain expertise, and not coding proficiency, amplifies effective use” (<a href="https://www.anthropic.com/research/claude-code-expertise">Anthropic</a>). In data, domain expertise means knowing what the number <em>should</em> be. That’s the scarce input now.</p><h3>Five rules for the new bottleneck</h3><h3>1. Decide where AI is allowed to be wrong</h3><p>Don’t start with “can an agent do this?” Start with “what does a wrong answer cost, and who would notice?”</p><p>An agent can write the pipeline that calculates commission. An LLM shouldn’t sit inside it. Code an agent writes gets reviewed once, then runs the same way every day. An LLM inside the pipeline gives a fresh answer on every run, and nobody reviews those. Where money moves, keep the run deterministic.</p><p>Amazon’s new guidance for engineers agrees: “If your work does not benefit from current AI tooling, that is a finding, not a failing” (<a href="https://www.aboutamazon.com/news/company-news/amazon-software-engineer-role-guideline-ai">Amazon</a>).</p><h3>2. Measure trust, not usage</h3><p>Usage metrics get gamed. Meta dropped AI usage from engineer performance reviews after “token maxxing” (<a href="https://the-decoder.com/meta-drops-ai-usage-from-engineer-performance-reviews-after-tokenmaxxing-backfires/">The Decoder</a>). Uber ranked AI usage on internal leaderboards and burned through its annual AI budget in four months (<a href="https://techcrunch.com/2026/06/02/uber-caps-employee-ai-spending-after-blowing-through-budget-in-four-months/">TechCrunch</a>).</p><p>Track the constraint instead:</p><ul><li>Time to first review, by risk tier</li><li>Reverts on agent-assisted changes</li><li>Data issues that consumers found before the team did</li><li>Reconciliation pass rate on published numbers</li></ul><p>Token spend belongs on a budget, not a leaderboard.</p><h3>3. Spend human attention by blast radius</h3><p>Human attention is the scarcest resource on a data team. Tier it:</p><ul><li><strong>Tier 0:</strong> docs, formatting, internal tooling. Machine review only.</li><li><strong>Tier 1:</strong> internal analysis and exploratory pipelines. Machine review plus a human skim.</li><li><strong>Tier 2:</strong> anything that touches a metric definition, a join key, money, customer data or permissions, plus the tests that guard them. A named owner approves with evidence: a reconciliation or an output diff, not just a code diff.</li></ul><p>Diff size doesn’t set the tier. A one-line join-key change is Tier 2.</p><p>The Claude Code team runs a version of this. A classifier decides whether a change needs zero, one or two human approvals. From June to September, their merged pull requests more than doubled while the share that had to be reverted fell by more than 40% (<a href="https://claude.dev/blog/solving-code-review-at-anthropic/">Claude blog</a>).</p><h3>4. Give every number a human owner</h3><p>Agents can’t be accountable. The Linux kernel wrote it into its rules: “AI agents MUST NOT add Signed-off-by tags” (<a href="https://docs.kernel.org/process/coding-assistants.html">kernel.org</a>).</p><p>The data version:</p><ul><li>Every published metric has a named owner and a written definition.</li><li>Every merge has someone who can explain it without the agent. <strong>If you can’t explain it, you can’t merge it.</strong></li></ul><p>This isn’t about blame. When a wrong number ships, only a person can change what the team does next.</p><h3>5. Spend the time agents save on verification</h3><p>That same month I moved our revenue reporting pipeline, 41 dbt models, to a new platform, with agents doing the translation. The rebuild took three days.</p><p>Checking it was the real work. Only 9 of the 41 models matched the old system exactly. We traced each of the other 32 differences to a cause: timing, tie order, or defects in the old pipeline. None was a translation bug. On the revenue figure our quarterly targets are built on, the two systems differed by $5.</p><p>Agents made the port fast. The reconciliation turned the switch from a guess into a judgment call.</p><p>So the time agents save goes to verification:</p><ul><li><strong>Golden test sets</strong> for every AI analyst before anyone outside the team uses it.</li><li><strong>Reconciliation</strong> of every Tier 2 number against an independent source.</li><li><strong>Guards that refuse bad writes.</strong> A freshness gate once froze our revenue dashboard for days rather than publish numbers built on stale data. That’s the right way to fail.</li><li><strong>Evaluators that can’t edit:</strong> a <a href="https://medium.com/@umairabro/your-agent-is-amnesiac-the-filesystem-isnt-a6c6ec20ecc2">read-only agent</a> that marks nothing as passed until it has opened the evidence.</li></ul><h3>The job now</h3><p>Management isn’t disappearing. It’s changing shape. Engineering managers are starting to run 12–14 people instead of 4–8, and more companies expect them to be hands-on (<a href="https://newsletter.pragmaticengineer.com/p/the-revised-rules-of-engineering">The Pragmatic Engineer</a>). I did that pipeline port myself.</p><p>The old job: assign work, keep the queue moving, grow output.</p><p>The new job: design the system that decides what gets trusted. Where AI may be wrong. Who owns each number. How review is tiered. What gets reconciled. What the token budget is.</p><p>And grow the people who can tell right from wrong. In an Anthropic trial, developers who learned a new library with AI help scored 50% on a quiz afterwards, against 67% for those who coded by hand. The biggest gap was in debugging (<a href="https://www.anthropic.com/research/AI-assistance-coding-skills">Anthropic</a>). So give juniors the verification work: golden tests, reconciliations, explaining changes before merge. That’s how people learn what “right” looks like.</p><p>In May I wrote that <a href="https://medium.com/@umairabro/i-argued-agentic-engineering-was-the-fix-for-vibe-coding-i-was-half-right-c875b3c4f6bc">the harness is the moat</a>. One level up, <strong>your org is the harness</strong>: review tiers are its hooks, owners its permissions, golden test sets its evaluators, and the token budget its circuit breaker.</p><h3>One thing to do this week</h3><p>Run a bottleneck audit. It takes an afternoon.</p><ol><li>Take the last 20 changes your team merged with agent help.</li><li>For each, note how long it waited for review, which tier it should have been, who approved it, and whether anyone checked the output against reality or only ran the tests.</li><li>Find where work waited longest, and where “checked” meant “the tests passed”.</li><li>Move one person to that step for a week.</li></ol><p>Twenty rows will tell you more about your real throughput than any adoption dashboard.</p><p>Agents made code cheap. They didn’t make trust cheap.</p><p>The bottleneck moved. Move with it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99d409fc9129" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Agent Is Amnesiac. The Filesystem Isn’t.]]></title>
            <link>https://medium.com/@umairabro/your-agent-is-amnesiac-the-filesystem-isnt-a6c6ec20ecc2?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6c6ec20ecc2</guid>
            <category><![CDATA[engineering-strategy]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Wed, 13 May 2026 14:27:35 GMT</pubDate>
            <atom:updated>2026-05-15T10:30:32.135Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A working manual for long-running coding agents every file, every hook, every script with the load-bearing pieces flagged.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JWd6RQ6abtJBG5bd.jpg" /></figure><p>A few weeks ago I argued that vibe coding is a tech-debt factory and that agentic engineering might be the fix. I owed you the operating manual.</p><p>The thesis fits on one line. If you take nothing else from this post:</p><blockquote><strong><em>Your coding agent is amnesiac across sessions. Your filesystem isn’t. The entire discipline of running long-running agents is making the filesystem do the remembering so the model doesn’t have to.</em></strong></blockquote><p>That’s it. Everything else is corollary.</p><h3>The proof: ~400 lines of glue</h3><p>On May 5, 2026, Anthropic published the take-home from the Long-Running Agents station at Code with Claude 2026. It lives at <a href="https://github.com/anthropics/cwc-long-running-agents"><strong>anthropics/cwc-long-running-agents</strong></a><strong>.</strong></p><p>It’s not a model. It’s not a framework. It’s a directory you copy into your repo .a handful of hook scripts and one subagent definition, all shell + markdown, about 400 lines total. The README is explicit that it’s an “event demo” of patterns, not a turnkey harness.</p><p>What makes it worth reading carefully is that it strips the long-running pattern down to three primitives. Anthropic names them, and you can audit each one in a single file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USAIKDsMx_tQ6ZIky7GY3Q.png" /></figure><p>Two operator-control hooks (kill-switch.sh, steer.sh) round it out for human-in-the-loop intervention.</p><p>That’s the whole repo. Read it in 30 minutes.</p><p>The supporting papers are <a href="https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents">Effective Harnesses for Long-Running Agents</a> (Nov 26, 2025) and <a href="https://www.anthropic.com/engineering/harness-design-long-running-apps">Harness Design for Long-Running Application Development</a> (Mar 24, 2026). The Nov paper introduces the two-agent split initializer + coding agent and the artifacts that survive session resets. The Mar paper goes further, prototypes a GAN-inspired planner/generator/evaluator system, then <em>simplifies it back down</em> when Opus 4.6 lands and makes some of the scaffold redundant. Read them in that order; the second is most interesting because it documents the model-improvement → harness-simplification feedback loop directly.</p><p>Geoffrey Huntley arrived at a converging shape independently with <a href="https://ghuntley.com/ralph/">Ralph Wiggum</a> (gone viral late 2025; Anthropic now ships an <a href="https://github.com/anthropics/claude-plugins-official/tree/main/plugins/ralph-loop">official plugin</a> wrapping the technique). When two groups land on the same architecture without coordinating, that’s usually a signal worth reading carefully.</p><p>It also explains the Karpathy line from his Feb 4, 2026 anniversary tweet on vibe coding:</p><blockquote>“Agentic because the new default is that you are not writing the code directly 99% of the time, you are orchestrating agents who do and acting as oversight. Engineering to emphasize that there is an art &amp; science and expertise to it.”</blockquote><p>Twelve words. Drew the line between vibe coding and a profession.</p><p>The Anthropic repo is deliberately minimal. What I’m walking through below is a <em>production</em> setup built on top of it, hardened over six weeks of running it for real on data infrastructure work. The three primitives stay; everything else is engineering judgment you’d want for a team workflow:</p><ul><li><strong>Spec slices</strong> as the unit of agent-deliverable work</li><li><strong>Conformance YAMLs</strong> wired to the evaluator</li><li><strong>A slash command</strong> that runs the build → evaluate → PR loop end-to-end</li><li><strong>Worktree isolation</strong> for parallel agents</li><li><strong>A Ralph loop</strong> for overnight autonomy</li><li><strong>Failure-mode hardening </strong>the parts that bit me</li></ul><p>Anthropic’s example is a web app verified with Playwright screenshots. My adaptation is data infrastructure verified with pytest against conformance YAMLs. The shape is identical; the verifier is what changes.</p><h3>Day zero: the harness lives in .claude/</h3><p>Tree you’re aiming for:</p><pre>your-repo/<br>├── CLAUDE.md                       # ≤60 lines of project conventions<br>├── specs/<br>│   └── schema-drift-check.md       # one slice = one spec<br>├── tests/<br>│   └── conformance/<br>│       └── schema-drift-check.yaml<br>├── .claude/<br>│   ├── settings.json<br>│   ├── settings.local.json         # gitignored<br>│   ├── agents/<br>│   │   └── evaluator.md<br>│   ├── hooks/<br>│   │   ├── track-read.sh           # from Anthropic<br>│   │   ├── verify-gate.sh          # from Anthropic<br>│   │   ├── commit-on-stop.sh       # from Anthropic<br>│   │   ├── pre-tool-use-policy.sh  # your custom secret/file guardrails<br>│   │   └── session-start.sh        # your context loader<br>│   └── commands/<br>│       └── ship-slice.md<br>├── feature_list.json               # written by initializer<br>└── PROGRESS.md                     # append-only handoff log</pre><h3>Step 1: Bootstrap from the canonical example</h3><pre>git clone https://github.com/anthropics/cwc-long-running-agents /tmp/cwc<br>cp -r /tmp/cwc/claude-code-config/.claude /path/to/your/repo/<br>chmod +x /path/to/your/repo/.claude/hooks/*.sh</pre><p>Then in Claude Code, paste this:</p><blockquote>Walk me through what each hook and the evaluator subagent does. List the paths and environment variables I’ll need to adapt for this project. Don’t change anything yet.</blockquote><p>Claude reads each file, summarizes the integration points, flags what to retarget. Adapt from there.</p><h3>Step 2 : .claude/settings.json</h3><p>This is the wiring file. Hooks register here. Without it, scripts under .claude/hooks/ never fire.</p><pre>{<br>  &quot;$schema&quot;: &quot;https://json.schemastore.org/claude-code-settings&quot;,<br>  &quot;permissions&quot;: {<br>    &quot;additionalDirectories&quot;: [&quot;specs&quot;, &quot;tests/conformance&quot;, &quot;feature_list.json&quot;],<br>    &quot;allow&quot;: [&quot;Read&quot;, &quot;Write&quot;, &quot;Edit&quot;, &quot;Bash(git:*)&quot;, &quot;Bash(uv:*)&quot;],<br>    &quot;ask&quot;:   [&quot;Bash(rm:*)&quot;, &quot;Bash(gh pr merge:*)&quot;, &quot;Bash(git push:*)&quot;],<br>    &quot;deny&quot;:  [&quot;Bash(rm -rf:*)&quot;]<br>  },<br>  &quot;hooks&quot;: {<br>    &quot;PreToolUse&quot;: [{<br>      &quot;matcher&quot;: &quot;Edit|Write|MultiEdit&quot;,<br>      &quot;hooks&quot;: [{ &quot;type&quot;: &quot;command&quot;, &quot;command&quot;: &quot;.claude/hooks/pre-tool-use-policy.sh&quot; }]<br>    }],<br>    &quot;SessionStart&quot;: [{<br>      &quot;hooks&quot;: [{ &quot;type&quot;: &quot;command&quot;, &quot;command&quot;: &quot;.claude/hooks/session-start.sh&quot; }]<br>    }],<br>    &quot;Stop&quot;: [{<br>      &quot;hooks&quot;: [{ &quot;type&quot;: &quot;command&quot;, &quot;command&quot;: &quot;.claude/hooks/commit-on-stop.sh&quot; }]<br>    }]<br>  }<br>}</pre><p>Three permission tiers. allow is the agent&#39;s free movement zone. ask requires explicit confirmation. deny is the wall.</p><h3>Step 3: The hook contract, correctly</h3><p>Hooks are shell scripts. JSON in via stdin, exit code out. The contract — straight from <a href="https://code.claude.com/docs/en/hooks">Anthropic’s hooks reference</a> — is the part most blog posts get wrong:</p><ul><li><strong>Exit 0</strong> : success, action proceeds.</li><li><strong>Exit 1</strong> : <em>non-blocking</em> error. The action still proceeds. Stderr is shown to the user only.</li><li><strong>Exit 2</strong> : blocking error. The action is stopped <em>and</em> stderr is fed back to Claude as feedback it can reason about.</li></ul><p>If your hook is meant to <em>enforce</em> a policy, you must exit 2. exit 1 looks like it works in casual testing and silently fails when it matters.</p><blockquote><strong><em>Known issue worth knowing.</em></strong><em> On Write/Edit/MultiEdit specifically, </em><em>exit 2 doesn&#39;t always block — </em><a href="https://github.com/anthropics/claude-code/issues/13744"><em>issue #13744</em></a><em> is still open. For hard-policy edits, the more reliable mechanism is structured JSON output with </em><em>&quot;permissionDecision&quot;: &quot;deny&quot;. Treat shell-exit blocking as best-effort on file writes.</em></blockquote><h3>Hook 1: pre-tool-use-policy.sh (your guardrails)</h3><pre>#!/usr/bin/env bash<br>payload=$(cat)<br>path=$(echo &quot;$payload&quot;    | jq -r &#39;.tool_input.file_path // empty&#39;)<br>content=$(echo &quot;$payload&quot; | jq -r &#39;.tool_input.content   // empty&#39;)<br><br># AWS keys, .env, anything else you want to wall off<br>if echo &quot;$content&quot; | grep -qE &#39;AKIA[0-9A-Z]{16}|aws_secret_access_key&#39;; then<br>  echo &quot;Blocked: AWS credential pattern in $path&quot; &gt;&amp;2<br>  exit 2<br>fi<br>if echo &quot;$path&quot; | grep -qE &#39;\.env(\.|$)&#39;; then<br>  echo &quot;Blocked: .env edits require human action&quot; &gt;&amp;2<br>  exit 2<br>fi<br>exit 0</pre><p>Eight lines. Catches the failure mode of an agent helpfully echoing your .env into a commit.</p><h3>Hook 2: commit-on-stop.sh (Anthropic ships this one)</h3><p>The Anthropic-shipped version commits whatever’s still uncommitted at session end so git log stays a faithful second record of work. If you want to extend it into a &quot;tests must pass to stop&quot; gate, layer your own on top:</p><pre>#!/usr/bin/env bash<br>slice=$(cat feature_list.json | jq -r &#39;.current_slice&#39;)<br>if [ -f &quot;tests/conformance/${slice}.yaml&quot; ]; then<br>  if ! uv run pytest &quot;tests/conformance/${slice}.yaml&quot; -q; then<br>    echo &quot;Tests failing for slice ${slice}. Cannot stop.&quot; &gt;&amp;2<br>    exit 2   # blocks the stop, re-injects failure into the agent<br>  fi<br>fi<br>exit 0</pre><p>exit 2 on Stop is the same mechanism Anthropic&#39;s official Ralph plugin uses to keep an agent looping until convergence.</p><h3>Hook 3: session-start.sh (your context primer)</h3><p>Runs on every new session. Output gets fed into the agent’s context as system metadata, so the agent boots already knowing what it was working on:</p><pre>#!/usr/bin/env bash<br>echo &quot;=== PROGRESS ===&quot;<br>tail -n 50 PROGRESS.md 2&gt;/dev/null<br>echo<br>echo &quot;=== CURRENT SLICE ===&quot;<br>cat feature_list.json 2&gt;/dev/null | jq &#39;.features[] | select(.passes==false) | .id, .description&#39; | head -10<br>echo<br>echo &quot;=== RECENT COMMITS ===&quot;<br>git log --oneline -10<br>exit 0</pre><p>No “what was I doing again?” round trip.</p><h3>The spec the agent reads first</h3><p>The agent reads the spec before it reads any code. If the spec is ambiguous, the agent will make it concrete in whatever direction is most convenient which is almost never the direction you wanted.</p><p>Every slice gets a one-page spec. Three sections. That’s it.</p><p>specs/schema-drift-check.md:</p><pre># specs/schema-drift-check.md   ·   slice 01 of 03<br>## Intent<br>Detect schema changes in our 30+ bronze tables and alert before<br>they break downstream consumers. Tier-1 only this slice.<br>## Context<br>Tables:        catalog.bronze.*  (~hourly, 30+ tables)<br>Pattern:       src/alerts/dispatch.py  (PagerDuty webhook)<br>Conventions:   threecolts.alerts.bronze.&lt;table&gt;<br>Tier-1 list:   CLAUDE.md § Tables-we-own<br>Libraries:     pyspark 3.5, deequ 1.2 already in env<br>Gotchas:       reorder ≠ rename (use ordinal + name)<br>               NULL → NOT NULL = P0 drift<br>               ignore _delta_log, _metrics<br>## Acceptance   (the verifier runs these)<br>See tests/conformance/schema-drift-check.yaml<br>  · 01-detects-added-column      &#39;currency&#39; added → alert<br>  · 02-dedupes-within-1h         same event twice → one alert<br>  · 03-ignores-comment-changes   comment edit → no alert<br>## Out of scope<br>Tier-2/3 tables, auto-remediation  →  separate slices</pre><p>Each section has a different job. <strong>Intent</strong> is what we’re building and why. <strong>Context</strong> is the curated codebase intelligence what an AI needs that humans assume. <strong>Acceptance</strong> is machine-checkable criteria.</p><p>The “Out of scope” block is load-bearing. Without it the agent will try to fix tier-2/3 tables, add auto-remediation, and you’ll get a 40-file PR you can’t review.</p><h3>The conformance file the verifier runs</h3><p>tests/conformance/schema-drift-check.yaml:</p><pre>slice: schema-drift-check<br>cases:<br>  - id: 01-detects-added-column<br>    given:<br>      table: catalog.bronze.orders<br>      add_column: { name: currency, type: string, nullable: true }<br>    expect:<br>      alerts:<br>        - severity: P1<br>          namespace: threecolts.alerts.bronze.orders<br>          contains: &quot;added column: currency&quot;<br>- id: 02-dedupes-within-1h<br>    given:<br>      events:<br>        - { at: &quot;2026-05-13T10:00:00Z&quot;, table: &quot;catalog.bronze.orders&quot;, drift: &quot;col_added&quot; }<br>        - { at: &quot;2026-05-13T10:30:00Z&quot;, table: &quot;catalog.bronze.orders&quot;, drift: &quot;col_added&quot; }<br>    expect:<br>      alert_count: 1<br>  - id: 03-ignores-comment-changes<br>    given:<br>      table: catalog.bronze.orders<br>      change_column_comment: { name: id, comment: &quot;primary key, unique per tenant&quot; }<br>    expect:<br>      alert_count: 0</pre><p>Each case has a given and an expect. The verifier reads the YAML, runs each case as a parametrized pytest, captures stdout, compares to expected. Anything else coverage, lint, types is on the side; these three cases are the contract.</p><p>GitHub’s <a href="https://github.com/github/spec-kit">Spec-Kit</a> (90k+ stars as of May 2026) emits the same shape from /speckit.specify. If you want tooling, use it. If you don&#39;t, the format above works in any text editor.</p><h3>The evaluator subagent that never grades itself</h3><p>This is the highest-leverage pattern in the stack. Self-grading is the failure mode agents skew positive on their own work. The Mar 2026 paper documents this directly: “agents tend to respond by confidently praising the work even when, to a human observer, the quality is obviously mediocre.”</p><p>The fix is structural, not procedural. You don’t ask the builder to “review more carefully.” You spawn a separate subagent in a separate context with restricted tools.</p><p>.claude/agents/evaluator.md:</p><pre>---<br>name: evaluator<br>description: Grade a slice. No code edits.<br>tools: Read, Bash, Grep, Glob<br>model: sonnet<br>---<br># Inputs (read in this order)<br>1. specs/{{slice}}.md<br>2. git diff main..HEAD<br>3. tests/conformance/{{slice}}.yaml<br>4. /tmp/evidence/{{slice}}/*<br><br># Rule (DEFAULT-FAIL)<br>Every Acceptance case starts FAILING. Mark passing only<br>after opening the evidence and verifying it yourself.<br><br># Output (exact shape)<br>VERDICT: PASS | NEEDS_WORK<br>FINDINGS: case &lt;id&gt;, pass|fail, evidence path, root cause<br># On NEEDS_WORK: stop. Output = next builder prompt.</pre><p>Three principles encoded:</p><p><strong>No Write/Edit tools.</strong> The evaluator can’t fix what it finds that’s the builder’s job. Forcing the round-trip is the point.</p><p><strong>Default-FAIL contract.</strong> Every criterion starts false. The evaluator can only mark a case passing after it opens the evidence file. This is the same primitive Anthropic’s verify-gate.sh enforces at the hook level. Without it, agents quietly redefine &quot;done.&quot;</p><p><strong>Fresh context.</strong> The evaluator never saw the build. It can’t rationalize the choices the builder made.</p><h3>How findings flow back</h3><pre>claude --agent evaluator -p &quot;slice=schema-drift-check&quot; \<br>  | tee /tmp/eval-output.txt \<br>  | grep -q &quot;VERDICT: PASS&quot; \<br>  || claude -p &quot;$(cat /tmp/eval-output.txt)\n\nFix all findings. Commit. Then stop.&quot;</pre><p>Either pipe it as above or wrap it in a Ralph loop. Either way, you’ve closed the build → evaluate → rebuild loop without a human in the middle.</p><h3>Four artifacts that survive the session gap</h3><p>A single Claude Code session has a context window. A long-running task doesn’t. The agent will hit the window, compact, restart and on every restart it forgets everything that wasn’t written down.</p><p>Anthropic’s Nov 2025 pattern is two prompts and four artifacts. The initializer prompt runs once at the start; the coding agent runs many times against the same prologue.</p><h3>The initializer prompt</h3><p>You paste this once, when the slice plan is decided:</p><blockquote><em>You are bootstrapping a long-running coding task. The slice plan is in </em><em>specs/. Do not write feature code yet. Do these in order, then stop:</em></blockquote><blockquote>Read every file in specs/. Build a feature_list.json with one entry per slice. Each entry: { id, description, acceptance_ref, depends_on, passes: false }.</blockquote><blockquote>Write init.sh that boots the dev stack and runs migrations. Make it idempotent.</blockquote><blockquote>Write PROGRESS.md with four sections: ## Done, ## In progress, ## Next, ## Notes. Leave empty.</blockquote><blockquote>Create the initial git commit with these four artifacts.</blockquote><blockquote>Print “INIT_DONE” and stop.</blockquote><p>That produces the four artifacts: a checklist (feature_list.json) with every entry passes: false, a boot script (init.sh), a handoff note (PROGRESS.md), and the initial commit anchoring the timeline.</p><h3>feature_list.json schema</h3><pre>{<br>  &quot;current_slice&quot;: &quot;schema-drift-check&quot;,<br>  &quot;features&quot;: [<br>    {<br>      &quot;id&quot;: &quot;schema-drift-check&quot;,<br>      &quot;description&quot;: &quot;Detect schema changes in tier-1 bronze tables and alert.&quot;,<br>      &quot;acceptance_ref&quot;: &quot;tests/conformance/schema-drift-check.yaml&quot;,<br>      &quot;depends_on&quot;: [],<br>      &quot;passes&quot;: false<br>    },<br>    {<br>      &quot;id&quot;: &quot;drift-dedup-window&quot;,<br>      &quot;description&quot;: &quot;Dedupe identical drift events within a 1h window.&quot;,<br>      &quot;acceptance_ref&quot;: &quot;tests/conformance/drift-dedup-window.yaml&quot;,<br>      &quot;depends_on&quot;: [&quot;schema-drift-check&quot;],<br>      &quot;passes&quot;: false<br>    }<br>  ]<br>}</pre><p>Note depends_on. The coding agent reads this and picks slices that have all dependencies passing. Parallel-safe slices can run in worktrees simultaneously.</p><p>Anthropic’s Nov 2025 paper uses JSON for this file deliberately the model is less likely to inappropriately overwrite a structured JSON file than a Markdown one.</p><h3>The coding agent prologue</h3><p>Every coding-agent session runs this first. The SessionStart hook injects most of it:</p><pre># 1. orient<br>pwd<br>cat PROGRESS.md<br>git log --oneline -20<br><br># 2. plan<br>cat feature_list.json | jq &#39;.features[] | select(.passes==false)&#39;<br># 3. boot<br>./init.sh<br># 4. work - one feature, one session</pre><p>The structure does the remembering. The model doesn’t have to.</p><h3>One slash command ships a slice</h3><p>Slash commands are markdown files with YAML frontmatter — nothing magic. They’re workflows you invoke as a human.</p><p>.claude/commands/ship-slice.md:</p><pre>---<br>description: |<br>  Implement a slice from a spec. Run conformance,<br>  open a PR, mark passes:true in feature_list.json.<br>argument-hint: &quot;&lt;slice-id&gt;&quot;<br>allowed-tools: Read, Write, Edit, Bash, Grep, Glob<br>model: sonnet<br>---<br># Ship slice: {{args}}<br>1. Read `specs/{{args}}.md`. Refuse if any section is missing.<br>2. Create worktree: `git worktree add ../wt-{{args}} -b feat/{{args}}`<br>3. Implement against the spec. Follow the Context section&#39;s patterns.<br>4. Run conformance: `uv run pytest tests/conformance/{{args}}.yaml`<br>5. Invoke evaluator: `claude --agent evaluator -p &quot;slice={{args}}&quot;`<br>6. If PASS: `git push -u origin feat/{{args}} &amp;&amp; gh pr create -f`<br>7. If NEEDS_WORK: feed findings back. Re-implement. Go to 4.<br>8. Update `feature_list.json`: `passes: true` for &quot;{{args}}&quot;.<br>9. Append to `PROGRESS.md`. Stop. Report PR number.</pre><p>Invoke interactively:</p><pre>/ship-slice schema-drift-check</pre><p>This is what makes Monday-morning autonomy possible. You don’t memorize a sequence of commands; the sequence is a file the agent reads.</p><h3>Worktrees: parallel agents without merge conflicts</h3><p>Native in Claude Code <a href="https://code.claude.com/docs/en/worktrees">since v2.1.49</a> (Feb 19, 2026). Worktree-create/remove hooks landed in v2.1.50 for non-git VCS support.</p><p>Each worktree gets its own working directory. Shared .git history. No file-level contention.</p><p><strong>Two paths.</strong></p><p>Manual drive multiple terminals yourself:</p><pre>claude --worktree slice-X            # or shorthand: claude -w slice-X<br># creates .claude/worktrees/slice-X/ on branch worktree-slice-X<br>git worktree list                    # see all active<br>git worktree remove .claude/worktrees/slice-X</pre><p>Subagent Claude Code dispatches into worktrees automatically. Add to the subagent frontmatter:</p><pre>---<br>name: slice-implementer<br>tools: Read, Write, Edit, Bash, Grep<br>isolation: worktree<br>---</pre><p>When the subagent finishes with no changes, its worktree is removed silently. With changes, the path is returned so you can review or merge. Add a .worktreeinclude file (gitignore syntax) to copy env files into each new worktree.</p><p>A practical cap of 4–8 worktrees per developer seems to be where most teams land. Past that, PR review queue becomes the bottleneck, not Claude. Prune weekly.</p><h3>Running overnight: Ralph, with safety</h3><p>The starkest expression of the whole discipline is three lines of bash:</p><pre>while :; do<br>  cat PROMPT.md | claude<br>done</pre><p>The Ralph loop, named after the Simpsons character by Geoffrey Huntley. Each iteration sees the previous iteration’s work through the filesystem and git history. Same prompt. Different filesystem.</p><p>The model doesn’t talk to itself between iterations. It doesn’t need to. The codebase carries the state.</p><h3>The safer way</h3><p>Bare bash is fine for proofs of concept. For overnight runs, use the <a href="https://github.com/anthropics/claude-plugins-official/tree/main/plugins/ralph-loop">official plugin</a>. It implements the same pattern via a Stop hook (exit 2 re-injects the prompt) and adds a completion-promise sentinel that must be wrapped in &lt;promise&gt; tags:</p><pre>/ralph-loop &quot;Implement the slices in feature_list.json one at a time. \<br>  After each: invoke the evaluator subagent. \<br>  If PASS, commit. If NEEDS_WORK, re-implement. \<br>  When every entry has passes:true, output &lt;promise&gt;ALL_SLICES_PASS&lt;/promise&gt;.&quot; \<br>  --max-iterations 30 \<br>  --completion-promise &quot;ALL_SLICES_PASS&quot;</pre><p>Three parameters that matter:</p><ul><li><strong>--max-iterations is your real safety net, not the sentinel.</strong> Default to a number where the worst case (every iteration is a full agent run) costs less than the dollar threshold you&#39;d hate to spend by accident. The sentinel uses exact string matching, which is brittle.</li><li><strong>--completion-promise must be impossible to emit without satisfying the goal.</strong> &quot;DONE&quot; is too easy to fake. &quot;ALL_SLICES_PASS&quot; forces the agent to flip every passes: true to legitimately emit it.</li><li><strong>The </strong><strong>&lt;promise&gt;...&lt;/promise&gt; wrapping is mandatory</strong> for the plugin&#39;s Stop hook to detect completion — it greps for the tag, not just the sentinel string.</li></ul><p>Huntley calls this “deterministically bad in an undeterministic world.” Better to fail predictably than succeed unpredictably.</p><h3>Monitoring while it runs</h3><p>In another terminal:</p><pre># Watch progress<br>watch -n 30 &#39;tail -n 20 PROGRESS.md; \<br>  echo; jq -r &quot;.features | map(select(.passes)) | length&quot; feature_list.json&#39;<br># Tail the agent&#39;s stdout<br>tail -f ~/.claude/projects/$(basename $PWD)/conversation.jsonl | jq -r &#39;.text&#39;<br># Watch cost in real time<br>claude usage --watch</pre><p>You wake up to either a PR queue, a NEEDS_WORK queue with the evaluator’s findings ready to feed back in, or — worst case — --max-iterations hit and a clean halt.</p><h3>Cost: what’s actually on the meter</h3><p>This is where I have to be careful. The numbers I’ve seen quoted in similar posts are usually rough guesses, and the only authoritative cost data comes from Anthropic’s own Mar 2026 paper, which is for full-app builds, not single features.</p><p><strong>The published reference points:</strong></p><ul><li><em>Retro game maker, V1 three-agent harness (Opus 4.5):</em> 6 hours, $200.</li><li><em>DAW, V2 simplified harness (Opus 4.5):</em> 3h 50min, $124.70. Planner $0.46, build rounds $36–$71 each, QA rounds $3–$4 each.</li><li><em>Solo agent comparison (same DAW prompt):</em> 20 min, $9 but the output didn’t actually work.</li></ul><p>So a <em>full app build</em> with the harness is in the $100–$200 range with Opus 4.6/4.7. A single slice with a fresh-context evaluator round-trip is meaningfully smaller, but I can’t give you a precise number without telling you something I haven’t measured.</p><p><strong>Two cost levers that are real and load-bearing:</strong></p><ul><li><strong>Caching matters most.</strong> The baseline tax tool registry, system prompt, CLAUDE.md, skills index is roughly 30–50k tokens that load on every session. Cached, it drops ~10×. If you&#39;re not caching, you&#39;re leaving most of the cost on the floor.</li><li><strong>Route reads to Sonnet.</strong> Codebase exploration is read-heavy and doesn’t need Opus. The implementer-builder distinction is real money: Opus 4.7 is $5/$25 per Million Token , Sonnet 4.6 is $3/$15, Haiku 4.5 is $1/$5.</li></ul><p><strong>The Opus 4.7 tokenizer caveat.</strong> Opus 4.7 (released Apr 16, 2026) holds the same $5/$25 rate as 4.6 but uses a new tokenizer. The same input can map to 1.0–1.35× more tokens than under 4.6. The headline rate didn’t change; effective cost per request did. Audit your usage after upgrading.</p><p><strong>The expensive thing isn’t the model.</strong> It’s the engineering hours you spend untangling bad output. Spec quality dominates total cost. A 30-minute spec saves three hours of agent thrashing.</p><h3>What to do after reading this</h3><p>Four actions. Don’t do all four at once. Pick the one whose absence is hurting most.</p><p><strong>1. Pick one slice. Write a one-page spec.</strong> Intent + Context + Acceptance. 30 minutes max. Refuse to start coding without it. Notice what you didn’t realize was ambiguous until you tried to write it down.</p><pre>mkdir -p specs tests/conformance<br>$EDITOR specs/your-slice.md<br>$EDITOR tests/conformance/your-slice.yaml</pre><p><strong>2. Set up the harness.</strong> Copy the .claude/ directory from cwc-long-running-agents. Trim your CLAUDE.md to ≤60 lines. Wire the three Anthropic primitives. Add the evaluator subagent with no Write/Edit tools.</p><pre>git clone https://github.com/anthropics/cwc-long-running-agents /tmp/cwc<br>cp -r /tmp/cwc/claude-code-config/.claude .<br>chmod +x .claude/hooks/*.sh<br>$EDITOR CLAUDE.md<br>$EDITOR .claude/settings.json<br>git add .claude/ CLAUDE.md &amp;&amp; git commit -m &quot;harness: initial .claude scaffold&quot;</pre><p><strong>3. Start the agent in a worktree.</strong> One slice, one worktree, one PR. The parent repo stays clean. The slice ships or fails on its own.</p><pre>claude --worktree your-slice<br># inside the session:<br>&gt; /ship-slice your-slice</pre><p><strong>4. After the evaluator catches a failure, ratchet the lesson.</strong> Encode it as a hook, a skill, or a new section in your spec template. The next agent can’t make the same mistake. This is how the harness compounds — and it’s what separates a working setup from a stagnant one.</p><pre># Example: evaluator caught the agent committing a real value in .env.example<br>$EDITOR .claude/hooks/pre-tool-use-policy.sh<br>git add .claude/hooks/ &amp;&amp; git commit -m &quot;ratchet: block .env writes with real values&quot;</pre><h3>Reading list</h3><p>Five canonical sources, in this order:</p><ol><li><strong>Karpathy.</strong> <a href="https://x.com/karpathy/status/1886192184808149383">Agentic engineering — 1-year anniversary tweet</a>, Feb 4, 2026. The definition.</li><li><strong>Anthropic.</strong> <a href="https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents">Effective harnesses for long-running agents</a>, Nov 26, 2025. Two-agent framework.</li><li><strong>Anthropic.</strong> <a href="https://www.anthropic.com/engineering/harness-design-long-running-apps">Harness design for long-running application development</a>, Mar 24, 2026. GAN-inspired three-agent system, then its simplification on Opus 4.6.</li><li><strong>Anthropic.</strong> <a href="https://github.com/anthropics/cwc-long-running-agents">cwc-long-running-agents</a>, May 2026. The code companion — three primitives in ~400 lines.</li><li><strong>Huntley / Anthropic.</strong> <a href="https://ghuntley.com/ralph/">Ralph Wiggum</a> + <a href="https://github.com/anthropics/claude-plugins-official/tree/main/plugins/ralph-loop">official plugin</a>. The autonomous loop pattern, formalized.</li></ol><p>And the workflow tooling: GitHub <a href="https://github.com/github/spec-kit">Spec-Kit</a>, 90k+ stars as of May 2026.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6c6ec20ecc2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Argued Agentic Engineering Was the Fix for Vibe Coding. I Was Half Right.]]></title>
            <link>https://medium.com/@umairabro/i-argued-agentic-engineering-was-the-fix-for-vibe-coding-i-was-half-right-c875b3c4f6bc?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/c875b3c4f6bc</guid>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Thu, 07 May 2026 11:58:46 GMT</pubDate>
            <atom:updated>2026-05-07T11:58:46.129Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The four-layer stack that actually ships production code in 2026</em></p><p>A few weeks ago I argued that <a href="https://medium.com/@umairabro/agentic-engineering-the-contract-that-replaces-vibe-coding-22ceaa8f4a8e">Agentic Engineering: The Contract That Replaces Vibe Coding</a> , I was half right.</p><p>Vibe coding <em>is</em> a tech debt factory. that part holds. But agentic engineering alone isn’t the fix. It’s the <em>top</em> of a four-layer discipline that’s been crystallizing across the field for the last 18 months, and unless you understand all four layers, you’re going to keep hitting the same walls.</p><p>This is the follow-up. The vocabulary, the proof points, and the three concrete artifacts to copy.</p><h3>The proof point: 100,000 lines of working C compiler</h3><p>Let me start with the number that should make every tech leader stop scrolling.</p><p>In February, Nicholas Carlini at Anthropic published <a href="https://www.anthropic.com/engineering/building-c-compiler">a write-up</a> on building a working C compiler with Claude Code. The numbers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CQRkL5u56vxXcDxCaFkWPg.png" /></figure><ul><li><strong>100,000 lines</strong> of working C compiler. In Rust.</li><li><strong>16 Claude Opus 4.6 agents</strong> running in parallel.</li><li><strong>~2,000 Claude Code sessions</strong> across two weeks.</li><li><strong>~$20,000</strong> in API spend (2 billion input tokens, 140 million output tokens).</li><li>It compiles real-world C including a bootable <strong>Linux 6.9 kernel</strong> on x86, ARM, and RISC-V.</li></ul><p>Stop and read that again. A team of agents, running on a model anyone with an API key can use, built a non-trivial compiler from scratch in two weeks for less than the monthly burn of a single senior engineer.</p><p>This wasn’t a model release. It was a harness release. The capability was already in Opus 4.6 anyone could have called the same model and gotten nowhere close. What unlocked the result was the system around the model: the orchestration of 16 parallel workers, the context discipline, the feedback loops, the boundaries.</p><p>That system has a name now. Or rather, four names and that’s the discipline I want to lay out.</p><h3>The four-layer stack</h3><p>Here’s the mental model I’ve been using for the last six months. Read it bottom-up.</p><p><strong>Layer 1- Prompt engineering:</strong></p><p>What you say to the model. This is the layer everyone learned first. It’s also the layer most teams still operate at exclusively, even though it stopped being the bottleneck about a year ago.</p><p><strong>Layer 2 - Context engineering:</strong></p><p>What enters the model’s window. Anthropic’s Applied AI team called context “a precious, finite resource” in their <a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents">September 2025 post</a>. You can have the world’s best prompt, but if your context is junk too much, too little, the wrong shape the agent fails. Context engineering is the discipline of choosing what enters the window on purpose.</p><p><strong>Layer 3 - Harness engineering:</strong></p><p>The <em>system</em> around the model. Tools, sandboxes, hooks, sub-agents, sensors, guides, feedback loops, permissions, state, memory. This is the layer that turns “I asked an LLM” into “I have a working agent.” Mitchell Hashimoto coined the term in his <a href="https://mitchellh.com/writing/my-ai-adoption-journey">“My AI Adoption Journey” post on February 5, 2026</a> and it stuck.</p><p><strong>Layer 4 - Agentic engineering:</strong></p><p>The discipline that owns all three layers underneath. The human practice of building, observing, and improving the system over time. This is what I wrote about last time and what I now realize was incomplete on its own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIBrPonLpgEI8QPGHIUBLw.png" /></figure><p>Most teams I talk to operate at Layer 2. The frontier operates at Layer 3 with intention. The discipline is moving up the stack and the gap between teams that have made the move and teams that haven’t is becoming the dominant performance variable, even when everyone is using the same model.</p><h3>Same model. Different harness. The harness is the moat.</h3><p>If you only believe one stat from this article, believe this one.</p><p>In March, Stanford’s IRIS Lab and MIT released <a href="https://arxiv.org/abs/2603.28052">Meta-Harness</a>, a paper on automatically optimizing the system around a fixed model. They took the same Claude Opus 4.6 model and ran two harnesses against TerminalBench-2’s 89 hard, real-world coding tasks.</p><p>The hand-engineered one Terminus-KIRA, the strongest open baseline for that model class hit 74.7%.</p><p>The auto-evolved harness Meta-Harness discovered same model, just a smarter system around it hit 76.4% and ranked #2 across all Opus 4.6 agents on the official leader board. On the cheaper Claude Haiku 4.5, it ranked #1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CbKOYS_zSHhqNiveawTyhg.png" /></figure><p>That’s not a huge absolute gap on this specific benchmark. But the spread across published Opus 4.6 harnesses on the same leader board is dramatic:</p><p>Claude Code’s stock harness scores 58.0%, while the best hand-tuned harnesses break 80% that’s a ~1.4× spread on the same model from harness changes alone. Stanford’s paper opens with an even more aggressive claim borrowed from prior work that changing the harness can produce up to a 6× performance gap on a benchmark (Tian et al., SWE-bench Mobile, 2026).</p><p>The exact multiplier doesn’t matter. What matters is that the gap exists, it’s large, and it’s reproducible.This is the experiment that ends every “we just need a better model” debate. The model is becoming the commodity. The harness is becoming the moat.</p><p>If you’re a tech leader reading this, sit with that for a second. Your competitive advantage in 2026 isn’t going to come from picking GPT-5.5 over Opus 4.7. It’s going to come from the system you build around whichever model you pick.</p><h3>Three artifacts to ship this week</h3><p>The four layers stop being theoretical the moment you build them. Here are three concrete artifacts.</p><h4>1. A real CLAUDE.md</h4><p>CLAUDE.md is the file Claude Code reads at the start of every session. Most teams either don’t have one, or they ran /init once and forgot it existed. Both are mistakes.</p><p>The pattern that works:</p><p>markdown</p><pre># CLAUDE.md</pre><pre>## Stack<br>React 18 · TypeScript · Vite · Tailwind</pre><pre>## Commands<br>Build: pnpm build<br>Test: pnpm test<br>Lint: pnpm lint --fix</pre><pre>## Structure<br>src/ · tests/ · docs/</pre><pre>## Style<br>Functional components only. Tokens, not hex.</pre><pre>## Boundaries<br>✅ Always — run tests, follow naming conventions<br>⚠️ Ask first — schema changes, new dependencies<br>🚫 Never — commit secrets, edit /vendor or CI config</pre><p>Three principles I’ve landed on after about a hundred CLAUDE.md iterations:</p><ol><li>Encode only what Claude can’t infer from the code itself. If your package.json says react@18, don&#39;t write &quot;we use React 18&quot; in CLAUDE.md. Encode the judgments what conventions you&#39;ve chosen, what trade-offs you&#39;ve made, what you&#39;ve explicitly decided not to do.</li><li>Nest CLAUDE.md per package. Closest file wins. Mono-repos benefit from this most.</li><li>Version-control it. Review changes like code. It’s a system artifact, not a sticky note.</li></ol><p>The boundary pattern (✅ ⚠️ 🚫) .It works because it gives you three slots <em>always</em>, <em>ask first</em>, and <em>never </em>and the middle slot is where most of the productive negotiation happens.</p><h3>2. Three deterministic hooks</h3><p>Prompt instructions get somewhere between 70–90% compliance. Hooks get 100. When the rule is “never delete .env,” usually isn’t enough.</p><p>Three hooks every Claude Code team should ship in .claude/settings.json:</p><p><strong>PreToolUse block secrets in writes</strong></p><pre>{<br>  &quot;matcher&quot;: &quot;Edit|Write&quot;,<br>  &quot;hooks&quot;: [{<br>    &quot;type&quot;: &quot;command&quot;,<br>    &quot;command&quot;: &quot;./hooks/no-secrets.sh&quot;<br>  }]<br>}</pre><p>Scan content before any Edit or Write. Block on hardcoded keys, tokens, .env paths.</p><p><strong>PostToolUse auto-format every edit</strong></p><pre>{<br>  &quot;matcher&quot;: &quot;Edit|Write&quot;,<br>  &quot;hooks&quot;: [{<br>    &quot;type&quot;: &quot;command&quot;,<br>    &quot;command&quot;: &quot;prettier --write $FILE &amp;&amp; eslint --fix $FILE&quot;<br>  }]<br>}</pre><p>Run prettier and the linter on every saved file. Failures are fed back to the agent as context the agent learns what passed and what didn’t.</p><p><strong>Stop Gate completion on tests</strong></p><pre>{<br>  &quot;hooks&quot;: [{<br>    &quot;type&quot;: &quot;command&quot;,<br>    &quot;command&quot;: &quot;npm test || exit 2&quot;<br>  }]<br>}</pre><p>Block “task complete” until the suite is green. Forces the agent to fix breakage, not declare victory.</p><p>The pattern across all three: <strong>write hook error messages so they teach the fix.</strong> Every failure becomes context for the next attempt ,It changes everything once you internalize it.</p><p>Boundaries belong in code, not in polite requests in CLAUDE.md.</p><h3>3. One sub-agent for the noisy work</h3><p>Sub-agents are context firewalls. Long sessions fill the context window with multi-thousand-line tool outputsgit diff against a large branch, a full test log, an exploratory grep across the repo and the agent starts looping. The fix isn&#39;t a bigger context window. It&#39;s moving the noisy work to a <em>separate</em> context window. Only the summary returns to the parent.</p><p>The shape, in .claude/agents/code-reviewer.md:</p><pre>---<br>name: code-reviewer<br>description: Senior code reviewer. Use proactively after writing code.<br>tools: Read, Grep, Glob<br>disallowedTools: Edit, Write<br>model: sonnet<br>---<br><br>You are a senior code reviewer.<br>When invoked:<br>1. Run `git diff` to see changes<br>2. Focus analysis on modified files<br>3. Output: actionable feedback with file refs<br></pre><p>A few things to notice:</p><ul><li>disallowedTools: Edit, Write this reviewer is read-only by design. The harness enforces the role boundary.</li><li>model: sonnet cheaper, faster model for the focused task. Save Opus for the orchestration.</li><li>description is the most important field. Most sub-agent failures are <em>invocation</em> failures, not execution failures write the description like a one-line job posting.</li></ul><p>Use sub-agents when exploration would flood parent context, when reviews can run in parallel, when a cheaper model is good enough, or when you need restricted tool permissions. Don’t use them for single tool calls or for tasks where one step’s output frames the next.</p><h3>The harness loop in action</h3><p>Let me walk you through how this actually plays out, with a single example, before generalizing.</p><p><strong>Monday morning.</strong> You’re using Claude Code to ship a feature. Halfway through, the agent declares “task complete.” You merge. CI breaks. Tests didn’t pass.</p><p><strong>Wednesday.</strong> Different feature, same agent, same codebase. The agent declares “task complete.” CI breaks again. Different reason this time, but the underlying behavior is identical: the agent is calling itself done before verifying its work.</p><p>This is the moment that matters. You have two options.</p><p><strong>Option A</strong> is to update CLAUDE.md. Add a line that says <em>“Always run tests before declaring task complete.”</em> The agent will read it. Most of the time, it’ll comply. Some of the time usually the time that hurts most, when context is full and the session is long it won’t.</p><p><strong>Option B</strong> is to encode the rule in code. Three lines in .claude/settings.json:</p><pre>{<br>  &quot;hooks&quot;: {<br>    &quot;Stop&quot;: [{<br>      &quot;hooks&quot;: [{<br>        &quot;type&quot;: &quot;command&quot;,<br>        &quot;command&quot;: &quot;npm test || exit 2&quot;<br>      }]<br>    }]<br>  }<br>}</pre><p>Now the agent <em>cannot</em> declare completion until tests pass. The hook fires every Stop event. Exit code 2 mechanically blocks the action. The behavior isn’t a request anymore ,it’s a property of the system.</p><p>That’s the difference, and it’s bigger than it looks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J7zcn9jKZYUE_eBtNFhjEw.png" /></figure><p>The compliance gap isn’t a Claude Code quirk. It’s a property of probabilistic systems. Anything that must happen has to happen outside the LLM’s reasoning chain. That’s what hooks are for.</p><p>But the hook isn’t the interesting part. The loop is the interesting part.</p><h3>The loop</h3><p>What you just did has four steps. Together they form a cycle that repeats every time you catch a new failure mode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yt_fv1KjHPGO61m-wtSnNA.png" /></figure><p>You observed a pattern (a failure happening twice once is noise, twice is signal). You diagnosed why the prompt-level fix wouldn’t hold (probabilistic compliance under pressure). You encoded the rule mechanically (a Stop hook). You verified the failure stopped happening and immediately started looking for the next pattern.</p><p>That’s the loop. It runs forever. Every time it completes, your harness gets a little stronger and your agent gets a little more reliable.</p><h3>Four worked examples</h3><p>Here are four runs through the loop from the last quarter. The pattern is identical every time observe, diagnose, encode, verify but the encoding mechanism varies based on what kind of failure you’re closing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kr7qxJhlnKHjfG5jSxWyqw.png" /></figure><p>Notice what’s <em>not</em> on this list: “rewrote the prompt to be clearer,” “added more examples to CLAUDE.md,” “asked Claude to be more careful.” Those are Layer 1 fixes. They don’t compound. The Layer 3 fixes do.</p><p>Every encoded failure makes your harness a little more capable than it was yesterday. That’s the compounding asset. Six months in, your harness is a record of every failure your team has ever caught and a guarantee that none of them happen again.</p><p>This is the loop, and the loop is the discipline. You don’t get good at agentic engineering by writing better prompts. You get good at it by closing the gap between observed failures and encoded fixes one cycle at a time.</p><h3>Where you actually are</h3><p>Hands On Architects published a <a href="https://handsonarchitects.com/blog/2026/the-harness-model-ai-engineering-maturity-matrix">5-stage maturity matrix</a> in Q1 2026. Calibrate yourself honestly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DHzIcZIBCExfFe2yj-v7qQ.png" /></figure><p>There’s also a brownfield reality check that matters more than the matrix itself. Stages 1–3 work the same on brownfield and greenfield. Stage 4 doesn’t. Stage 4 needs architectural cleanliness most production codebases don’t have.</p><p>Three honest questions to ask yourself before you target Stage 4:</p><ul><li>Can you name what each module owns in <em>one sentence</em>?</li><li>Are dependencies <em>explicit and directional</em>?</li><li>Can agents verify changes <em>without you</em>?</li></ul><p>If any answer is “no” and for most production codebases I’ve seen, most answers are “no” you’re realistically aiming at Stage 3. That’s still meaningful progress. Stage 3 is where teams stop manually shepherding every prompt and start operating at a different gear.</p><p>Don’t lie to yourself about Stage 4. It’s not a tooling problem. It’s an architectural-cleanliness problem, and no amount of Claude Code wizardry fixes it.</p><h3>The bottom line</h3><p>Vibe coding is still a tech debt factory. Agentic engineering is still part of the fix. But it’s not the whole fix.</p><p>The whole fix is a four-layer discipline: prompt → context → harness → agentic engineering. Most teams operate at Layer 1. The frontier operates at Layer 3 with intention. The work is moving up the stack, and the teams that don’t make the move will keep wondering why their AI investments produce more tech debt instead of less.</p><p>Three things to do this week: a real CLAUDE.md, three hooks, one sub-agent. That’s the smallest viable harness investment that compounds.</p><p>The capability is already in the model. The leverage is in the system around it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c875b3c4f6bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic Engineering: The Contract That Replaces Vibe Coding]]></title>
            <link>https://medium.com/@umairabro/agentic-engineering-the-contract-that-replaces-vibe-coding-22ceaa8f4a8e?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/22ceaa8f4a8e</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 12:51:10 GMT</pubDate>
            <atom:updated>2026-04-14T12:51:10.719Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9KnIcB3UaTOzda6wCbwWtg.png" /></figure><p>Two years ago, I shipped a feature in two hours using Cursor. Prompt, accept, test, push. Fast enough to feel like cheating.</p><p>Last month, I kicked off a Claude Code session before bed. The brief: ingest three Zoho CRM modules, infer the schemas, land them as Delta Lake upserts, wire up a dbt transformation layer. By morning, the agent had generated 14 files across four directories, run the test suite twice, caught and resolved two failures without asking me a single question, and opened a PR that passed CI on the first try.</p><p>The difference between those two moments isn’t speed. It isn’t a better model. It’s an entirely different engineering discipline, one where I didn’t write a single line of code, but owned every architectural decision behind it.</p><p>If you lead a technical team, this is the shift that will define your 2026.</p><h3><strong>The Karpathy arc: from vibes to engineering</strong></h3><p>Here’s the shortest version of the last two years in AI-assisted development:</p><ul><li>February 2025: Andrej Karpathy tweets “vibe coding” into existence, prompt an LLM, accept the output, don’t look at the code, just keep going. It was intoxicating. Every engineer felt 10x overnight.</li><li>February 2026: Karpathy declares vibe coding <em>passé</em> and replaces it with <em>agentic engineering “you are not writing the code directly 99% of the time, you are orchestrating agents who do and acting as oversight.”</em></li></ul><p>One year. Same person. Completely different thesis. What happened?</p><p>Three things converged:</p><ul><li><strong>Models learned to act, not just speak.</strong> Claude, OpenAI, and Gemini gained the ability to execute code, call tools, browse the web, and loop on their own errors. They crossed the line from text generators to autonomous agents.</li><li><strong>Tooling made autonomy practical.</strong> Claude Code, OpenAI Codex, and Gemini CLI , none of which existed in early 2025 turned these capabilities into real development environments. Not autocomplete. Autonomous execution loops that write, test, debug, and iterate without you in the chair.</li><li><strong>Production exposed the lie.</strong> Every team that scaled vibe coding past the prototype stage learned the same brutal lesson. Unreviewed LLM-generated code doesn’t just have bugs , it has <em>architectural cancer</em>. Inconsistent patterns compounding across files. Implicit dependencies nobody intended. Logic errors subtle enough to pass the vibe check and catastrophic enough to wake you up at 3 AM.</li></ul><p>Agentic engineering exists because vibe coding broke at scale. The answer isn’t to hand everything to the AI. It isn’t to go back to writing every line by hand. It’s a new contract: the human owns the architecture, the agents own the implementation, and engineering discipline is what keeps the whole thing from falling apart.</p><h3>The contract: who owns what</h3><p>Agentic engineering isn’t a tool. It isn’t a framework. It’s a <strong>contract</strong> — a division of responsibility between the human, the agent, and the engineering system that binds them.</p><p><strong>The human owns the architecture.</strong> You decide what gets built, how components connect, which tradeoffs are acceptable, and where the hard boundaries are. You write the spec. You define the interfaces. You decide that this service talks to that database through this API not the other way around. None of this gets delegated. Ever</p><p><strong>The agent owns the implementation.</strong> It writes the code, runs the tests, debugs the failures, iterates toward a solution, and opens the PR. It can do this across dozens of files, for hours, without asking a single question, provided the spec is precise enough.</p><p><strong>Engineering discipline owns the bridge.</strong> Tests, CI pipelines, circuit breakers, token budgets, hard iteration limits these are the guardrails that prevent the contract from collapsing. Without them, you don’t have agentic engineering. You have an unsupervised intern with root access and zero self-doubt.</p><p>Simple to state. Hard to execute. Here’s what the contract demands in practice.</p><h4>1. The spec is the product</h4><p>In traditional development, specs were documentation useful in theory, routinely ignored, written after the fact half the time. In agentic engineering, the spec is the single most important artifact you produce. It isn’t documentation. It’s the <strong>input to your entire implementation pipeline.</strong></p><p>A vague spec when you’re pair-programming with Cursor just means extra iteration cycles. Annoying, but recoverable you’re still in the loop, course-correcting in real time.</p><p>A vague spec in an agentic workflow produces something far worse: a <strong>confidently wrong system that passes its own tests.</strong> The agent won’t flag ambiguity. It won’t pause to ask for clarification. It will make assumptions and ship them. And because the agent writes the tests too, those assumptions become self-reinforcing , the code passes because the tests were built around the same flawed interpretation of your intent.</p><p>Here’s what a good agentic spec actually looks like:</p><ul><li><strong>Architecture constraints</strong> : <em>“This pipeline must follow Delta Lake bronze/silver/gold layers. No direct writes to gold.”</em></li><li><strong>Interface contracts</strong> : <em>“The ingestion module exposes a </em><em>sync(source_id, since_timestamp) function. Returns a </em><em>SyncResult with </em><em>records_upserted, </em><em>records_failed, and </em><em>errors.&quot;</em></li><li><strong>Explicit edge cases: </strong><em>“If Zoho returns a 429, back off exponentially, max 3 retries. If it returns a 401, refresh the OAuth token exactly once. If the refresh fails, halt the run and alert.”</em></li><li><strong>Acceptance criteria as executable tests: </strong><em>“Done means: (a) incremental sync processes only records modified after the last watermark, (b) duplicate record IDs resolve by latest modified timestamp, 10K records complete in under 4 minutes.”</em></li></ul><p>In the teams I’ve seen executing this well, engineers spend 40–60% of their time on the spec before an agent writes a single line of code. That ratio feels backwards at first. It isn’t. The spec is where the engineering happens now. Everything after it is execution.</p><h4>2. Tests are the control plane, not the safety net</h4><p>This is the most counterintuitive shift in agentic engineering, and I cannot overstate it:</p><blockquote><strong><em>Your test suite is the leash. The quality of your tests determines the ceiling of your agent’s autonomy.</em></strong></blockquote><p>An agent backed by a robust test suite can iterate in a tight loop : write, run, fail, diagnose, fix, re-run — until every assertion passes. That loop can execute 50 times at 3 AM while you sleep, and you wake up to a working PR you can trust. The tests aren’t just verifying the code. They’re <strong>steering</strong> the agent. Every failing assertion is a concrete signal that redirects the next attempt.</p><p>An agent without tests has no feedback signal at all. It will generate plausible-looking code, declare itself finished, and open a pull request that reads beautifully , then silently corrupt your data for three weeks before anyone notices.</p><p>The math is straightforward: every hour invested in writing tests before handing work to an agent saves roughly five hours debugging agent output after. That’s the consistent pattern across every agentic deployment I’ve built or reviewed this year.</p><p>Write the tests first. Write them before the spec is fully finalized. Make them executable, not aspirational. Then turn the agent loose inside those rails.</p><h4>3. You review architecture, not diffs</h4><p>Traditional code review: you read the diff line by line syntax errors, logic bugs, style violations, missed edge cases.</p><p>Agentic code review: the diff is 500 lines across 14 files. The agent already iterated on it 12 times against the test suite. Syntax is clean. Style is consistent. Tests pass. Reading it line by line isn’t just impractical it’s the wrong frame entirely.</p><p>The questions that matter in an agentic review are fundamentally different:</p><ul><li><strong>Did the agent respect the boundaries in my spec?</strong> Or did it take a shortcut that tightly couples two modules I explicitly wanted decoupled?</li><li><strong>Are the tests testing the right invariants?</strong> Or did the agent write assertions that pass by definition verifying that the code does what it does, rather than what it should do?</li><li><strong>Did it introduce dependencies I didn’t authorize?</strong> A new pip package. An API call to an external service. A database write path that bypasses the abstraction layer I designed.</li><li><strong>Would I be comfortable with this running unsupervised for six months?</strong> Not “does it work right now” does it <em>survive</em> six months of edge cases, schema changes, and traffic spikes without someone babysitting it?</li></ul><h4><strong>4. Cost is an architecture constraint, not an ops metric</strong></h4><p>When you’re pair-programming with AI, costs are predictable roughly proportional to how many developers have Cursor seats. Easy to budget. Easy to forecast.</p><p>Agentic costs follow a different curve entirely. They scale with <strong>loop depth × context window size × number of parallel agents.</strong> And the variance is punishing a single edge case can trigger a retry chain that costs 50x the normal execution path.</p><p>I wrote about this in detail in my previous piece on <a href="https://medium.com/@umairabro/how-to-build-bankrupt-proof-ai-systems-in-2026-2ad99ebb8100">building bankruptcy-proof AI systems</a>, and every pattern I described there hits harder in agentic workflows. Tiered model routing. Circuit breakers. Budget-aware degradation. In agentic engineering, these aren’t optimizations you add later. They’re as fundamental to your system as your choice of database.</p><p>The logic is inescapable: if the agent owns the implementation, and the implementation runs autonomous loops that consume real money with every iteration, then <strong>cost governance is an architectural decision </strong>not a line item you discover on next month’s invoice.</p><p>Set token budgets per agent per session. Set hard iteration limits at the node level. Hash the agent’s state on every cycle and trip a circuit breaker if it produces the same hash twice it’s stuck in a loop, and every additional cycle is burning money for nothing. Downgrade to a cheaper model if the session crosses a budget threshold mid-run.</p><p>These aren’t performance tuning. They’re load-bearing walls.</p><h3><strong>What this looks like in data and AI engineering</strong></h3><p>Most of the agentic engineering conversation lives in the frontend and backend world shipping features, building UIs, deploying services. But the contract hits differently in data and AI engineering, where the stakes are quieter and the failures are slower to surface.</p><h4><strong>Pipeline development is becoming declarative</strong></h4><p>Instead of hand-writing Spark jobs and dbt models line by line, engineers are describing pipeline <em>intent </em>source systems, ingestion patterns, transformation logic, quality thresholds and letting agents generate the implementation. The DAG definitions, the notebook code, the schema migrations, the test fixtures. What used to take a data engineer a full sprint now takes a well-written spec and an overnight agent session.</p><p>The catch: pipeline code that <em>looks</em> correct can silently produce wrong data for weeks. A misinterpreted join condition, a timezone offset nobody specified, a deduplication strategy that resolves the wrong way. These aren’t the kinds of bugs that crash at runtime. They’re the kind that quietly poison every dashboard and model downstream. In pipeline work, the spec and the tests aren’t just important they’re existential.</p><h4><strong>System integrations are the perfect stress test for the contract.</strong></h4><p>CRM migrations, ERP syncs, third-party API connectors this is exactly the tedious, rules-heavy, context-dense work that agents handle remarkably well. Schema mapping, field translation, OAuth token management, retry logic for rate-limited APIs. An agent can generate hundreds of lines of integration glue in minutes.</p><p>But integration work is also where a single wrong field mapping can silently corrupt your data sync for weeks before anyone notices. The speed is a gift. The overconfidence is a trap. Every integration built by an agent needs guardrails that are proportional to the blast radius of a silent failure in data systems, the blast radius is almost always larger than it looks.</p><h4><strong>Modern data platforms are becoming agent runtimes.</strong></h4><p>Databricks, Snowflake, and the broader lakehouse ecosystem are evolving beyond pipeline orchestration into full agentic infrastructure environments where agents can query data, invoke models, trigger workflows, and act on results within a governed platform. That’s powerful. It also introduces a cost model most teams aren’t prepared for. Every agent loop that hits a warehouse query or a model serving endpoint consumes real compute. If the agent retries a failing SQL query in a reasoning loop, your compute bill doesn’t scale linearly it spikes. Observability at the agent-trace level, not just the pipeline level, is the infrastructure investment most data teams aren’t making yet. It will be the one they wish they’d made first.</p><h3>One thing to do this week</h3><p>If you’ve read this far, here’s a single concrete action:</p><p>Pick one task your team does repeatedly something that takes 2–4 hours of engineer time and try running it under the contract. Write a spec with explicit acceptance criteria. Write the tests first. Hand it to Claude Code or Codex. Don’t touch the keyboard. See what comes back.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22ceaa8f4a8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Bankruptcy-Proof AI Systems in 2026]]></title>
            <link>https://medium.com/@umairabro/how-to-build-bankrupt-proof-ai-systems-in-2026-2ad99ebb8100?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ad99ebb8100</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Tue, 24 Feb 2026 08:10:18 GMT</pubDate>
            <atom:updated>2026-02-24T09:34:40.925Z</atom:updated>
            <content:encoded><![CDATA[<h4>Three engineering strategies to control exponential costs in multi-agent ecosystems.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfbYDaIf8oTYeHgfom4oHA.png" /></figure><h3>The Reality Check: An O(2^n) Billing Problem</h3><p>In 2023, LLM integration was essentially stateless RPCs. A user sent a string, we appended it to a prompt template, sent it to an API, and returned the completion. It was O(1) in terms of tool invocation. Cost: ~$0.03.</p><p>Today, in 2026, we are building multi-agent state machines on orchestrators like Bedrock or Databricks Agent Bricks. We’ve replaced deterministic logic with probabilistic routing.</p><p>Let me share a post-mortem from a recent deployment. I asked our internal Knowledge Agent a boolean question about a vendor contract.</p><p>1. The <strong><em>Supervisor node</em></strong> routed the query to the <strong><em>Researcher node</em></strong>.<br>2. The Researcher executed a naive vector search, dumping 5000 input tokens of historical contracts into context.<br>3. The <strong><em>Summariser node</em></strong> drafted a response.<br>4. The <strong><em>Critic node</em></strong> (configured with a standard ReAct/Reflexion loop) threw a validation error on formatting and pushed the state back to the Researcher.</p><p>Because i didn’t implement a hard iteration limit or state-hash checker, the loop ran recursively. For 45 minutes, the agents re-processed those 5000 tokens on every single turn.</p><p>That simple boolean question cost us $<strong><em>26.45 in API tokens</em></strong>.</p><p>AI costs in agentic workflows don’t scale linearly with user traffic; they scale exponentially with internal system complexity. If we don’t architect financial guardrails at the application layer, perfectly functioning code will autonomously drain our infrastructure budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FfjaF0gbfTAa84_JsB5ayA.png" /></figure><h3>Telemetry: Where the Token Leaks Actually Happen</h3><p>When you audit your traces (and if you aren’t running LangSmith, Braintrust, Phoenix, or MLflow stop reading and go set that up), the budget isn’t bleeding from user prompts. It’s bleeding from orchestration overhead.</p><h3><strong>The Orchestration &amp; Model Tax</strong></h3><p>Every time an agent uses a tool, it generates hidden “thinking” tokens.</p><ul><li>In <strong>AWS Bedrock</strong>, the ReAct prompt structure forces the model to generate a Chain-of-Thought execution plan before touching your API. You pay for those tokens.</li><li>If you are running <strong>Claude Sonnet 4.6</strong> at $3.00/1M input and $15.00/1M output, a 50-step loop trying to fix a bad OpenSearch query isn’t just slow, it’s actively burning dollars. Output tokens cost 5x more than input tokens, verbose “reasoning” loops are financially toxic.</li><li>In <strong>Databricks</strong>, agents routing to SQL or Unity Catalog consume DBUs for every trace, evaluation, and logging event. If an agent loops on a bad SQL query <strong>(Write → Error → Think → Write),</strong> your DBU consumption spikes exponentially.</li></ul><h3>Context Creep (The “Prompt Stuffing” Anti-Pattern)</h3><p>Just because Gemini 3 Pro can handle 2 to 10 million tokens doesn’t mean you should use them, relying on lazy vector retrieval and dumping entire unoptimized document clusters into the context window. If your agent requires 5 steps to solve a problem, and you shoved a 50,000-token PDF into the prompt, you are paying to read that PDF 5 times. It is the most expensive CTRL+F operation on the planet.</p><h3>The Architecture Fix: The 3 Pillars of Resilient AI</h3><p>To fix this, we need to stop treating frontier LLMs like basic compute. We need strict routing, aggressive filtering, and hard-coded circuit breakers.</p><p><strong>Pillar 1: Tiered Routing (Stop Using GPT-5 for Regex)<br></strong>Move to a hybrid SLM + LLM architecture. We need a semantic router middleware sitting in front of our models.</p><ul><li><strong>L0-Semantic Cache</strong>: Check Redis/ElastiCache for vector intent. If the user asks for a refund policy and we’ve answered a semantically identical query in the last 24 hours, serve the cached string. <strong>Cost: $0</strong>.</li><li><strong>L1-SLMs (The Workhorses):</strong> Route basic extraction, classification, and summarization to sub-10B parameter models (Llama 3 8B, Phi-4, Claude 4.6 Haiku, or GPT-5). Expect sub-100ms latency at pennies per million tokens.</li><li><strong>L2-Frontier Models (The Heuristics Engine):</strong> Only escalate to Claude Opus 4.6, GPT-5.3, or Gemini 3 Pro for complex, multi-step reasoning.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8vCP4-7yoRrbnBhZbJQFMQ.png" /></figure><p><strong>Pillar 2: Precision RAG Pipeline</strong><br>Stop sending the whole book. We must implement a multi-stage retrieval pipeline:</p><ul><li><strong>Semantic Chunking &amp; Parent-Child Retrieval:</strong> Split data logically (by markdown headers), not arbitrarily. Index highly specific “child” chunks for accurate vector matching, but inject the broader “parent” context into the prompt.</li><li><strong>Hybrid Search + Cross-Encoder Reranking: </strong>BM25 (keyword) + Vector search is mandatory. Do not pass the raw K-nearest neighbours to the LLM. Pass the top 50 results through a fast reranker model and only send the top 3 heavily-scored chunks to the frontier model.</li><li><strong>Context Distillation:</strong> If you <strong><em>must</em></strong> query 50 documents, route them through an L1 SLM first to map-reduce them into a dense 500-token summary *before* the L2 model sees it.</li></ul><p><strong>Pillar 3: Defensive Engineering &amp; Circuit Breakers<br></strong>API rate limits protect the provider’s infrastructure. We need guardrails that protect our budget.</p><ul><li><strong>Hard Iteration Limits:</strong> Implement absolute MAX_TURNS variables at the node level. An agent gets 3 attempts. If it fails, it throws an exception and halts. No infinite while-loops.</li><li><strong>State Hashing:</strong> Compute a hash of the agent’s input, context, and selected tool parameters. If the agent generates an identical state hash twice in a single session, trip the circuit breaker. It’s stuck in a loop. Kill the run.</li><li><strong>Budget-Aware Downgrading:</strong> Inject token trackers into the session middleware. If a single support ticket burns past a $0.40 internal budget, the system dynamically revokes access to the L2 frontier model and forces the agent to finish the task using an L1 model (Graceful Degradation).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*25x21eRt8m-2IXwmJaBYoA.png" /></figure><h3>Infrastructure: The Build vs. Buy</h3><p>Finally, a note on compute economics versus token economics.<br>Never try to self-host open-source models just because the weights are “free.” When you move from APIs to bare-metal GPUs (or AWS EC2/RunPod), you shift from paying per token to paying per hour.</p><ul><li>Use Managed APIs (OpenAI, Anthropic, Bedrock, Databricks Provisioned Throughput) for spiky traffic, rapid prototyping, and complex reasoning. If your traffic drops to zero at 3 AM, your cost should drop to zero.</li><li>Self-Host only if we have strict air-gapped data sovereignty requirements, or if we can guarantee <strong><em>&gt;60% sustained GPU utilisation 24/7</em></strong>. If a dedicated inference cluster sits idle for 14 hours a day, the compute overhead will bankrupt us faster than the APIs ever would.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ad99ebb8100" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Bazaar’s ML Platform]]></title>
            <link>https://medium.com/bazaar-tech/building-bazaars-ml-platform-462e716688ec?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/462e716688ec</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Thu, 15 Sep 2022 07:19:17 GMT</pubDate>
            <atom:updated>2022-12-26T14:31:29.327Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Platform, Empowering Data Intelligence</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sj92xV5Wpjke0kBj" /></figure><p>Bazaar’s unwavering ambition is to be the best retail super-app in Pakistan that adds value to the traditional retail industry. To accomplish that, the app experience must be flawless for every Bazaar service. Let’s take our frequently used customer app as an example. We can try out dynamic pricing for incentive-based deals or can experiment with recommendations for accomplishing an incentive-based offer.</p><p>Bazaar has enormous data managed with our in-house Data Platform <a href="https://medium.com/bazaar-tech/building-bazaars-data-platform-83d132d4168b"><strong>Buraq</strong></a> that we can leverage to solve complex problems such as pricing strategy, effective supply chain, fraudulent user activity, and to provide our customers personalized experiences on our products. Machine learning (ML) is one of the tools we use to make sense of this data.</p><p>As Bazaar is making tremendous strides towards increasing the utilization of machine learning across the organization, more and more squads are organically considering building models to solve for their use cases.</p><p>Unfortunately, these serving model solutions demanded data scientists to understand the underlying infrastructure. Moreover, there is a lot of overlapping in the effort it took to build these model-serving solutions.</p><p>That’s why we embark on a new journey to build <strong>GreenSight</strong>: an easy-to-use, self-serve, end-to-end machine learning platform for everyone at Bazaar.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FWOYKaXG2xJsBO%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2F80s-shows-WOYKaXG2xJsBO&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2FWOYKaXG2xJsBO%2Fgiphy.gif%3Fcid%3D790b76117d0a56b06b694222a706a8c4bd276a1809592040%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f947e3eee60a39d0fa5ebe63f6b5bdd/href">https://medium.com/media/3f947e3eee60a39d0fa5ebe63f6b5bdd/href</a></iframe><p>As always, we wanted to be as lean as possible but also learn from other’s past experiences and have a long-term vision and solid base. So this time, we started with defining the success metrics for our MLOps practice; the general idea was to tie up our platform performance KPIs to actual business value.</p><p>We were inspired by <a href="https://www.linkedin.com/in/doppenheimer/">Diego Oppenheimer’s</a> take on MLOps and contrived our metrics from some of his descriptions. We want to measure <strong>impact</strong> and <strong>value.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j17WO0BOHXeI_yYGhNJjlQ.png" /><figcaption>*will follow with actual numbers and examples in a separate blog</figcaption></figure><p>To implement the platform, we broke the ML lifecycle into modules for creating it using a product mindset. We are building on top of <a href="https://medium.com/bazaar-tech/building-bazaars-data-platform-83d132d4168b"><strong>Buraq</strong></a>,<strong> </strong>which utilizes <a href="https://www.databricks.com/glossary/data-lakehouse#:~:text=What%20is%20a%20Data%20Lakehouse,(ML)%20on%20all%20data."><strong>Data Lakehouse</strong></a><strong> </strong>architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A9YWHuROTta1dMWAlxosDw.png" /><figcaption>10,000 ft view of buraq &amp; greensight</figcaption></figure><p>GreenSight has the following layers.</p><ul><li>Training Engine</li><li>Model &amp; Feature Artifactory</li><li>Data Access SDK</li><li>Notebooks &amp; Environments</li><li>A/B Engine</li><li>Inference Engine</li></ul><h3>Training Engine</h3><p>Training Engine is the module responsible for end-to-end automation for our ML Platform. After the algorithm is completed, the CI can trigger the Airflow DAG factory that performs the following steps:</p><ol><li>Get the latest version of features from Feature Store</li><li>Get the latest data to Train, the model</li><li>Get the latest data to Test the model</li><li>Move the model specifications to A/B Engine</li><li>Generate Model Library to Call Model As A Function using Inference Engine.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3GP7nW_tbSJuj4JLEmY5Q.png" /></figure><h3>Model &amp; Feature Artifactory</h3><p>this module has two elements:</p><ul><li>storing features for reusability and standardization</li><li>storing models for inference</li></ul><p>Currently, for feature stores, we are just storing them as HUDI tables in future; we have plans to adopt tools like <a href="https://feast.dev/"><strong>FEAST</strong></a><strong> </strong>for better cataloguing and ML-specific APIs.</p><p>For models, we are currently exploring models as a functional approach where the inference engine can directly call specific models by calling their functions. The idea is every time a model is submitted to production, we can create an inference config, and that inference engine can call to route client queries to a specific model for inference.</p><h3>Data Access SDK</h3><p>Our Data Platform is built on lakehouse architecture, which means most of our data resides in the cloud buckets we run Trino cluster to query over the data. To make access abstract for GreenSight and reduce friction for users, we created a custom SDK to simplify data fetching into notebooks and training pipelines. For example, a data scientist has to pass relevant parameters to get the data into a data frame.</p><pre>from buraq.data import DataProducts<br><br>df = DataProducts.get(Catalog,DP,Days,Columns,Limit) </pre><p>We are continuously improving our SDK and adding valuable features to standardize ML practice in Bazaar. For example, any given dataset coming out of Buraq can be unbalanced, affecting the performance of a given model. To deal with these issues, we introduced their functions in our SDK.</p><pre>from buraq.model.tools import UpSample, DownSample, Balance<br>from buraq.data import DataProducts</pre><pre>df = DataProducts.get(Catalog,DP,Days,Columns,Limit)</pre><pre>up_sampled_df = UpSample(df,columns_to_consider)<br>down_sampled_df = DownSample(df,columns_to_consider)<br>balanced_df = Balance(df,columns_to_consider)</pre><h3>Notebooks &amp; Environments</h3><p>Notebooks have been commonplace in the AI community for most of the last ten years, and every data scientist has worked with one. They allow data scientists to rapidly experiment and share insights through quick environment creation, interactive output, and code snippets that can be executed in any order. Bazaar is no different. Our data scientists use it heavily in their workflow, plus most of our analytics teams utilize them to perform ad-hoc BI and what-if analysis.</p><p>For building notebooks service for GreenSight, we started with Jupyter Hub on our Kubernetes Clusters and provided custom docker images that contain tools for specific personas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asEJ1PxQ7snO8IwH8RaumQ.png" /></figure><p>Each environment can support a variety of instances and can satisfy both CPU and GPU-based workloads.</p><h3>A/B Engine</h3><p>A/B Engine is one of the main modules for GreenSight. It is designed to initially take care of three core functions.</p><ul><li>Cohort Management</li><li>Experiment Management</li><li>Model Performance</li></ul><h4>Cohort Management</h4><p>Bazaar has a variety of customers. We eventually want to group customers into cohorts based on attributes like varying target markets, purchasing power, locality and preference and personalize our model outputs to that specific cohort. The cohort Manager will have the functionality to collect and manage metadata for these cohorts that can be lookup by the inference engine and route the request to the specific model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MDl8WYUPQGmtnKl6VF9cHg.png" /><figcaption>greensight inference</figcaption></figure><h4>Experiment Management</h4><p>from the start, we decided to reuse our model in multiple use cases and tune the hyper-parameter. To effectively achieve this, we needed a way to perform AutoML. We are currently using <a href="https://www.kubeflow.org/docs/components/katib/overview/"><strong>Katib</strong></a><strong> </strong>to run our experiments, A/B test on models and push performance metric to Model Performance.</p><h4>Model Performance</h4><p>this module has two jobs, one to collect and display performance metrics for ML models. We achieve this by moving performance data into data products and using Apache Superset to display a dashboard for model performance.</p><p>Secondly, it helps the inference engine to decide which model hyper-tuned model to use for a specific cohort.</p><h3>Inference Engine</h3><p>The inference engine is just a set of APIs written in Kotlin to help Bazaar’s product apps to leverage GreenSight. It’s built on “Data As A Service” Model but is focused on ML use cases. It has three jobs.</p><ol><li>Figure out which cohort the incoming client is in.</li><li>Figure out which model to use for inference.</li><li>Record the inference shown to calculate metrics.</li></ol><p>See the GreenSight inference diagram for the workflow</p><p>We are still in heavy development and brainstorming session. Some things may change. We will continue sharing knowledge as we build or figure out new things.</p><h3>Disclaimer:</h3><p><a href="https://www.bazaartech.com/"><strong>Bazaar Technologies</strong></a> believes in sharing knowledge and freedom of expression, and it encourages it’s colleagues and friends to share knowledge, experiences and opinions in written form on it’s <a href="https://medium.com/bazaar-tech">medium publication</a>, in a hope that some people across the globe might find the content helpful. However the content shared in this post and other posts on this medium publication mostly describe and highlight the opinions of the authors, which might or might not be the actual and official perspective of Bazaar Technologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=462e716688ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bazaar-tech/building-bazaars-ml-platform-462e716688ec">Building Bazaar’s ML Platform</a> was originally published in <a href="https://medium.com/bazaar-tech">Bazaar Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Bazaar’s Data Platform]]></title>
            <link>https://medium.com/bazaar-tech/building-bazaars-data-platform-83d132d4168b?source=rss-3ea379767a43------2</link>
            <guid isPermaLink="false">https://medium.com/p/83d132d4168b</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-as-a-service]]></category>
            <dc:creator><![CDATA[Umair Abro]]></dc:creator>
            <pubDate>Mon, 25 Oct 2021 09:41:20 GMT</pubDate>
            <atom:updated>2021-10-25T09:41:20.144Z</atom:updated>
            <content:encoded><![CDATA[<h4>A platform, capable of solving Petabytes scale problems</h4><p>I recently joined <a href="https://www.bazaar-tech.com/">Bazaar Technologies</a>, a Pakistani startup with big dreams and super talented team to make those dreams a reality.</p><p>At Bazaar, many people rely on data every day to do their jobs. In just fifteen months we have added 6+ applications to support the business with dozens of micro-services at the back-end. Which serve more than 200+ brands and 750k+ merchants in Pakistan. Tons of data is generated every day, while there are some data solutions already present for analysis, we have started to overload them.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxT5LMPwPc1xYY4sLIY%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FxT5LMPwPc1xYY4sLIY%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FxT5LMPwPc1xYY4sLIY%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="326" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa5869fdbeb97dfd749764e6b66c0abc/href">https://medium.com/media/aa5869fdbeb97dfd749764e6b66c0abc/href</a></iframe><p>So, we embarked on the journey to re-imagine our data platform. We wanted to be as lean as possible while tackling following personas.</p><ul><li>Data Engineers</li><li>ML Engineers &amp; Data Scientists</li><li>Data Analysts</li></ul><p>After deep brainstorming sessions and little bit of hair pulling, we finally came up with blueprints for our new data platform. At Bazaar we refer to it as “<strong>Buraq</strong>”, which signifies speed and multi-dimensional travel.</p><blockquote>The Burāq is a creature in Islamic tradition that was said to be a transport for certain prophets.</blockquote><p>Buraq is based upon delta architecture and mix of our unique take on Data-Lakehouse and Data-Mesh philosophy.</p><p>Buraq is provisioned on two different types of clusters, one of them in a Kubernetes cluster which is primarily used to host supporting tools and applications (Airflow, Hue, Superset, Prometheus etc.), the second one is a dedicated Map Reduce Clusters for Heavy Data Processing and Machine Learning Tasks.</p><p>Buraq can be broken down into following layers.</p><ul><li>Injection/Integration Layer</li><li>Processing Layers</li><li>Service API Layer</li><li>Query Layer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPkqqKv866CHSwdSzvFFUg.png" /><figcaption>Buraq High Level Design</figcaption></figure><h3>Injection / Integration Layer</h3><p>At Bazaar we adopted the data mesh practice for decentralized data ownership. Teams in Bazaar can just bring their own bucket (object storage) with data in parquet, avro, csv and json formats and connect it to Buraq in a Plug n Play fashion. This approach enables central data team to focus on further evolving and improving Buraq.</p><p>After attaching the data buckets, we move data to our based layer cleaning and governance process which is created using Apache Hudi. We have four level in this process</p><ul><li><strong>Raw Level</strong> — unmodified data which reflects the actual data source state</li><li><strong>Bronze Level </strong>— governess and quality layer which enforces necessary compliance and provides ability to clean, transform and filter data.</li><li><strong>Silver Level </strong>— in this layer we maintain our take on Data Vault based modeling techniques to create de-normalized lake house.</li><li><strong>Gold Level</strong> — this is the final state for our data life cycle where we create Data Products which are consumed by our end users</li></ul><h3>Processing Layer</h3><p>This layer is super abstract, most of the time people creating the pipeline don’t know if the back-end data will be real-time or batch, this is achieved by using continuously running HUDI Streamer which read data from both the buckets and Kafka topics and maintain MOR (merge-on-read) tables. Our main processing framework is Apache Spark, which really works well for us.</p><h3>Service API Layer</h3><p>From the get-go we planned Buraq to be a DAAS (data as a service), so that we can enable analytics and machine learning inside our core apps. Buraq API Layer is a set of APIs which cater to these types of use cases.</p><h3>Query Layer</h3><p>As all of our data reside in data-buckets, we needed some sort of query engine to retrieve them, here <strong>Trino</strong> was simple decision as we already had hands-on knowledge over it. Some of us were already contributing to this project, and were formally acknowledged on <a href="https://trino.io/users.html#bazaar_technologies">Trino’s website</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4zdXT3QGwagU9FDc5FIyQw.png" /><figcaption>Bazaar (بازار) listed on Trino’s page</figcaption></figure><h3>What are the next efforts?</h3><h4>1. Automated Data Quality</h4><p>Bazaar’s data analysts &amp; data scientists would rely on the data ingested and processed on Buraq. Solving for data quality is the need of the hour. We have a few solutions in the works.</p><h4>2. User Facing Analytics</h4><p>As a Data Driven Organization, we might want to enable our customers to utilize certain kinds of analytics.</p><h4>3. Feature Store</h4><p>Data science can bring competitive advantage to any startup, we are working to reduce the frictions and time for getting models in productions, building a feature store in one of those efforts.</p><h4>4. Minimize Compute Cost</h4><p>We are constantly looking for ways to reduce ownership cost of Buraq, we are now trying to use efficient ARM instances along with some optimizations to our Kubernetes cluster.</p><p>We will follow up with more detailed blogs on our data platform design decisions and tech stacks. Each component of Buraq deserves it’s own blog. Buraq is all set to solve Petabytes scale problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83d132d4168b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bazaar-tech/building-bazaars-data-platform-83d132d4168b">Building Bazaar’s Data Platform</a> was originally published in <a href="https://medium.com/bazaar-tech">Bazaar Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>