<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moses Njau on Medium]]></title>
        <description><![CDATA[Stories by Moses Njau on Medium]]></description>
        <link>https://medium.com/@mosesnjau49?source=rss-eeb195852e1b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oH8wQgPcN6vl6n2jzWJvNQ.jpeg</url>
            <title>Stories by Moses Njau on Medium</title>
            <link>https://medium.com/@mosesnjau49?source=rss-eeb195852e1b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 11:15:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mosesnjau49/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Protocol Over Participants]]></title>
            <link>https://medium.com/data-unlocked/protocol-over-participants-c639e2be0f64?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c639e2be0f64</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Sun, 10 May 2026 13:58:46 GMT</pubDate>
            <atom:updated>2026-05-11T07:16:43.541Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Why the multi-agent systems that ship look different from the demos</blockquote><p>Watch a multi-agent system run for sixteen days and the agents start to look almost incidental. They come and go. Different model providers fill the orchestrator seat one week and the validator seat the next. Workers spawn, write, commit, and exit. What persists is the substrate they operate against: a write-coherent codebase, a contract that defines correctness independently of any implementation, and a protocol for handing off context that does not depend on memory.</p><p>This is the design choice that seems to separate multi-agent systems that reach production from multi-agent systems that demo. It is also the choice most easily missed by anyone reading a multi-agent paper, because the papers are about the agents. What follows traces the architecture using Factory’s <em>missions</em> framework as the working example, and follows where the choice leads — for the engineering role, for team economics, and for what looks worth paying attention to next.</p><h3>The Bottleneck Nobody Disputes</h3><p>Three years ago, the limiting factor in applying language models to software was the model’s ability to write correct code. That has changed. Frontier models now solve a wide class of engineering problems on the first attempt and a wider class within a few iterations. The bottleneck has migrated. As Luke Alvoeiro of Factory put it at AI Engineer 2026: today’s models are smart enough to figure out fifty features in a backlog; what is missing is the bandwidth to supervise their implementation.</p><p>Every commit needs review. Every architectural decision needs a human in the loop. The supervisor becomes the rate-limiter, not the writer. The opening question of any serious multi-agent architecture is therefore not <em>how do we make agents smarter</em> but <em>how do we let agents work without supervision for long stretches of time without losing coherence</em>. If the answer is “we can’t,” multi-agent stays a marketing term. If the answer is some specific protocol, there is real engineering to do.</p><h3>Five Patterns and One Mistake</h3><p>Most multi-agent literature is a taxonomy fight. Different frameworks use different terminology for what are essentially the same five interaction patterns. Following Alvoeiro’s nomenclature: <em>delegation</em>, where a parent spawns a sub-agent for a discrete task; <em>creator–verifier</em>, where one agent produces work and a second reviews it with clean context; <em>direct communication</em>, where agents talk without a coordinator and state fragments across conversations; <em>negotiation</em>, where agents communicate over a shared resource and the productive form is positive-sum; and <em>broadcast</em>, where one agent publishes to many. None of these is novel; what differs across frameworks is which patterns get composed and how.</p><p>The architectural question is how to compose them into a system whose total behaviour exceeds the sum of its parts. Most frameworks pick one or two and hope the others emerge. They tend not to. And the failure they most commonly fail through is parallelism.</p><p>The instinct to scale a multi-agent system by adding agents and running them in parallel — ten workers should be ten times more productive — almost never holds for write-heavy work. Parallel agents conflict. They make inconsistent architectural decisions because none of them sees the others’ in-flight work. They duplicate utilities. They step on each other’s files. Every conflict is paid for in tokens at merge time. The coordination overhead grows faster than the throughput gain, and the system runs slower in wall-clock terms than the same agents running one after the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEKBI3CVUHtlDyHqloMnpg.png" /></figure><p>This failure points at where the architectural work actually happens. Agents have no shared state worth speaking of. They cannot see each other’s in-flight changes. The only thing they coordinate around is whatever they all read from and write to. If that <em>something</em> is unreliable, no improvement to the agents fixes it. The architectural question becomes less about what shape the agents take and more about what shape the substrate takes.</p><h3>The Substrate</h3><p>Three pieces of substrate make missions work. None of them is an agent.</p><p><strong>Git, as the shared memory model.</strong> Missions runs features serially: only one worker or one validator is active at any moment. Within a feature, read-only operations parallelise freely — codebase exploration, API research, documentation reads, code review across multiple files. These are pure reads, so concurrency is free. But every write goes through git, one at a time. A protocol that uses commits as message-passing has to serialise writes; that is what keeps it coherent. Throughput per task drops. Correctness compounds across the days the mission runs.</p><p><strong>The validation contract.</strong> If there is one architectural choice in missions worth borrowing on its own, it is this: the validation contract is written before any code. Most agentic systems write tests after implementation. The model is asked to write tests, it inspects the code it just produced, and it generates assertions that pass against that code. Tests look comprehensive. Coverage is high. Everything green. But the tests have been shaped by the implementation, not by the requirements. They confirm decisions; they do not catch drift. A validation contract inverts this. During planning, before a single line of code exists, the orchestrator produces a structured set of assertions — for a complex project, often hundreds — that any correct implementation must satisfy. Each feature is mapped to one or more assertions; the sum of features must cover every assertion. The contract is the specification, expressed as an executable test of the finished system. This is what stops multi-day drift. Correctness has an external referent that exists before the agents start, and a worker cannot reshape what <em>done</em> means by writing a test that confirms what it built.</p><p><strong>Structured handoffs.</strong> A worker that finishes a feature does not say “done.” It produces a structured handoff that records what was completed, what was left undone, what commands were run, what exit codes those commands returned, what issues were discovered, and whether the worker abided by the procedures the orchestrator defined for it. This is the memory protocol of the system, and it does not depend on any agent remembering anything. The orchestrator inspects handoffs at milestone boundaries, identifies gaps, and scopes corrective work as new features. The system self-heals not by hoping agents will keep state, but by forcing them to externalise state into an artefact the next agent can consume cleanly.</p><p>Together these three carry most of the architectural weight. Everything else — orchestrators, workers, validators — is the population operating inside this substrate, and the population is interchangeable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TtQ4q6SgVd_UfWbW26OjyQ.png" /></figure><h3>The Population</h3><p>Given a substrate, what shape do the agents take?</p><p>The <strong>orchestrator</strong> owns planning. When a user describes a goal, the orchestrator interrogates it: surfaces unclear requirements, asks strategic questions, scopes through conversation. Its output is the validation contract, a feature decomposition, and the per-feature skills the workers will use. It does not write production code. Its job is to <em>emit substrate</em>.</p><p><strong>Workers</strong> own implementation. A worker is assigned one feature at a time, starts with clean context, reads its specification, writes the code, and commits. The next worker inherits a working codebase. Each worker is single-purpose by design — the architecture refuses to let one agent accumulate enough context to start drifting. Workers <em>mutate substrate</em>.</p><p><strong>Validators</strong> own verification. They run after each milestone and have never seen the implementation. The <em>scrutiny validator</em> runs the test suite, type checks, lints, and spawns dedicated code-review agents per completed feature. The <em>user-testing validator</em> spawns the application and interacts with it through computer use — fills forms, clicks buttons, confirms that pages render and flows complete. Most of a mission’s wall-clock time is spent here, waiting for the application to do real work, not generating tokens. This is the validator that catches the class of bug where the code looks correct but the system is broken end to end. Both validators are adversarial by construction. They never share context with the workers. They can run on a different model family entirely, which is a design choice rather than an accident: validation should not be biased by the same training data that produced the implementation. Validators <em>read substrate adversarially</em>.</p><p>The roles stay clean because the substrate underneath them does. Each role is defined by which kind of substrate operation it performs, not by which model implements it. That is what allows the population to be interchangeable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RqVepK0OMtYQKB25p0Fnkg.png" /></figure><h3>Why This Compounds</h3><p>Once the population is interchangeable, two compounding advantages fall out.</p><p>The first is <strong>the right model in each seat</strong>. Planning, implementation, and validation each have a different cognitive shape. Planning rewards slow, careful reasoning. Implementation rewards fast, fluent code generation. Validation rewards precise instruction following and resistance to being talked out of an error. No single model is best at all three. Missions can fill each seat with a different model — and ideally with different providers entirely, so the validator’s training data does not overlap with the implementer’s enough to inherit the same blind spots. Alvoeiro calls the skill of choosing well <em>droid whispering</em>: the ability to mentally model how different models fail and which seat each is best suited to. The business significance is structural. An organisation locked to one provider is bounded by that provider’s weakest capability. An organisation that substitutes per role captures the maximum of each provider’s strengths. If foundation models continue to differentiate rather than converge, the gap between these two strategies should keep widening.</p><p>The second is <strong>less exposure to the bitter lesson</strong>. Every team building multi-agent systems faces the same fear: a model release next month makes the scaffolding obsolete. Missions hedges this by pushing as much logic as possible into prompts and skills rather than deterministic code. About seven hundred lines of text define how the orchestrator decomposes features and handles failures; four sentences in this text can change the system’s execution strategy meaningfully. Worker behaviour is driven by skills the orchestrator emits per mission. The deterministic layer is thin and handles only bookkeeping: running validators, blocking progress on unresolved handoffs, checkpointing state. When a new frontier model arrives, the same orchestrator prompts tend to produce better plans, the same worker skills tend to produce better implementations, the same validator instructions tend to produce sharper review — without the architecture itself needing rework.</p><h3>The Economics</h3><p>A team of five senior engineers without agentic infrastructure can typically drive perhaps ten work streams forward in any given sprint, because every work stream requires their attention to pass through the supervision bottleneck. With a missions-style architecture, the same five engineers can plausibly oversee thirty. The work that bottlenecks on the humans is the work that genuinely requires human judgement: architecture, product decisions, customer trade-offs. The work that does not — feature implementation, refactor execution, test scaffolding, dependency upgrades — runs autonomously.</p><p>This is not a developer-productivity argument. It is a capacity argument, and the unit of analysis is the team rather than the engineer. The team’s output ceiling tends to rise with the ratio of supervised work to autonomous work. Many teams will find that ratio larger than they expected, because most software engineering is not novel design — it is competent execution against a known specification.</p><p>There is a second-order effect worth naming. The codebase produced by a missions-style system tends to end up cleaner than the codebase the team started with, because the validation contract forces high test coverage, the worker skills enforce structure, and the orchestrator’s plans push toward modular feature boundaries. The agents leave the system more amenable to further automation. This is the opposite of the typical experience with offshored or outsourced execution, where the codebase usually decays.</p><h3>What Is Not Yet Solved</h3><p>Several questions remain unanswered. <em>Can missions be composed into super-missions</em> where a meta-orchestrator coordinates several mission-level workflows — the communication patterns suggest yes, but the coordination overhead is unknown. <em>How heavily does the framework’s leverage depend on the read–write ratio of the work</em> — codebase exploration is read-heavy and parallelises well; greenfield product work is write-heavy and serialises tightly. <em>What happens to the validation contract when the goal itself shifts during execution</em> — for research-flavoured work the contract becomes the bottleneck rather than the safeguard, and missions in their current form likely struggle in genuinely exploratory domains. <em>And how does the cost scale with duration</em> — prompt caching offsets some of it, but multi-day execution remains expensive enough that the economics only make sense when the alternative is human engineering hours.</p><p>None of these is fatal. They are the visible edges of an architecture whose interior is now mostly mapped.</p><h3>What This Suggests</h3><p>The argument has been about a single architecture. What it suggests, more cautiously, is a shape worth watching for elsewhere.</p><p>The shape is recognisable: a population of interchangeable agents operating against a substrate that holds. The substrate carries the architectural weight. The agents do work but don’t accumulate it. When a system of this kind runs for days without drifting, this is usually why.</p><p>Whether the shape generalises to other domains — legal review, scientific research, design — is open. It requires that correctness be specifiable in advance, that work decompose cleanly, and that an external substrate can hold the state agents would otherwise have to remember. Software engineering meets these conditions well. Many domains do not.</p><p>Where the shape does fit, what tends to follow is a shift in where engineering attention earns its return. Less in making any single agent more capable. More in designing the substrate the agents work against — the contracts, the handoff schemas, the write-coordination story, the verification topology. None of that is glamorous. It is probably where the more durable architectural choices in agentic systems will live for the next several years.</p><p>Whether agents become commodity, or substrate becomes commodity, or both, or neither — too early to say. What is already clear is that the systems running for sixteen days at a time are not running because the agents are smart. They are running because the substrate around them is coherent.</p><p>The agents do the work. The protocol holds it together.</p><h3>References</h3><p>Alvoeiro, L. (2026). <em>The Multi-Agent Architecture That Actually Ships.</em> AI Engineer Conference, San Francisco.</p><p>Sutton, R. (2019). <em>The Bitter Lesson.</em> Incomplete Ideas blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c639e2be0f64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/protocol-over-participants-c639e2be0f64">Protocol Over Participants</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design the environment, not the agents]]></title>
            <link>https://medium.com/data-unlocked/design-the-environment-not-the-agents-2cad0287f9ce?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cad0287f9ce</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[dataengineering-future]]></category>
            <category><![CDATA[swarm-intelligence]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Thu, 02 Apr 2026 21:27:22 GMT</pubDate>
            <atom:updated>2026-04-03T07:22:44.690Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The highest-leverage work in multi-agent AI systems isn’t writing better prompts. It’s designing the substrate — the shared memory, queues, tool contracts, and feedback loops — that agents coordinate through. The swarm’s intelligence lives in the environment, not the individuals inside it.</blockquote><p>In 1921, William Beebe recorded something that would become a touchstone in the study of collective intelligence. He watched a column of army ants — tens of thousands of them — moving in a perfect circle. Not a foraging route. Not a patrol pattern. A loop. Each ant was following the pheromone trail of the ant ahead of it. The trail circled back on itself. The colony marched until the ants died of exhaustion.</p><p>Entomologists call it an <em>ant mill</em>. It is swarm intelligence failing in the most literal way possible: the substrate was corrupt, and no individual agent had the vantage point to know it.</p><p>Modern agentic AI systems can fall into ant mills. A task decomposes across four agents. One agent writes a finding to shared memory with a subtle framing error. The next reads it, builds on it, amplifies it. The third validates it against the fourth’s output, which was derived from the same flawed premise. The orchestrator synthesizes a confident, well-structured, entirely wrong answer. Every agent behaved rationally. The system failed.</p><p>The lesson is the same one Beebe’s ants taught us a century ago: in collective systems, <strong>the substrate determines the outcome</strong>. You cannot engineer your way to good swarm behavior by making individual agents smarter. You have to engineer the environment they move through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*hdaJLHfheDPtbCRi.jpeg" /></figure><blockquote>“Evolution didn’t make smarter ants. It made better pheromones.”</blockquote><h3>The substrate is the unit of design</h3><p>When teams build multi-agent systems, they tend to invest disproportionately in agent prompts. This is understandable — prompts are legible, iterable, and give immediate feedback. But in a well-designed multi-agent system, the prompt is almost the least important variable. What matters is the surface the agent operates on.</p><p>That surface has four components:</p><p><strong>Memory</strong> — shared memory schema Analog: pheromone chemistry. Governs what agents can communicate, and how that communication decays or compounds.</p><p><strong>Queue</strong> — task queue design Analog: trail branching patterns. Governs how work is discovered, prioritized, and delegated without a controller.</p><p><strong>Tools</strong> — tool interfaces Analog: sensory apparatus. Governs what actions are available, what side effects they produce, and what they return.</p><p><strong>Signals</strong> — feedback loops Analog: quorum sensing. Governs how the system detects drift, failure, and completion without a central observer.</p><p>The diagram below shows how agents relate to the substrate. The agents themselves are interchangeable — the substrate is where the system’s intelligence lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*49BBNznupoIYqOdhJ87ApA.png" /></figure><p>Each of these deserves serious architectural attention. Most production systems give serious attention to none of them.</p><h3>Shared memory schema</h3><p>Shared memory is how agents communicate without talking to each other. This is stigmergy — the same principle that lets ants coordinate logistics across kilometers of terrain without a dispatcher. One agent modifies the environment; another reads that modification and acts on it. The environment is the message.</p><p>In software terms, this usually means a database, a vector store, a structured JSON blob, or a document that agents read from and write to. The schema of that store is the pheromone chemistry. Get it wrong and you get ant mills. Get it right and you get emergent coordination that no orchestrator planned.</p><h3>What makes a good memory schema</h3><p>A well-designed memory schema has three properties. First, it is <em>typed</em>. Agents shouldn’t be writing free-form text into a shared blob; they should be writing structured records with explicit fields. This makes reads predictable and prevents one agent’s prose style from confusing another.</p><p>Second, it is <em>versioned</em>. When Agent B reads a memory that Agent A wrote, it should know when that memory was written, what state the task was in when it was written, and whether anything has changed since. Without versioning, agents act on stale state with full confidence.</p><p>Third, and most importantly: it models <em>uncertainty explicitly</em>. This is where most schemas fail. An agent should be able to write not just a finding, but a confidence level, a list of assumptions, and a flag indicating whether the finding should be treated as a draft or a conclusion.</p><pre>// Weak schema — agents read this as ground truth<br>{<br>  &quot;finding&quot;: &quot;The API returns 403 on expired tokens&quot;<br>}</pre><pre>// Strong schema — agents read this with calibrated trust<br>{<br>  &quot;finding&quot;: &quot;The API returns 403 on expired tokens&quot;,<br>  &quot;confidence&quot;: &quot;high&quot;,<br>  &quot;source&quot;: &quot;direct_observation&quot;,<br>  &quot;assumptions&quot;: [&quot;token issued &gt; 24h ago&quot;, &quot;staging env&quot;],<br>  &quot;status&quot;: &quot;verified&quot;,<br>  &quot;written_by&quot;: &quot;research_agent_2&quot;,<br>  &quot;timestamp&quot;: &quot;2025-03-14T09:41:00Z&quot;,<br>  &quot;supersedes&quot;: null<br>}</pre><p>The second record takes twelve more tokens to write. It saves entire agent chains from chasing a false premise. The overhead is trivially worth it.</p><blockquote><strong>The ant mill in memory:</strong> An agent writes a confident but incorrect finding. Downstream agents read it, trust it, and build on it — each layer of inference making the original error harder to spot. By the time the system produces output, the finding has been “confirmed” three times by three different agents, none of whom verified the original. The schema had no confidence field. There was no way to signal doubt.</blockquote><h3>Task queue design</h3><p>The task queue is the colony’s trail network. It determines how work is discovered, how it flows between agents, and how the system recovers when an agent fails or gets stuck. In hierarchical systems, this is usually managed by an orchestrator. But orchestrators are single points of failure, and they create bottlenecks as task volume grows.</p><p>More robust systems use a <em>work-stealing</em> or <em>declarative queue</em> model: agents don’t receive tasks, they claim them. The queue is a database of task states. An available agent scans for tasks in a claimable state, locks one atomically, executes it, and writes the result back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*kNEDd_Gl-Cs-KC171-sDzA.png" /></figure><p>This structure gives you several properties for free. <strong>Idempotency</strong>: a task’s result is written back to the same record, so re-running a failed task doesn’t create duplicates. <strong>Observability</strong>: you can query the table at any time and see exactly what state every task is in. <strong>Recovery</strong>: if an agent crashes mid-task, its claim expires and another agent can pick it up. <strong>Dependency management</strong>: a depends_on array lets you express complex task graphs without an orchestrator that understands the whole graph at once.</p><pre>-- Tasks are states, not messages<br>CREATE TABLE tasks (<br>  id          UUID PRIMARY KEY,<br>  type        TEXT NOT NULL,<br>  status      TEXT NOT NULL DEFAULT &#39;pending&#39;, -- pending | claimed | done | failed | blocked<br>  payload     JSONB,<br>  result      JSONB,<br>  claimed_by  TEXT,<br>  claimed_at  TIMESTAMPTZ,<br>  depends_on  UUID[],                          -- dependency graph, not just a list<br>  retries     INT DEFAULT 0,<br>  created_at  TIMESTAMPTZ DEFAULT NOW()<br>);</pre><h3>The dependency graph is load-bearing</h3><p>Most naive queue implementations treat tasks as a flat list. This breaks as soon as tasks have prerequisites. If Agent C needs the output of Agents A and B, and A fails, what happens to C? Without an explicit dependency model, C either blocks forever, reads stale data, or the orchestrator needs custom logic to handle every possible failure permutation.</p><p>With an explicit dependency graph baked into the queue schema, the resolution logic is generic: an agent claims a task only when all depends_on tasks are in done state. Failures propagate naturally — tasks that depend on a failed task transition to a blocked state, which can trigger alerts or fallback paths without special-case code.</p><h3>Tool interfaces</h3><p>A tool is how an agent acts on the world. The interface design — the name, parameters, return type, and side effects — determines how well agents can reason about what they’re doing and what the consequences are.</p><p>LLMs use tools by reading their descriptions and inferring how to call them. This means tool design is, in part, <em>prompt engineering at the interface level</em>. A poorly named tool with ambiguous parameters produces bad tool calls regardless of how good the agent’s system prompt is.</p><h3>Contracts over implementations</h3><p>The most important principle in tool design is: make the contract explicit and the implementation invisible. An agent should be able to read a tool’s schema and know exactly what it will get back, what states it can leave the world in, and what it cannot do. Side effects should be declared, not implied.</p><pre>// Weak tool contract — agent has to guess<br>{<br>  &quot;name&quot;: &quot;update_record&quot;,<br>  &quot;description&quot;: &quot;Updates a record in the database&quot;,<br>  &quot;parameters&quot;: { &quot;id&quot;: &quot;string&quot;, &quot;data&quot;: &quot;object&quot; }<br>}</pre><pre>// Strong tool contract — agent knows exactly what it&#39;s doing<br>{<br>  &quot;name&quot;: &quot;update_task_status&quot;,<br>  &quot;description&quot;: &quot;Transitions a task to a new status. Only valid transitions: pending→claimed, claimed→done, claimed→failed. Idempotent on same-status writes. Does NOT modify task payload.&quot;,</pre><pre>  &quot;parameters&quot;: {<br>    &quot;task_id&quot;: { &quot;type&quot;: &quot;string&quot;, &quot;description&quot;: &quot;UUID of the task to update&quot; },<br>    &quot;new_status&quot;: { &quot;type&quot;: &quot;string&quot;, &quot;enum&quot;: [&quot;claimed&quot;, &quot;done&quot;, &quot;failed&quot;] },<br>    &quot;reason&quot;: { &quot;type&quot;: &quot;string&quot;, &quot;description&quot;: &quot;Required when new_status is failed&quot; }<br>  },</pre><pre>  &quot;side_effects&quot;: [&quot;writes to tasks table&quot;, &quot;triggers downstream task eligibility check&quot;]<br>}</pre><p>The second contract is longer. It is also dramatically safer. An agent reading it knows it cannot accidentally modify task data when updating status. It knows transitions are constrained. It knows a failure requires a reason. The tool is harder to misuse — not through validation alone, but through legibility.</p><h3>Read/write symmetry</h3><p>One pattern worth enforcing: every write tool should have a corresponding read tool that returns data in the same shape. This lets agents verify their writes, supports retry logic, and makes the system’s state inspectable without side effects. It also makes debugging dramatically easier — you can replay any agent’s decision by re-reading the same state it read.</p><h3>Feedback signals</h3><p>The most underinvested component of almost every production agentic system is feedback. Biological swarms have continuous, ambient feedback — pheromone concentration is a real-time signal of trail quality that the entire colony reads and updates simultaneously. In LLM-agent systems, feedback is usually an afterthought: a human looks at the output, shrugs, and tries again.</p><p>This matters because goal drift in agent chains is real and insidious. Each agent in a chain slightly reinterprets the task based on the memory it inherits. By agent five or six, the task being executed may be meaningfully different from the task originally specified — and no individual agent is “wrong.” The drift is emergent. Catching it requires a signal that watches the system level, not the agent level.</p><h3>Structural feedback vs. semantic feedback</h3><p>There are two kinds of feedback to design for. <em>Structural feedback</em> catches mechanical failures: an agent didn’t produce output in the expected format, a tool call returned an error, a task exceeded its time budget. This is tractable with deterministic checks and should be implemented first.</p><p><em>Semantic feedback</em> is harder: did the agents stay on task? Is the output actually answering the right question? This requires either a dedicated critic agent — one whose entire job is to read the accumulated memory and ask “is this still the right task?” — or human-in-the-loop checkpoints at defined intervals.</p><pre># Minimal critic agent prompt for semantic drift detection<br>CRITIC_PROMPT = &quot;&quot;&quot;<br>You are a task auditor. You will be given:<br>1. The original task specification<br>2. The current state of shared memory<br>3. A list of completed sub-tasks</pre><pre>Your job is NOT to evaluate quality. Your only job is to determine<br>whether the agents are still working on the original task.</pre><pre>Return a JSON object:<br>{<br>  &quot;on_task&quot;: boolean,<br>  &quot;drift_detected&quot;: boolean,<br>  &quot;drift_description&quot;: string | null,   // null if on_task is true<br>  &quot;recommended_action&quot;: &quot;continue&quot; | &quot;reorient&quot; | &quot;escalate&quot;<br>}<br>&quot;&quot;&quot;</pre><p>A critic agent running at defined checkpoints — say, after every N tasks complete or every M minutes — gives the system a quorum-sensing mechanism. It doesn’t need to be right all the time. It just needs to catch catastrophic drift before the system has invested twenty more agent-steps in the wrong direction.</p><h3>Designing for failure visibility</h3><p>The last feedback consideration is instrumentation. Every agent action should emit a structured trace: what it read, what it decided, what it wrote, what tools it called, and what the tools returned. Not for debugging after the fact — for real-time observation of system health.</p><p>Without structured traces, a multi-agent failure looks like a bad final output. With structured traces, the same failure looks like “Agent 3 wrote a low-confidence finding with source: inference, which was then read by Agents 4 and 5 as source: observation.&quot; That&#39;s a substrate problem with a clear fix: enforce that source values propagate correctly through the memory schema.</p><h3>Putting it together</h3><p>The architecture that emerges from these four components looks less like a prompt-engineering exercise and more like a distributed systems design problem. Which, of course, it is.</p><p>A robustly designed multi-agent substrate has: a typed, versioned, uncertainty-aware memory schema; a declarative task queue with explicit dependency graphs and atomic claiming; tool contracts that fully specify behavior and side effects; and feedback signals at both the structural and semantic levels.</p><p>Agent prompts, in this architecture, are almost interchangeable. You can swap a cheaper model into a non-critical agent slot. You can replace a specialist agent with a generalist if the task demands change. The system continues to function because its intelligence is embedded in the environment, not the individuals moving through it.</p><blockquote>“The swarm’s intelligence lives in the substrate, not the individuals.”</blockquote><p>This is the design principle that matters. Not because agent capability is unimportant — it clearly is — but because substrate quality is the binding constraint for almost every production multi-agent system that exists today. The agents are already capable enough. The environments they move through are not.</p><p>Beebe’s ants weren’t stupid. They were following a broken trail. Build better trails.</p><p><em>For the dependency queue pattern, see the actor model literature and the design of systems like Temporal.io and Celery. For stigmergy in software, Dorigo &amp; Stützle’s</em> Ant Colony Optimization <em>(MIT Press) remains the canonical text. For semantic drift in agent chains, the AutoGen and LangGraph research papers both contain empirical observations worth studying.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cad0287f9ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/design-the-environment-not-the-agents-2cad0287f9ce">Design the environment, not the agents</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AI Doesn’t Make You Less Smart. It Exposes Whether You Ever Were.]]></title>
            <link>https://medium.com/data-unlocked/using-ai-doesnt-make-you-less-smart-it-exposes-whether-you-ever-were-d9b826b373f5?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9b826b373f5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[thinkin]]></category>
            <category><![CDATA[ideas]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 21:06:05 GMT</pubDate>
            <atom:updated>2026-03-26T21:15:38.115Z</atom:updated>
            <content:encoded><![CDATA[<p>There’s a gatekeeping narrative spreading through tech right now. That using AI to build, think, or ship makes you less of an engineer. Less rigorous. Less <em>legitimate</em>. That the real professionals — the serious ones — don’t need it.</p><p>I think that’s wrong. More than wrong — I think it’s a defence mechanism. A way of protecting an identity built on a benchmark that was always more fragile than it looked. And I think the people pushing it hardest are the ones with the most to lose if the benchmark changes.</p><p>It already has.</p><h3>The Old Definition of Smart Was Always a Proxy</h3><p>For years, the tech industry measured intelligence in the wrong currency. How fast can you write code? Can you recite the time complexity of every sorting algorithm? Can you invert a binary tree on a whiteboard while someone evaluates your composure?</p><p>That was the game. And a lot of people got very good at performing it.</p><p>But those metrics were never measuring what they claimed to. They were proxies — imperfect, gameable, culturally contingent proxies — for something much harder to quantify. Judgment. The ability to look at a messy, ambiguous, politically charged problem and find the real structure inside it. The instinct to ask <em>why are we building this</em> before asking <em>how</em>. The pattern recognition that comes from having been wrong enough times to know, viscerally, what a bad abstraction smells like before it becomes a six-month refactor.</p><p>The engineers everyone actually wanted — the ones who shaped systems that held up under real conditions, who could walk into a room full of disagreement and leave with a direction everyone would follow — were never valuable because they typed fast. They were valuable because they <em>thought clearly</em>. We just had no clean way to hire for that. So we hired for execution speed instead, because execution was where the bottleneck was.</p><p>The thinking was always cheap. The building was expensive. So we optimised for builders.</p><p>AI changed that equation completely. The building is no longer the bottleneck. The thinking is. And suddenly, the thing the industry spent decades undervaluing is the only thing that actually commands a premium.</p><h3>What AI Actually Changes — And Why It Matters</h3><p>The real shift isn’t about code generation. It’s about feedback loops, and what happens when you compress them.</p><p>Every engineering decision is a hypothesis. You have a belief — about how a system will behave under load, about whether this data model will serve the questions the business will eventually ask, about whether this abstraction will hold when the requirements inevitably shift. The gap between forming that hypothesis and finding out whether it was right is the feedback loop. And historically, in software, that loop has been punishingly slow.</p><p>A wrong architectural call might not reveal itself until you’re six months in. A flawed data model might only surface when the business asks a question it fundamentally cannot answer. The cost of being wrong was high enough that caution became rational — long planning cycles, architecture review committees, specification documents nobody fully read. All of that overhead existed because failure was expensive.</p><p>AI compresses the loop. You can prototype an approach in an afternoon, pressure-test it against edge cases in real time, explore three competing architectures in parallel, and arrive at a genuinely informed decision before a single line of production code exists. You can fail faster, at a smaller scale, with less collateral damage.</p><p>This is what I mean by atomic iteration. Your failures get smaller, cheaper, and earlier. Because failure is cheaper, you take more swings. Because you take more swings, the quality of what you finally ship goes up substantially. You are not just moving faster. You are operating at a finer granularity — catching wrong turns earlier, validating assumptions that used to require weeks of build time to test.</p><p>That is not a shortcut. That is a fundamentally superior epistemic process.</p><h3>Token Consumption Is a Badge of Honour</h3><p>There is a reframe worth making explicit here, because I think it matters for how engineers think about their own identity in this era.</p><p>Burning tokens is not a sign of weakness. It is a signal of how seriously you are operating.</p><p>Ivan Zhao, CEO of Notion, has watched this shift happen at close range — including to his own co-founder. Simon Last was once a typical “10x engineer.” Now he almost no longer writes code himself, instead coordinating three to four AI coding agents simultaneously — boosting his output by three or four times. That is not a story of someone who got lazy. That is a story of someone who upgraded their altitude. The code didn’t stop mattering. The <em>relationship</em> to the code changed.</p><p>Zhao has framed this shift in terms that go beyond individual productivity. He coined the phrase “manager of infinite minds” to describe what the best knowledge workers are becoming — a framing that Satya Nadella referenced at Davos alongside quotes from Steve Jobs and Bill Gates. The idea is precise: your value is no longer in what you personally execute, but in how effectively you direct, evaluate, and compound the work of the systems around you. You are not the worker. You are the mind that organises the workers.</p><p>Zhao compares AI to “the steel of organisations” — just as steel frameworks liberated load-bearing walls, AI has the potential to shoulder the burden of context and coordination, freeing humans from being the sole structural pillars of organisations. An engineer who refuses to engage with that shift is not being principled. They are declining a structural upgrade.</p><p>The badge of honour used to be how much you could produce without tools. Now it is how much leverage you can generate <em>with</em> them. That is not a lowering of standards. That is a maturation of the craft.</p><h3>The Replacement Myth Misunderstands What Businesses Actually Buy</h3><p>I’ll say this plainly: businesses do not hire engineers to write code. They hire engineers to solve problems. The code is a byproduct — the residue of a long chain of decisions about what to build, how to structure it, what to sacrifice, and how to make it legible enough that the next person can work with it.</p><p>A CEO does not wake up thinking <em>I need someone to write a for-loop</em>. They wake up thinking: I have a compliance gap that is costing us clients. I have a data platform three teams have stopped trusting. I have a product that is three months behind and nobody can tell me exactly why. <em>Who can fix this?</em></p><p>That question has never been answered by typing speed. It is answered by the person who can walk into that complexity, hold it clearly enough to see the actual problem underneath the stated one, and convert it into a working system that holds up under real conditions.</p><p>If you are that person — and you can now also compress the execution timeline dramatically — you are not replaceable. You are <em>multiplicative</em>. You become the architect and the construction crew simultaneously. Someone who can deliver in a week what used to require a quarter and a team of four.</p><p>The people facing genuine pressure are not the sharp problem-solvers using AI. They are the people whose entire value proposition was execution speed — who were never particularly interested in the problem space, who preferred the clean certainty of a well-scoped ticket over the messy upstream work of figuring out what the right ticket should be in the first place. AI did not create that vulnerability. It made it visible.</p><h3>What This Looks Like From Where I Stand</h3><p>I work in data engineering. Complex platforms, pipelines, semantic layers — the underlying systems that organisations depend on to make decisions they can actually trust. And I use AI constantly.</p><p>Not because I cannot write the code. Because the code was never the hard part.</p><p>The hard part is understanding what the business actually needs versus what they think they need — and those are very often not the same thing. The hard part is designing a data model that will still make sense in two years when the team has changed and the requirements have shifted in ways nobody predicted. The hard part is knowing when a problem is fundamentally a technical problem and when it is a people problem wearing a technical costume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f99zw0O4g4pasp4X.jpg" /></figure><p>AI lets me move through the execution fast enough to spend my best hours upstream — in the problem space, where the real thinking lives. I can prototype the consumption layer before the ingestion layer exists, which means I discover what the requirements actually are before I have committed to a structure that cannot serve them. My ideas compound faster. My understanding of a problem deepens more quickly because I can interrogate its edges cheaply before committing to anything.</p><p>The result is not that I do less thinking. I do more. And the work I ship is better — not just faster, but more coherent, more correct at the level of the problem it was actually built to solve.</p><h3>Intelligence as Leverage, Not Throughput</h3><p>There are two ways to think about the value an engineer creates. Throughput — output per unit of time. And leverage — the degree to which your decisions improve the output of everything around you.</p><p>Throughput is linear. One engineer working harder produces more output, up to a ceiling. Leverage is exponential. An architectural decision made well at the start shapes every line of code written for the next two years. A data model designed with genuine domain understanding reduces the cost of every query, every report, every analytical question for as long as the system exists.</p><p>The entire history of software engineering, looked at clearly, is a history of shifting value from throughput to leverage. Assembly to high-level languages. Manual memory management to garbage collection. Custom infrastructure to cloud primitives. Each transition made execution cheaper and raised the value of the decisions made above it.</p><p>AI is the latest and most dramatic instance of this pattern. The engineers who already understood that leverage was where the value lived are the ones best positioned to use it well. They were already playing the right game. AI just gave them better tools to play it with.</p><h3>The New Definition</h3><p>Smart now is the quality of the problems you choose to engage with. How quickly a spark in your mind becomes something real in the world. The judgment you bring when AI gives you a confident, fluent, completely wrong answer — and you catch it, because you understand the domain deeply enough to know what correct actually looks like from the inside.</p><p>AI does not replace expertise. It amplifies it. In the hands of someone who thinks clearly, it is extraordinary. In the hands of someone who doesn’t, it is just a faster way to produce something that looks right and isn’t. The difference between those two outcomes is not the tool. It is the person holding it.</p><p>The leverage has shifted entirely to the thinker. To the person who brings genuine understanding before reaching for the tool. To the person who can look at the output and know immediately whether it is solving the right thing.</p><p>That person, working with AI, is not less intelligent than the engineer who refuses to. They are operating at a different altitude entirely.</p><p>Use the tools. Burn the tokens. Build without apology. The people telling you that makes you less smart are defending a definition of smart the world has already moved past.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9b826b373f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/using-ai-doesnt-make-you-less-smart-it-exposes-whether-you-ever-were-d9b826b373f5">Using AI Doesn’t Make You Less Smart. It Exposes Whether You Ever Were.</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The End of Pipeline Engineering: A Synthesis for Data That Survives Business Evolution]]></title>
            <link>https://medium.com/data-unlocked/the-end-of-pipeline-engineering-a-synthesis-for-data-that-survives-business-evolution-953d3384d42f?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/953d3384d42f</guid>
            <category><![CDATA[bounded-context]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 16:29:36 GMT</pubDate>
            <atom:updated>2026-02-21T21:44:13.621Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Why entities and events replace dimensions and facts, why declaration replaces orchestration, and why the ontology your business needs was never going to come from a workshop.</em></h4><p>For three decades, data engineering has been organized around a single metaphor: the pipeline. Data flows from source systems through a series of transformations — bronze to silver to gold, staging to warehouse to mart — until it arrives in a shape that business users can consume. The entire profession is built around this image. We are pipeline plumbers, and the warehouse is the reservoir.</p><p>This metaphor has outlived its usefulness.</p><p>The pipeline metaphor assumes that the hard problem in data is <em>moving</em> and <em>transforming</em> data. It isn’t. The hard problem is <em>surviving business change without losing institutional memory</em>. Every organization that has operated a data warehouse for more than two years knows the feeling: a KPI gets retired, a customer segmentation model changes, a business unit reorganizes, and suddenly the “single source of truth” is a graveyard of orphaned tables, stale transformations, and dashboards that no one trusts.</p><p>This article proposes a synthesis — one where data engineers stop building pipelines and start declaring products, where dimensions and facts give way to entities and events, where the business ontology emerges from usage rather than being designed in workshops, and where business evolution becomes a first-class feature instead of a system-breaking threat.</p><p>I should be honest about what this article is and isn’t. The framework described here is sound and implementable. But it’s not a solved problem with off-the-shelf tooling. Some pieces exist today. Others require platform engineering that most organizations would need to build or buy. I’ll be explicit about where the boundaries are.</p><p>I should also be honest about what’s new here and what isn’t. The individual ideas in this article are not novel. Entity/event modeling comes from Eric Evans’s domain-driven design (2003). Append-only logs with derived materialized views come from Martin Kleppmann (2014). Product thinking for analytical data comes from Zhamak Dehghani (2019). Declarative infrastructure is standard practice in platform engineering. Bounded context integration patterns are Evans and Fowler. The strangler fig migration is Fowler, 2004. Any well-read practitioner will recognize every ingredient.</p><p>What none of these provide individually is a coherent modeling paradigm for analytical data products that survives business evolution — one that connects domain modeling to storage architecture to serving projections to emergent metadata. The contribution here is the integration, not the invention. Most influential frameworks in software and data engineering — Dehghani’s data mesh, Kimball’s dimensional modeling, Kleppmann’s unified treatment of distributed data systems — are syntheses of existing ideas into something a practitioner can act on. This article attempts the same: a unified framework that connects pieces that previously existed in separate conversations.</p><h3>Part I: What’s Actually Broken</h3><h3>The pipeline chain is a liability masquerading as architecture</h3><p>The modern data stack promised agility. What it delivered was a different kind of complexity. A typical production environment in 2025 looks like this: Fivetran or Airbyte extracting from dozens of sources, landing data in a cloud warehouse, dbt transforming it through 200+ models organized into staging, intermediate, and mart layers, and a BI tool connecting to the final tables.</p><p>Each link in this chain is a potential failure point. Each dbt model encodes assumptions about upstream schema that may not hold tomorrow. Each transformation copies data from one layer to the next, creating redundancy that inflates storage costs and makes lineage opaque. When something breaks — and it always breaks — the data engineer traces the failure backward through layers of SQL files, trying to reconstruct which assumption was violated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_molD1iI_UHH8GMbQuXvUw.png" /></figure><p>Joe Reis, author of <em>Fundamentals of Data Engineering</em>, captures the state of things bluntly: data warehouses are full of incomprehensible tables, and the business doesn’t know how to use any of it. The lack of a data model is still a data model — just a terrible one. The pipeline-first mindset creates the illusion of productivity while accumulating technical debt at every layer.</p><h3>Dimensional modeling solves the wrong problem at the wrong time</h3><p>Ralph Kimball’s dimensional modeling methodology, introduced in 1996 with <em>The Data Warehouse Toolkit</em>, remains the most widely used approach to organizing analytical data. The approach is deceptively simple: identify the business process, declare the grain, identify the dimensions, identify the facts. Build star schemas around fact tables and dimension tables. Use conformed dimensions to standardize meaning across departments.</p><p>Kimball’s principles are sound. The problem is <em>when</em> they get applied and <em>where</em> they live.</p><p>Traditional dimensional modeling requires upfront design. Before a single row flows, a team of data engineers and business analysts sits in a room and decides: “Customer is a dimension. Order is a fact table. These are the attributes. This is the grain.” The resulting star schema is then implemented as physical tables in the warehouse, and every pipeline is built to populate and maintain those tables.</p><p>This creates three problems that compound over time.</p><p>First, <strong>the classification is use-case dependent, not intrinsic</strong>. A customer is a dimension when you’re analyzing orders — it’s the descriptive context around a transaction. But customer <em>activity</em> is a fact when you’re analyzing engagement patterns — the customer becomes the grain, and the measures are behavioral metrics. A store is a dimension when you’re slicing revenue. Store <em>openings and closings</em> are facts when you’re analyzing real estate strategy. Forcing a permanent classification at design time bakes in one perspective and makes every other perspective harder.</p><p>Second, <strong>the design is brittle to business change</strong>. When the CMO switches customer segmentation from revenue-based tiers (Gold, Silver, Bronze) to behavioral clusters (Engaged, At-Risk, Dormant), the dimensional model breaks. The dim_customer table needs to be restructured. Should you update historical records to the new segments (destroying history)? Implement a slowly-changing dimension type 2 (adding complexity)? Create a new dimension entirely (breaking every existing dashboard)? Each option creates a different kind of mess.</p><p>Third, <strong>the model lives in the wrong layer</strong>. Star schemas exist as physical tables in the warehouse. This means the dimensional model is coupled to the storage technology. Replacing the warehouse, adding a new serving store, or supporting a consumption pattern that doesn’t fit star schemas (like an ML feature store or a key-value cache) requires building entirely separate data paths.</p><p>Kimball’s insight — that business users need data organized around business processes, with clear measures and descriptive context — remains absolutely correct. What’s wrong is implementing that insight as the core architecture rather than as a rendering concern.</p><h3>Data mesh identified the problem but left a gap</h3><p>Zhamak Dehghani’s data mesh, introduced in 2019 and elaborated in her 2022 O’Reilly book <em>Data Mesh: Delivering Data-Driven Value at Scale</em>, represents the most significant rethinking of data architecture in the past decade. Data mesh’s four principles — domain ownership, data as a product, self-serve platform, and federated computational governance — correctly identify that the centralized pipeline model doesn’t scale organizationally.</p><p>Dehghani’s key contribution is the concept of the <em>data product quantum</em>: an independently deployable, high-cohesion component that encapsulates all the structural elements needed to share data as a product — the data itself, the code that maintains it, the metadata that makes it discoverable, and the infrastructure that serves it.</p><p>But data mesh, as typically implemented, inherits a blind spot from its software engineering roots. It defines <em>what</em> a data product is and <em>who</em> owns it, but it doesn’t prescribe <em>how</em> the data within a product should be modeled to survive business evolution. Most data mesh implementations end up with domain teams building dimensional models inside their data products — the same star schemas as before, just decentralized. The pipeline is shorter (it stays within the domain), but the same brittleness to change remains.</p><p>The gap is this: data mesh gives you the organizational structure and the product mindset. It doesn’t give you the data modeling paradigm that makes products resilient to the business changes that will inevitably come.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gnwg24lS438_wlCtyWv0hQ.png" /></figure><h3>Part II: Entities and Events — A Modeling Paradigm for Durability</h3><h3>The two primitives</h3><p>If you strip away the accumulated conventions of data warehousing, there are only two kinds of things in any business domain:</p><p><strong>Entities</strong> are things that exist and have identity. A customer, a store, a product, an employee, a contract. An entity has a stable identifier that other things reference — orders point to customers, transactions point to stores, shipments point to products. The identity survives everything. A customer can change their name, email, address, and loyalty tier. They’re still the same customer. The customer_id never changes.</p><p><strong>Events</strong> are things that happened. An order was placed, a payment was processed, a customer visited a store, a product was restocked. An event is immutable from birth — it happened at a specific time, involved specific entities, and produced specific measurable values. You can’t un-place an order or un-process a payment. Events have timestamps. Events reference entities. Events carry measures.</p><p>This is not a new idea. Event-driven architecture, event sourcing, and domain-driven design all operate on variations of this distinction. What this article does is apply it as the foundational modeling paradigm specifically for <em>analytical</em> data products — replacing the fact/dimension split that Kimball introduced and filling the modeling gap that Dehghani’s data mesh left open.</p><h3>The intellectual foundation: turning the database inside out</h3><p>The architecture described in this article has a direct ancestor in Martin Kleppmann’s work. In <em>Designing Data-Intensive Applications</em> (2017) and his 2014 talk “Turning the Database Inside Out,” Kleppmann makes an argument that has profound implications for analytical data architecture, even though he frames it in terms of operational systems.</p><p>Kleppmann’s core insight is this: a traditional database bundles storage, indexing, caching, and materialized views into a single system. If you unbundle those concerns, you get a more resilient and flexible architecture. The source of truth becomes an append-only log of immutable events. Everything else — the queryable tables, the search indexes, the caches, the materialized views — is a <em>derived view</em> of that log. Each derived view can be rebuilt from the log at any time. If you add a new consumer that needs the data in a different shape, you derive a new view from the same log. The views are disposable. The log is the authority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wjKZ171sRs-lklUvtCzhA.png" /></figure><p>This is exactly what the entity/event paradigm does for analytical data, and the parallel is not accidental.</p><p>The data product’s canonical dataset in Iceberg — the append-only, snapshot-isolated, time-traveling record of entities and events — is Kleppmann’s log. The serving intents — the star schema in PostgreSQL, the flat table in StarRocks, the key-value projection in Redis, the metadata endpoint for AI agents — are Kleppmann’s derived views. Each one is a materialized projection of the same source of truth, shaped for a specific consumer, independently rebuildable if the projection logic changes.</p><p>Kleppmann argues that this architecture is more resilient because the log survives everything. If a derived view is corrupted, you rebuild it from the log. If you need a new type of view, you derive it without touching the existing ones. If the derivation logic has a bug, you fix it and re-derive — the log is unaffected.</p><p>Applied to data products: if a serving intent produces incorrect output, you fix the intent’s SQL and re-materialize from the same quality-approved Iceberg snapshot. If the finance team needs a new star schema, you add a new serving intent — no aggregate product changes needed. If a quality gate reveals that a transformation produced bad data, the Iceberg snapshot is discarded and consumers continue reading the previous good snapshot. The canonical data survives everything. The projections are replaceable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*SvtoAe2KzfN1z0fFgLU8JA.png" /></figure><p>Where this article extends Kleppmann’s architecture into analytical territory is in two directions he didn’t address. First, the <em>quality gate as a prerequisite for promotion</em>. Kleppmann describes derived views as eventually consistent with the log — consumers see new data once the derivation pipeline processes it. In the analytical context, “eventually consistent” isn’t good enough. Data must pass quality checks before any consumer sees it. The stage-then-promote pattern using Iceberg snapshot isolation adds a verification step between the log and the derived views that Kleppmann’s architecture implies but doesn’t mandate. Second, the <em>ontology as emergent metadata</em>. Kleppmann focuses on the mechanical properties of logs and derived views. The entity/event product paradigm adds semantic metadata — identity declarations, attribute lifecycles, cross-domain polyseme governance — that turns the system from a data processing architecture into a living map of business concepts.</p><p>But the foundation is Kleppmann’s, and this article should be read as building on his work, not as independent invention. If you understand “turning the database inside out,” you understand this framework. The append-only entity/event product is the log. The serving intents are the derived views. The rest is governance, quality enforcement, and tooling.</p><h3>Why this distinction is more durable than facts and dimensions</h3><p>The fact/dimension classification is a <em>consumption concern</em>. It describes how data should be organized for querying by BI tools. The entity/event classification is a <em>domain concern</em>. It describes what the data <em>is</em> in the real world.</p><p>Consider the practical difference. In Kimball’s model, you decide upfront: “Customer is a dimension.” You build dim_customer with its attributes and fact_orders with its measures. When the business changes the customer segmentation model, you have to restructure dim_customer — because the model is encoded in the physical schema.</p><p>In the entity/event model, you have a customer entity product that captures the customer as it is, with all its attributes, timestamped at each extraction. When the segmentation changes from Gold/Silver/Bronze to Engaged/At-Risk/Dormant, nothing structural changes. Monday’s record says customer_id: C-4471, segment: Gold, extracted_at: 2026-03-10. Tuesday&#39;s record says customer_id: C-4471, segment: Engaged, extracted_at: 2026-03-11. Both records exist. The entity&#39;s identity is stable. An attribute evolved. No schema migration. No SCD logic in the core model. No broken dashboards.</p><p>The star schema still gets built — but at the serving layer, as a projection of the underlying entity/event data, shaped for the specific BI tool that needs it. The same entity/event data can produce different star schemas for different consumers simultaneously. Finance gets their star with dim_customer containing segmentation. Marketing gets a different star with dim_customer including a derived activity_status attribute. The data scientist gets a denormalized flat table for feature extraction. Each rendering is disposable and replaceable without touching the underlying data product.</p><h3>Entity subtypes that matter in practice</h3><p>Not all entities behave the same way.</p><p><strong>Reference entities</strong> change rarely and are referenced by almost everything else: stores, products, currencies, departments. These are the entities that traditional data warehousing would call “slowly changing dimensions.” In the entity/event model, they’re just entities with low change frequency. Each extraction captures the full state with a timestamp, and the serving layer decides whether to present the latest state, track history, or offer both.</p><p><strong>Transactional entities</strong> are created once, modified briefly, then become effectively immutable: orders, invoices, contracts, shipments. An order is created, transitions through a few statuses (pending → confirmed → shipped → delivered), then freezes. These have both entity characteristics (stable identity, evolving attributes during their active period) and event characteristics (they happen at a point in time, they carry measurable values). The source product captures them as the source system presents them — typically as a table of current state — and doesn’t force them into one category.</p><p><strong>Associative entities</strong> represent relationships between other entities: a customer-store loyalty membership, an employee-department assignment, a product-supplier contract. These have their own identity and lifecycle, and their primary purpose is linking other entities.</p><h3>Event subtypes that matter in practice</h3><p>Events also come in different shapes.</p><p><strong>Atomic events</strong> are single occurrences with clear boundaries: a point-of-sale transaction, an API call, a login event. One row per occurrence, immutable, append-only.</p><p><strong>State-change events</strong> record entity transitions: “Customer C-4471 changed segment from Gold to Engaged at midnight.” These are the most powerful pattern for building an emergent ontology, because they literally record business evolution as it happens.</p><p><strong>Aggregated events</strong> are summaries: “Store S-0042 had 1,247 visitors on March 15.” The individual occurrences weren’t captured, but the daily aggregate was. The grain is coarser, but the modeling principle is the same — something happened in a time period, involving specific entities, with measurable values.</p><blockquote>Yes. The same real-world occurrence can manifest as all three simultaneously depending on which source system recorded it.</blockquote><blockquote>A customer walks into a FreshMart store and buys groceries at 09:42 on March 15th.</blockquote><blockquote>The POS system records an <strong>atomic event</strong>: transaction T-9901, customer C-4471, store S-0042, €47.50, 09:42:17. One row, one occurrence, immutable.</blockquote><blockquote>That same transaction pushes the customer’s lifetime spend past €10,000, which triggers the loyalty system to record a <strong>state-change event</strong>: customer C-4471 changed tier from Silver to Gold at 09:42:17. Same moment. Same underlying occurrence. Different system, different lens.</blockquote><blockquote>The store’s footfall counter, which has no concept of individual transactions, records an <strong>aggregated event</strong> at end of day: store S-0042 had 1,247 visitors on March 15. That customer’s visit is one of the 1,247. Same occurrence, absorbed into a summary because the recording system operates at a coarser grain.</blockquote><blockquote>One person buying groceries. Three event subtypes. Three source systems.</blockquote><blockquote>This tells you something important about the classification: <strong>these aren’t types of events in the real world. They’re capture patterns — how the source system chose to record what happened.</strong> The real world just has things that happen. The source system decides the grain, the scope, and what to keep. A POS system captures the atomic grain because it needs to process each transaction. A loyalty engine captures state transitions because it cares about threshold crossings. A footfall counter captures daily aggregates because that’s the resolution of its sensor.</blockquote><h3>Part III: From Pipelines to Products</h3><h3>The data product as the unit of architecture</h3><p>In this paradigm, the data engineer’s primary artifact is not a pipeline, a dbt model, or an Airflow DAG. It is a <em>data product</em>: a declared unit that encapsulates what the data is, where it comes from, what quality it must meet, and how consumers can access it.</p><p>A data product is defined through declarative configuration — not imperative pipeline code. The engineer declares the inputs (source connections or upstream product references), the output schema (with explicit separation of identity columns from attribute columns), the quality constraints (checks that must pass before data is promoted to serving stores), and the serving intents (how different consumers need to access the data).</p><p>The platform handles orchestration, execution, quality enforcement, and serving. The engineer never writes an Airflow DAG, never manages a cron schedule, never manually provisions a database table. They describe <em>what</em> the data product is, and the platform figures out <em>how</em> to make it happen.</p><p>This inverts the traditional relationship between the data engineer and the infrastructure. In the pipeline world, the engineer is a plumber — they build and maintain the pipes. In the product world, the engineer is a domain translator — they understand what the business does and declare data products that capture it.</p><h3>Three archetypes — source, aggregate, consumer</h3><p>Data products fall into three archetypes, following the pattern Dehghani identified in data mesh:</p><p><strong>Source-aligned products</strong> capture entities and events directly from operational systems. One per source system. The transformation is minimal — light cleansing, type coercion, timestamp standardization. The engineer’s job is to faithfully capture what the source gives them, not to reshape it. A CRM source product captures customers as entities. A POS source product captures transactions as events. A store management source product captures stores as entities.</p><p><strong>Aggregate products</strong> combine entities and events to produce enriched datasets. This is where business logic lives. An aggregate product joins order events with store, customer, and product entities to produce an enriched event — every order line now carries the context of the store it happened in, the customer who placed it, and the product that was sold. The grain stays at the event level. Entity attributes ride along for context.</p><p>Aggregate products are organized around <em>business capabilities</em>, not around specific reports or KPIs. “Store performance measurement” is a business capability. “Monthly revenue by store” is a report. The aggregate product serves the capability; the report is a serving intent that projects the product into a specific shape for a specific consumer.</p><p>This distinction is critical for durability. If the aggregate product were organized around the report, it would need to change whenever the report requirements changed. Organized around the capability, it remains stable as long as the business capability exists — which is usually years, not months.</p><blockquote><strong>Example: FreshMart’s finance domain.</strong></blockquote><blockquote>The wrong approach organizes around reports. The CFO asks for three reports, so a data engineer builds three aggregate products: monthly-revenue-by-store, monthly-revenue-by-store-by-segment, and weekly-top-products-by-region. Three products, three transformations, three quality gates. All three join the same source products, use the same revenue calculation, and differ only in their GROUP BY clause.</blockquote><blockquote>Six months in, someone finds a bug — gift card redemptions were being counted as revenue. The fix is a one-line change to the revenue formula. But the formula exists in three separate products. The engineer fixes product one, forgets product two. The CFO sees different revenue numbers depending on which report they open. Trust evaporates.</blockquote><blockquote>The right approach organizes around the capability. “Store performance measurement” is one aggregate product: store-performance. Its transformation joins orders to stores, customers, and products once, producing an enriched event at the order-line grain. No aggregation. No GROUP BY. The revenue formula — revenue - discount_amount - gift_card_amount — exists in exactly one place.</blockquote><blockquote>The three reports become serving intents on the same product.</blockquote><blockquote>When the segmentation model changes from Gold/Silver/Bronze to Engaged/At-Risk/Dormant, no code changes anywhere — the GROUP BY doesn’t care about specific values. When the CFO wants a fourth report (quarterly revenue by product category), it’s a new serving intent — four lines of SQL, no new transformation, no new quality gate. When the revenue bug is found, the fix is in one place and all serving intents get the corrected number on the next materialization.</blockquote><blockquote>The boundary test: “store foot traffic analysis” sounds like it could be another serving intent. It shouldn’t be. It uses a different primary source (footfall counters, not POS), it’s owned by a different team (store operations, not finance), and it changes for different reasons (sensor technology changes, visit definition changes). Different capability, different owner, different evolution timeline — different product.</blockquote><p><strong>Consumer-aligned products</strong> are shaped for specific use cases: ML feature stores, API-ready views, reporting datasets optimized for a particular tool. These are the most volatile and the most disposable. When a use case dies, the consumer product retires, but the aggregate product it consumed is unaffected.</p><h3>Internal decomposition — one product is not one query</h3><p>A common objection is worth addressing head-on: if each aggregate product is a single, self-contained transformation, what happens when that transformation is 400 lines of SQL joining twelve source products with conditional logic for regional variations, currency conversions, and fiscal calendar alignment?</p><p>The answer: aggregate products can and should decompose internally. CTEs, staging views, helper functions, modular SQL files — all of these are valid inside a data product. The product is not “one query.” It is a self-contained unit of deployment with a single owner, single quality gate, and single version. Internally, it can be as well-structured as any software module.</p><p>The critical difference from a dbt DAG is <em>scope of visibility</em>. In a dbt project, intermediate models are independently addressable artifacts. Other models can depend on them, they can be run independently in broken states, and they create coupling across what should be product boundaries. A data product’s internal CTEs and staging views are implementation details — no external product can reference them. They’re private methods, not public APIs.</p><p>This doesn’t mean anything goes inside a product. The same software engineering principles that make code maintainable apply here: meaningful naming, single-responsibility functions, clear separation of join logic from business logic, testable units. But the decomposition happens <em>inside</em> the product boundary, contained and invisible to consumers.</p><p>The practical test for when one product should become two: different team ownership, different evolution timeline, different consumer base. If any two of those three diverge, it’s probably two products. The “same business reason” heuristic is a starting point, but these three concrete signals are what you actually measure.</p><h3>Identity separation as a structural invariant</h3><p>Every data product, regardless of archetype, enforces a structural invariant: <strong>identity columns are separated from attribute columns in the schema declaration.</strong></p><p>Identity columns are immutable and governed by global standards. They’re the customer_id, store_id, order_id that other products reference. They can never be changed in a version update, because changing an identity would break every downstream reference.</p><p>Attribute columns can evolve freely. They can be added (a minor version bump — additive, no downstream breakage), deprecated (a minor version with warning, consumers notified), or removed (a major version bump, requiring downstream acknowledgment).</p><p>This separation means that when a KPI retires, it’s an attribute that disappears — the entity it described still exists, its identity still resolves, and every downstream product that referenced the entity (not the specific KPI) continues working.</p><p>Udi Dahan, the creator of NServiceBus and one of the foremost thinkers on service-oriented architecture, provides the design heuristic here: microservices should be built around entity <em>attributes</em> that change together, not around entities themselves. Amazon’s product page is composed by dozens of microservices — one owns the title, another the price, another the reviews. They all share the same product identifier, but they change independently. Applied to data products: the identity is the shared anchor, and the attributes are independent concerns that evolve at their own pace.</p><h3>Part IV: Dimensional Modeling Doesn’t Die — It Moves</h3><h3>The serving layer as projection</h3><p>Star schemas are not wrong. They remain the best interface for BI tools. Tableau, Power BI, Superset, and Looker all work best when they can join a central fact table to surrounding dimension tables. The insight of this paradigm is that star schemas are a <em>rendering</em> of the data, not the data itself.</p><p>The same underlying entity/event data product can simultaneously produce:</p><ul><li>A star schema in PostgreSQL for executive dashboards, with fact_store_performance surrounded by dim_store, dim_customer, dim_product, and dim_date.</li><li>A denormalized flat table in a columnar store like StarRocks or ClickHouse for ad-hoc analyst exploration at full grain.</li><li>A key-value projection in Redis for operational lookups (“what’s store S-0042’s current month-to-date revenue?”).</li><li>An API-shaped view in PostgreSQL for mobile applications with paginated access.</li><li>A metadata endpoint for AI agents to discover, evaluate, and query the product.</li></ul><p>One transformation. One quality gate. Multiple projections. Each shaped for a different consumer. Each disposable and replaceable without touching the core product.</p><p>This means the Kimball vs. Inmon debate evaporates. You don’t choose one approach for your entire architecture. You produce whatever shape each consumer needs from the same underlying entity/event truth. Finance gets a Kimball star. The data science team gets a denormalized One Big Table. The ops team gets a key-value cache. The AI agent gets metadata and a connection reference. Each projection is a serving intent — a few lines of configuration that describe the shape and the target store.</p><h3>SCD logic as a serving concern</h3><p>Slowly changing dimensions are one of the most complex aspects of traditional data warehousing. In Kimball’s methodology, you must decide for each dimension attribute whether to use Type 1 (overwrite), Type 2 (track history with valid-from/valid-to dates), or Type 3 (keep current and previous values in separate columns).</p><p>In the entity/event model, this decision is unnecessary at the data product level. The underlying Iceberg data naturally contains every version of every entity attribute, timestamped at extraction. The SCD type is chosen at the serving layer based on what each consumer needs.</p><p>If the finance team wants a Type 1 dimension (always latest values), the serving intent selects the most recent record per entity. If they want a Type 2 dimension (full history), the serving intent generates valid_from and valid_to columns from the timestamp sequence. If marketing wants a Type 3 dimension (current and previous segment), the serving intent uses a window function to grab the current and lag values.</p><p>The same entity data, three different SCD implementations, no additional ETL logic. The choice is deferred to the last possible moment and can be changed without rebuilding any pipeline.</p><h3>The serving layer needs governance, not just automation</h3><p>This is where I need to be honest about a complexity trade-off. Moving dimensional modeling, SCD logic, and consumer-specific shaping to the serving layer doesn’t make that work disappear. It <em>displaces</em> it from the transformation layer to the serving layer. Whether that’s a net improvement depends on how well the serving layer is governed.</p><p>Each serving intent needs an owner — typically the consumer team that depends on it, not the data product owner. The finance team owns their star schema intent. The ML team owns their feature store intent. When the upstream aggregate product bumps a version, even an additive minor version, the platform can verify that all serving intents still <em>execute</em>. But it cannot verify they still produce <em>semantically correct</em> output. A new column in the aggregate product won’t break any existing serving intent, but it might represent a dimension attribute that should be added to dim_store — and only a human familiar with the finance team&#39;s reporting needs would know that.</p><p>The governance model for serving intents has three components:</p><p><strong>Assertion tests.</strong> Each serving intent should have automated assertions that run after every materialization — expected row counts within tolerance, expected value ranges on key columns, expected schema. These are the equivalent of dbt tests, but scoped to the serving projection. They catch execution-level failures automatically.</p><p><strong>Semantic review on version bumps.</strong> When the upstream aggregate product changes (any version bump), the platform notifies all serving intent owners with a diff: “Here’s what changed in the aggregate. Here are your intents that reference affected columns.” The intent owner reviews and decides whether their projection needs updating. This is a human review that the platform facilitates but cannot automate.</p><p><strong>Ownership tracking.</strong> The platform tracks who owns each serving intent and who consumes its output. Orphaned intents — those with no active owner or no active consumers — are flagged for retirement. This prevents the serving layer from accumulating dead projections the same way warehouses accumulate dead tables.</p><p>The serving layer’s complexity is <em>essential</em> complexity — it reflects the genuine diversity of consumer needs. The pipeline chain’s complexity was <em>accidental</em> — it came from managing state across multiple independently deployed transformations. Essential complexity can be governed. Accidental complexity can only be endured.</p><h3>Part V: Surviving Business Evolution</h3><h3>The three enemies of data integrity</h3><p>Every data platform eventually degrades. The degradation comes from three sources that traditional architectures fail to prevent:</p><p><strong>Mutation</strong> — overwriting historical data so you can’t tell what changed or when. When a pipeline does an UPDATE to reflect a segmentation change, the old segment values are gone. Six months later, nobody can explain why last quarter&#39;s numbers look different from what they remember.</p><p><strong>Tight coupling</strong> — binding downstream consumers to specific column names, value domains, or schema details instead of to business intents. When a column name changes, every downstream query breaks. When a value domain changes (Gold → Engaged), every filter breaks.</p><p><strong>Big-bang schema changes</strong> — forcing all consumers to update simultaneously when a producer changes. A single upstream schema change triggers a cascade of failures across every dependent dbt model, every dashboard, every API.</p><h3>How the paradigm addresses each one</h3><p><strong>Mutation is eliminated through append-only storage with snapshot isolation.</strong> Apache Iceberg, the open table format that now powers most modern lakehouse architectures, provides the architectural foundation. As the Iceberg specification states, reads are isolated from concurrent writes and always use a committed snapshot. Writes support adding and removing files in a single atomic operation and are never partially visible.</p><p>In practice, this means every materialization writes a new Iceberg snapshot. The previous snapshot remains available for time-travel queries. Monday’s data says the customer segment was Gold. Tuesday’s data says it’s Engaged. Both snapshots exist. Consumers querying at any historical point see the data as it was at that point. Nothing was overwritten.</p><p>Quality gates run against the new snapshot before it becomes visible to consumers. If the data fails quality checks, the snapshot is expired and garbage-collected. Consumers continue reading the previous good snapshot. They experience stale-but-correct data during failures, never fresh-but-wrong data. This is the “no bad data served” guarantee that becomes architecturally possible only when you own the execution stack.</p><p><strong>Tight coupling is broken through identity separation and semantic intent contracts.</strong> Instead of consuming a specific column named customer_segment with expected values [&#39;Gold&#39;, &#39;Silver&#39;, &#39;Bronze&#39;], a downstream product declares a semantic intent: &quot;I need customer classification data refreshed daily.&quot; When the upstream product changes its segmentation methodology, the semantic intent is still satisfied — it&#39;s still customer classification data, still refreshed daily. The contract holds even as the implementation evolves.</p><p>This echoes Dahan’s approach to service boundaries: a service is the technical authority for a specific business capability, and all related data and business rules reside within the boundaries of the service. Other services don’t bind to internal implementation details — they bind to the capability itself.</p><p><strong>Big-bang changes are prevented through version coexistence.</strong> When a data product bumps from v1 to v2, the platform doesn’t immediately retire v1. Both versions coexist during a configurable deprecation window. Consumers pinned to v1 continue receiving v1 data. They get notifications about the upcoming retirement with a deadline. They migrate on their own schedule, not on the producer’s schedule. After the deprecation window, v1 stops materializing and consumers still pinned to it get explicit errors directing them to update.</p><p>This transforms schema evolution from a “stop the world” event into a gradual migration that each consumer handles independently.</p><h3>Corrections, errors, and legitimate mutation</h3><p>The append-only model described above handles business evolution cleanly — the segmentation changed, both values coexist historically, the entity survives. But not all changes are evolution. Sometimes the old data was simply <em>wrong</em>.</p><p>A store’s square meterage logged as 240 instead of 2400 is not business evolution. It’s a data entry error. A transaction amount recorded with a decimal point in the wrong place is a mistake that should be corrected. The append-only model needs a correction story, and the article would be dishonest without one.</p><p>The answer is <strong>correction events</strong> — a specific pattern that distinguishes “the business changed” from “the previous value was wrong.”</p><p>When a data entry error is identified, the source product emits a correction record with metadata that links it to the record it corrects:</p><pre>store_id | sqm  | _extracted_at          | _correction_of<br>S-0042   | 2400 | 2026-03-17T10:00:00    | 2026-03-15T02:00:00</pre><p>The _correction_of field points to the timestamp of the record being corrected. Downstream consumers can handle this according to their needs. A BI serving intent might replace the old value in its projection (effectively Type 1 behavior for corrections). An audit serving intent keeps both records with the correction flagged. An aggregate product with period-specific materializations can trigger a backfill of affected time periods using the corrected value.</p><p>The key design principle: corrections propagate <em>through the same pipeline</em> as normal data. They aren’t side-channel operations that bypass quality gates. A correction record is ingested, passes quality checks, and promotes to serving stores just like any other record. The metadata distinguishes it from evolution, but the mechanics are identical.</p><p>For regulatory deletion requirements like GDPR right-to-erasure: Iceberg supports row-level deletes through position delete files. The deletion is a metadata operation that marks specific rows as removed. On the next compaction, the data files are physically rewritten without the deleted rows, and a new snapshot is created. This satisfies the regulatory requirement while maintaining the snapshot chain for non-deleted data. The platform should treat regulatory deletions as a first-class operation with audit logging, not as an afterthought.</p><p>For financial restatements — where a company needs to revise previously reported figures — the correction event pattern extends naturally. The restated values are appended with a _restatement_period marker, and serving intents that target financial reporting can be configured to show &quot;as originally reported&quot; or &quot;as restated&quot; views from the same underlying data.</p><h3>The ontology as emergent property — with honest caveats</h3><p>Here’s the payoff of the entire paradigm, but also where I need to be most careful about overclaiming.</p><p>Traditional approaches to building business ontologies — the maps of entities, relationships, capabilities, and processes that describe what a business actually does — treat ontology creation as a prerequisite. You hold workshops. You interview stakeholders. You document entity definitions. You build a canonical data model. Then you implement it.</p><p>This rarely works at scale. The workshop produces the <em>aspirational</em> ontology, not the <em>real</em> one. The people who know the real entities and processes are not the people who attend workshops. And by the time the canonical model is implemented, the business has already changed.</p><p>In the entity/event paradigm, much of the ontology can <em>emerge</em>. Every source product that declares “I capture the customer entity&quot; adds a node to the graph. Every aggregate product that references customer_id and store_id adds edges. Every serving intent that projects the data into a star schema reveals which entities are dimensions in which context. Every attribute deprecation records a death. Every new attribute records a birth.</p><p><strong>But emergence has limits, and those limits matter.</strong></p><p>Emergence works within a bounded context — a single domain where one team has enough shared understanding to self-correct. The CRM team knows what customer means to them. The billing team knows what customer means to them. The problem is that these might be <em>different things with the same name</em>.</p><p>Eric Evans, in <em>Domain-Driven Design</em>, calls this the bounded context problem. Martin Fowler, in his extensive writing on DDD, emphasizes that cross-context integration requires <em>deliberate</em> patterns — not emergent discovery. The patterns Evans identifies are:</p><p><strong>Shared kernel</strong> — two domains agree that they’re talking about the same entity and commit to a shared definition. Both teams own the definition jointly. Changes require agreement from both. This is appropriate for genuinely shared entities like customer_id when it truly means the same thing across domains.</p><p><strong>Customer-supplier</strong> — one domain produces an entity definition, and other domains consume it. The producer commits to stability. Consumers accept the producer’s definition. This is appropriate when one domain is the clear authority (CRM is the authority on customer identity, other domains accept that).</p><p><strong>Anticorruption layer</strong> — two domains have entities that look similar but aren’t. Each domain maintains its own definition and builds a translation layer at the boundary. This is appropriate when Sales’ “customer” (an account with a contract) is fundamentally different from Marketing’s “customer” (an individual who receives campaigns).</p><p><strong>Published language</strong> — domains agree on a shared vocabulary for cross-domain communication without merging their internal models. This is appropriate for reporting contexts where multiple domains contribute to a shared view.</p><p>The platform can <em>detect</em> when cross-context integration is needed — when two source products declare entities with the same name, or when an aggregate product joins entities from different domains. But the platform cannot <em>decide</em> which integration pattern is appropriate. That’s a political and organizational decision. “What is a customer?” is frequently a power struggle between Sales, Marketing, and Finance, and no amount of schema inference resolves it.</p><p>What the platform does: when the second source product in a different domain declares a customer_id, it flags the conflict and forces a decision. &quot;Is this the same customer entity as the one in crm-customers (shared kernel), or a different concept that shares an identifier (anticorruption layer), or a downstream consumer of the same identity (customer-supplier)?&quot; A human makes the call. The platform records and enforces it.</p><p>This governance is not a “formalization step” that happens at month six. It happens the first time two domains touch the same entity name, which is usually week two. The ontology <em>structure</em> emerges from usage. The <em>semantics</em> — what things actually mean and how they relate across domains — require human governance from the start.</p><p>After twelve months of normal operation, with this governance in place, the platform’s catalog contains a complete map of the business: which entities exist, when they appeared, which attributes they’ve gained and lost, which capabilities they participate in, how they relate across domains (and which integration pattern governs each relationship), who consumes them, and how they’re served. This map was never fully designed in a workshop. It was assembled from declarations, usage patterns, and governance decisions recorded over time.</p><h3>Part VI: The Practical Shift</h3><h3>What the data engineer’s job becomes</h3><p>The engineer stops building pipelines and starts declaring products. Their work is:</p><p><strong>Recognizing entities and events in source systems.</strong> Looking at the CRM and saying “customers are entities, deals are transactional entities, activities are events.” Not designing a model — just recognizing what the source gives them.</p><p><strong>Writing source product declarations.</strong> Choosing the right extraction strategy (incremental vs. full refresh), defining identity and attribute columns, writing quality checks. This is genuine engineering work, but it’s scoped to one product at a time.</p><p><strong>Writing aggregate product transformations.</strong> This is where business logic lives — the SQL or Python that joins entities and events into enriched datasets. One transformation per business capability, internally decomposed as needed for maintainability, but self-contained within the product boundary.</p><p><strong>Declaring serving intents.</strong> “Finance needs a star schema in PostgreSQL. The ops team needs a Redis cache. The AI agent needs metadata access.” Each intent is a configuration that shapes the canonical dataset for a specific consumer.</p><p>That’s it. No pipeline orchestration. No infrastructure management. No staging layer maintenance. No SCD implementation in the core model. No Kimball vs. Inmon debates.</p><h3>What happens to existing skills</h3><p>Data engineers trained in dbt, Airflow, Kimball modeling, and medallion architecture bring valuable background knowledge. The SQL skills transfer directly — aggregate product transformations are SQL. The data modeling intuition transfers — recognizing entities, events, grain, and business processes is exactly what Kimball trained people to do. What changes is where these skills are applied: not in building and maintaining pipeline chains, but in declaring products and their serving projections.</p><p>The closest existing skill set is actually someone who’s done domain-driven design in software engineering. They think in terms of entities, events, bounded contexts, and ownership — which maps directly to how data products work. A software engineer who understands DDD will pick up this paradigm faster than a data engineer who’s spent ten years writing dbt models, because the dbt engineer has to unlearn the pipeline mindset first.</p><h3>The honest adoption path — coexistence, not replacement</h3><p>I won’t pretend this paradigm can be adopted in six months. Any organization with an existing data platform — warehouse, dbt models, Airflow DAGs, Tableau dashboards — faces a multi-year coexistence period. Pretending otherwise would undermine credibility with anyone who’s actually done a platform migration.</p><p>Martin Fowler’s strangler fig pattern is the right mental model. You don’t replace the old system. You grow the new system around it, gradually redirecting traffic until the old system can be decommissioned.</p><p><strong>Phase 1: Wrap, don’t replace (months 1–3).</strong> The first data products don’t extract from operational systems. They wrap <em>existing</em> warehouse tables as source inputs. The old pipeline keeps running. The new product layer sits downstream of it, adding quality gates, serving intents, and catalog metadata that the old system lacked. The business sees immediate value — their existing data, now with SLAs and lineage — without any disruption to current workflows.</p><p><strong>Phase 2: Bypass selectively (months 3–12).</strong> As trust builds, individual source products start extracting directly from operational systems, bypassing the old pipeline for <em>that specific source</em>. The old pipeline continues running for its existing consumers that haven’t migrated. Both systems coexist. Some data flows through both paths simultaneously — the old pipeline feeding legacy consumers and the new product feeding new consumers. This dual-path period is uncomfortable but unavoidable.</p><p><strong>Phase 3: Migrate consumers (months 6–24).</strong> Consumer-by-consumer, reporting and analytical workloads are migrated. A Tableau dashboard is repointed from the old warehouse table to the new serving store. An ML model is retrained on features from the new product instead of the old mart table. This is the slowest phase because it requires coordination with every consumer team, and some teams will resist migration for perfectly valid operational reasons.</p><p><strong>Phase 4: Decommission selectively (months 12–36).</strong> When all consumers of a specific old pipeline path have migrated to the new products, that path is decommissioned. This happens path by path, not all at once. Some legacy paths may never fully decommission — if the cost of migration exceeds the cost of maintenance, it’s rational to leave them running.</p><p>The total timeline for a mid-size enterprise is two to three years. For a large enterprise with hundreds of existing pipelines and thousands of dashboard consumers, it could be longer. The paradigm’s value is realized incrementally — every product deployed adds quality gates, lineage, and serving flexibility that didn’t exist before — but the full transition is a multi-year program, not a quarterly project.</p><h3>Testing strategy</h3><p>The article would be incomplete without addressing how data products are tested. Quality gates on data (null ratios, uniqueness, range checks, freshness) are necessary but not sufficient. The transformation logic itself — the SQL or Python that turns inputs into outputs — needs testing.</p><p><strong>Unit tests for transformation logic.</strong> Each aggregate product’s transformation should have tests with fixed input datasets and expected output datasets. You provide known input records, run the transformation, and assert that the output matches expectations. This catches bugs in join logic, calculation errors, and edge cases (null handling, timezone conversions, fiscal calendar boundaries). These tests run in CI before the product is deployed, not in production after data has already flowed.</p><p><strong>Contract tests between products.</strong> When product B declares a dependency on product A, the contract (expected schema, expected identity columns, expected freshness) should be verifiable without running the full pipeline. Contract tests assert that product A’s declared output satisfies product B’s declared input requirements. If product A bumps a version that would violate product B’s contract, the contract test fails before deployment.</p><p><strong>Serving intent assertion tests.</strong> Each serving intent has automated assertions that run after every materialization: expected row counts within tolerance, expected value ranges on key measures, expected schema in the target store. These catch execution-level failures in the serving projection.</p><p><strong>End-to-end canary tests.</strong> Synthetic products with deterministic data run through the full lifecycle: extraction, transformation, quality gates, promotion, serving. If a canary product produces unexpected results, the platform itself has a bug. One canary is designed to pass all quality checks (verifying that good data is served). Another is designed to fail quality checks (verifying that bad data is blocked). If the “bad” canary somehow passes, the quality system has a false negative — the most dangerous kind of bug.</p><h3>Part VII: The Current State of Tooling</h3><p>I owe the reader honesty about what exists today versus what needs to be built.</p><p><strong>What exists and is production-ready:</strong></p><p>Apache Iceberg provides the append-only, snapshot-isolated storage layer described throughout this article. Time travel, schema evolution, and atomic commits are real capabilities available today through Spark, Trino, Flink, and most cloud data platforms.</p><p>Declarative orchestration frameworks like Dagster (with its asset-based programming model) provide a closer fit to product-oriented thinking than traditional DAG-based orchestrators like Airflow. Dagster’s software-defined assets are conceptually close to data products, though they don’t enforce all the patterns described here.</p><p>dbt can serve as the transformation and serving layer for teams that want to adopt this paradigm incrementally. dbt models can be organized into product-like boundaries, and dbt tests can implement quality gates. It’s not the ideal tool for this paradigm, but it’s a pragmatic starting point.</p><p>Data catalogs like OpenMetadata, DataHub, and Atlan can serve as the ontology store — recording entities, attributes, lineage, and usage patterns.</p><p><strong>What exists but requires significant configuration:</strong></p><p>Multi-store serving — promoting the same quality-gated dataset to PostgreSQL, Redis, StarRocks, and an MCP endpoint simultaneously — is technically possible with custom IO managers in Dagster or custom operators in Airflow, but there’s no off-the-shelf solution that handles it declaratively from a single configuration.</p><p>Quality gates that are architecturally mandatory (not opt-in) and that operate on Iceberg snapshots before promotion require custom engineering. Great Expectations and Soda can run checks, but the stage-then-promote pattern tied to Iceberg snapshot isolation needs orchestration glue.</p><p>Version coexistence — running v1 and v2 of a data product simultaneously during a deprecation window — is not a feature of any existing tool. It requires custom control plane logic.</p><p><strong>What doesn’t exist yet:</strong></p><p>A unified platform that takes declarative product manifests (YAML files declaring inputs, outputs, quality gates, and serving intents) and handles the full lifecycle — validation, deployment, orchestration, quality enforcement, multi-store promotion, version management, catalog registration, and consumer notification — does not exist as an off-the-shelf product.</p><p>This is the platform gap. The framework described in this article requires either building this platform internally or waiting for the market to produce it. Several companies are working in this direction, and the convergence of Iceberg, declarative orchestration, and data product thinking suggests it’s a matter of when, not if.</p><p>In the meantime, the paradigm can be adopted <em>conceptually</em> with existing tools — organizing data products as bounded units with explicit contracts, even if the enforcement is manual rather than platform-automated. The principles (entities over dimensions, declaration over orchestration, serving-layer projection, append-only evolution) are applicable today. The full automation is coming.</p><h3>Conclusion: The Pipeline Era Is Over</h3><p>The framework proposed here is not a marginal improvement to existing data engineering practices. But neither is it an invention. It’s a synthesis — connecting domain-driven design, distributed systems architecture, data product thinking, and declarative infrastructure into a unified approach that none of those traditions provides on its own.</p><p>Data engineering, as practiced for three decades, has been organized around <em>how data moves</em>. This framework organizes around <em>what data is</em>. The pipeline was the unit of work; now the product is. Dimensions and facts were the modeling primitives; now entities and events are. The warehouse was the center of gravity; now the serving layer is a disposable projection. The ontology was a prerequisite; now it’s an emergent property — governed by humans at integration boundaries, but grown from usage rather than designed in workshops.</p><p>The complexity doesn’t disappear. It would be dishonest to claim otherwise.</p><p>Dependency chains between data products are not accidental complexity that this paradigm eliminates — they’re essential complexity that reflects real business relationships. Store performance depends on orders, which depend on customers and products. That dependency exists regardless of architecture. In the pipeline world, it was encoded as implicit references between dbt models or Airflow task dependencies. In this paradigm, it’s declared explicitly in product input manifests. The chain is the same length. What changes is that the chain is visible, governed, and version-managed rather than implicit, fragile, and discovered only when something breaks.</p><p>Similarly, the serving layer’s complexity — multiple projections, each with its own owner, tests, and governance — is essential complexity that reflects the genuine diversity of consumer needs. The pipeline chain encoded this diversity as intermediate models and mart tables scattered across a dbt project. This paradigm concentrates it in serving intents, which makes the complexity governable but doesn’t make it smaller.</p><p>What the paradigm <em>does</em> eliminate is accidental complexity: the state management across independently deployed transformation stages, the implicit coupling between models that should be independent, the medallion layers that exist for organizational convention rather than business necessity, the SCD logic baked into transformation code when it belongs in the serving projection. That accidental complexity is real, it’s expensive, and it’s gone.</p><p>The honest framing: this paradigm trades poorly-located complexity for well-located complexity. The total amount of essential complexity is unchanged — the business is still the business. But essential complexity that’s declared, bounded, tested, and owned is fundamentally different from essential complexity that’s scattered, implicit, and discovered through production failures.</p><p>The data engineer of the next decade doesn’t build pipelines. They declare products. They recognize entities and events in the business domain. They write transformations that enrich events with entity context. They declare serving intents that project the data into whatever shape each consumer needs. And they let the platform handle everything in between.</p><p>The ontology? It builds itself — with a human hand on the steering wheel at the boundaries that matter.</p><h3>References &amp; Further Reading</h3><ol><li><strong>Dehghani, Z.</strong> (2022). <em>Data Mesh: Delivering Data-Driven Value at Scale.</em> O’Reilly Media.</li><li><strong>Dehghani, Z.</strong> (2020). “Data Mesh Principles and Logical Architecture.” martinfowler.com.</li><li><strong>Kimball, R. &amp; Ross, M.</strong> (2013). <em>The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling.</em> 3rd Edition. Wiley.</li><li><strong>Kleppmann, M.</strong> (2017). <em>Designing Data-Intensive Applications.</em> O’Reilly Media.</li><li><strong>Kleppmann, M.</strong> (2014). “Turning the Database Inside Out with Apache Samza.” Strange Loop Conference. martin.kleppmann.com.</li><li><strong>Evans, E.</strong> (2003). <em>Domain-Driven Design: Tackling Complexity in the Heart of Software.</em> Addison-Wesley.</li><li><strong>Fowler, M.</strong> (2014). “Bounded Context.” martinfowler.com.</li><li><strong>Fowler, M.</strong> (2004). “Strangler Fig Application.” martinfowler.com.</li><li><strong>Dahan, U.</strong> (2014). “Finding Service Boundaries — Illustrated in Healthcare.” NDC London. particular.net.</li><li><strong>Apache Iceberg Specification.</strong> “Serializable Isolation.” iceberg.apache.org/spec.</li><li><strong>Reis, J.</strong> (2023). “Is Kimball Still Relevant?” joereis.substack.com.</li><li><strong>Skelton, M. &amp; Pais, M.</strong> (2019). <em>Team Topologies: Organizing Business and Technology Teams for Fast Flow.</em> IT Revolution Press.</li><li><strong>Moronta, S.</strong> (2025). “Beyond Kimball and Data Vault: A Hybrid Modeling Architecture for the Modern Data Stack.” Medium.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=953d3384d42f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/the-end-of-pipeline-engineering-a-synthesis-for-data-that-survives-business-evolution-953d3384d42f">The End of Pipeline Engineering: A Synthesis for Data That Survives Business Evolution</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Memory Problem in AI Agents Is Half Solved. Here’s the Other Half.]]></title>
            <link>https://medium.com/data-unlocked/the-memory-problem-in-ai-agents-is-half-solved-heres-the-other-half-ebbf218ae4d5?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebbf218ae4d5</guid>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[memories]]></category>
            <category><![CDATA[anthropic-claude]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Fri, 20 Feb 2026 10:40:16 GMT</pubDate>
            <atom:updated>2026-02-20T10:42:27.459Z</atom:updated>
            <content:encoded><![CDATA[<h4>Atleast i think it is</h4><p>I’ve been building data platforms for a while — enterprise environments with complex pipelines, regulatory requirements, the kind of systems where a wrong decision costs real money. Over the past year, I started integrating AI agents into my workflows. Not as toys. As actual tools doing actual work: writing pipeline code, debugging transformations, helping me reason through architectural decisions.</p><p>And I kept running into the same problem. Every time I started a new session, the agent had no idea what happened in the last one. I’d spent an hour the day before working through a tricky data validation issue with Claude. We’d tried three approaches. Two failed for specific reasons. The third worked but had a caveat. The next morning, I opened a new session to continue the work, and the agent suggested the first approach — the one we’d already proven didn’t work.</p><p>Every single session starts from scratch. The agent that spent 45 minutes yesterday figuring out your authentication flow? Gone.” Mem0 raised $24M to build a memory layer for AI applications. Letta raised $10M to build stateful agents. Claude Code recently shipped an auto-memory feature where Claude writes notes to itself between sessions. There are MCP servers for memory popping up weekly — CodeMem, Claude-Mem, ShieldCortex, and dozens more on PulseMCP.</p><p>The industry clearly agrees this is a problem. And a lot of smart people are building solutions. But after using several of these tools and thinking hard about what I actually need, I think the current generation of memory solves the easier half of the problem and leaves the harder half untouched.</p><h3>What the Current Memory Tools Do Well</h3><p>Let me give credit where it’s due. The tools being built right now are genuinely useful.</p><p>Mem0 extracts facts from conversations, stores them in a vector database with optional graph relationships, and retrieves relevant facts in future sessions. Their research shows a 26% accuracy improvement over OpenAI’s built-in memory and 90% token savings compared to stuffing full conversation history into the context. Those are real numbers.</p><p>Letta (formerly MemGPT) takes a more ambitious approach. Born out of UC Berkeley research, it treats the LLM like an operating system — agents actively manage their own memory using built-in tools for reading, writing, and searching persistent storage. Instead of a passive retrieval system, Letta agents decide what to remember, when to consolidate, and what to archive. Their latest architecture introduces sleep-time agents that handle memory consolidation asynchronously — the agent responds to you in real time while a background process distills and organizes what it learned.</p><p>Claude Code’s auto-memory is elegant in its simplicity. It creates a persistent directory where Claude writes notes about project patterns, key commands, and preferences. These notes load into every future session. The agent effectively maintains a notebook about your project that survives across restarts.</p><p>EchoVault stores session observations in SQLite with full-text search, writes Markdown files you can browse in Obsidian, and works across Claude Code, Cursor, and Codex through a single MCP server. One memory system for all your agents.</p><p>These tools solve the <strong>persistence</strong> problem. Information that used to vanish when you closed a session now survives. Your agent can remember that you prefer TypeScript strict mode, that the project uses pytest, that you fixed a race condition in auth.ts last Tuesday. That’s a meaningful improvement over the blank-slate experience.</p><p>But here’s what’s been nagging at me: they’re all storing <em>what happened</em>. None of them are storing <em>what we learned from what happened</em>.</p><h3>The Difference Between a Diary and a Notebook</h3><p>I’ve been trying to articulate this distinction for a while, and the clearest way I can put it is: the current memory tools are diaries. They record events. What I actually need is a notebook — something that records lessons.</p><p>My session logs say: “Session 47: Changed the connection pool from 10 to 20. CPU usage went up 15%. The timeout issue wasn’t resolved.”</p><p>The lesson inside that log is: “Increasing the connection pool causes CPU load without fixing timeouts in this service.”</p><p>Mem0 would extract the fact: “User changed connection pool to 20 in session 47.” EchoVault would record the observation: “Modified pool config, CPU increased.” Claude Code’s auto-memory might note: “Connection pool is currently set to 20.”</p><p>All accurate. All useful for context. None of them capture the lesson — the causal relationship between the action and the outcome, the thing that should prevent me (or the agent) from trying the same approach next time.</p><p>When the agent encounters a similar timeout issue three weeks later, it might retrieve the fact that the connection pool was changed to 20. But it doesn’t know <em>why</em> that happened, whether it <em>worked</em>, or whether it should try the same thing again. It has the diary entry without the margin note that says “this didn’t help — look at the fraud check service instead.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H3viunO7vOFcf05HY-Ar-Q.png" /></figure><p>The academic world has a useful vocabulary for this. Research on long-term memory in AI agents distinguishes between three types: <em>semantic memory</em> (facts and knowledge), <em>episodic memory</em> (records of specific experiences), and <em>procedural memory</em> (learned skills and behavioral patterns from experience). Most current tools are building semantic and episodic memory. What I’m describing — lessons extracted from outcomes that evolve with new evidence — is closer to procedural memory. The kind that captures how to do things, not just what things are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*JLEGWdTxnfSWJVtBqiFC7w.png" /></figure><h3>Why This Matters More Than It Sounds</h3><p>You might think this is a small distinction. The agent has the facts. It can figure out the lesson on its own, right?</p><p>Sometimes, yes. If the relevant session transcript is retrieved and the agent has enough reasoning capacity to re-interpret the events, it can re-derive the lesson. But this re-derivation has to happen every time. If five past sessions all demonstrate the same pattern, a retrieval system might pull chunks from all five sessions, and the agent processes five redundant data points to arrive at one insight. Every session, it’s doing interpretation work instead of decision work.</p><p>More importantly, the agent can’t always re-derive the lesson correctly from raw events. “Changed pool to 20, CPU went up” doesn’t tell you that the CPU increase is causally related to the pool change (vs. a concurrent deployment). It doesn’t tell you that three separate sessions all showed the same pattern, making the lesson more reliable. It doesn’t tell you that the lesson was valid in January but the service was rewritten in March, so it might not apply anymore.</p><p>These are the things I keep track of intuitively as a human working with the same codebase. The agent can’t, because nothing in the current memory tools captures them.</p><h3>What I Think Is Missing: Beliefs</h3><p>I started calling the missing layer a “belief store” because that’s what it feels like to me. Not a database of facts. Not a log of events. A collection of things we believe to be true about this system, based on accumulated experience, with varying degrees of confidence.</p><p>A belief is different from a fact or an event in three ways.</p><p>First, a belief is <strong>causal</strong>. It doesn’t just say what happened — it says what we think it means. “Increasing the connection pool causes CPU regressions without fixing timeouts” is a belief. “Connection pool was changed to 20” is a fact. The belief is more useful for decision-making because it tells the agent what to <em>expect</em> if it tries the same thing.</p><p>Second, a belief has <strong>confidence</strong>. A belief confirmed by five independent sessions is more reliable than one based on a single observation. A belief that was last confirmed six months ago is less reliable than one confirmed last week. Facts don’t have confidence — the connection pool either is 20 or it isn’t. Beliefs live on a spectrum, and the agent should weight them accordingly.</p><p>Third, a belief <strong>evolves</strong>. When new evidence supports it, confidence goes up. When new evidence contradicts it, confidence goes down. When no evidence touches it for long enough, it decays. Facts are static until they’re updated. Beliefs are dynamic — they reflect the system’s evolving understanding of how things work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHyBvd-t_pHyEflXShhrVg.png" /></figure><h3>The Research That’s Getting Close</h3><p>Here’s what’s interesting: academia is converging on this problem from several directions. Nobody has shipped the full pattern as a production tool, but the pieces exist.</p><p><strong>Reflexion</strong> is the closest predecessor. Published at NeurIPS 2023 by Shinn et al., it introduced a simple but powerful idea: instead of updating model weights after failures, convert the failure signal into verbal feedback and store it in an episodic memory buffer. The agent retries with its own plain-language reflection as additional context. On coding benchmarks, Reflexion improved accuracy by 14% on reasoning tasks without the agent ever seeing the correct answer — it just got better at analyzing its own mistakes. The framework converts binary success/failure signals into what the authors call “semantic gradients” — natural language descriptions of what went wrong that are more informative than scalar rewards.</p><p>This is close to belief extraction, but it operates within a single task across retries, not across different tasks over time. It’s the right extraction mechanism pointed at the wrong timescale.</p><p>A more recent paper — Multi-Agent Reflexion (MAR), from December 2025 — addresses the obvious weakness in single-agent reflection: confirmation bias. When the same model generates actions, evaluates its own behavior, and produces reflections, it tends to reinforce its own reasoning patterns rather than genuinely learning. MAR separates acting, diagnosing, critiquing, and aggregating into different agents. Their results showed meaningful gains over single-agent Reflexion — a 6.2-point improvement on HumanEval coding benchmarks. This resonates with something I’ve been thinking about: the model extracting the lesson shouldn’t be the same model that made the decision.</p><p><strong>Memᵖ</strong> (from Zhejiang University and Alibaba) tackles the problem I care about most directly. It’s a procedural memory framework that distills past agent trajectories into both fine-grained step-by-step instructions and higher-level script-like abstractions. The critical piece is the update loop: as the agent completes more tasks, its memory evolves — not by simply adding new entries, but by reflecting on failures to correct and revise existing procedures. The researchers found that the most effective strategy wasn’t just accumulating successful experiences, but actively revising the memory when things went wrong. That’s belief evolution, expressed as procedure refinement.</p><p>One finding from Memᵖ should interest anyone building in this space: procedural memory transfers across models. When memories generated by GPT-4o were given to a much smaller model (Qwen2.5), the smaller model saw significant performance gains. This means you could run belief extraction with a capable model and deploy the accumulated knowledge on cheaper, faster models. The beliefs are portable.</p><p><strong>Causal Knowledge Graphs</strong> represent the heavy-infrastructure version of what I’m describing. The CausalKG framework, proposed by Jaimini and Sheth, goes beyond flat beliefs to represent causality as a graph structure — nodes are entities (services, configurations, metrics), edges are causal relationships annotated with effect sizes and confidence levels. This supports something flat belief stores can’t: counterfactual reasoning. The agent can ask not just “does X cause Y?” but “if I hadn’t changed X, would Y still have happened?” Fujitsu has been developing a commercial Causal Knowledge Graph that extracts known causal relationships from documents, estimates unknown ones from data, and integrates them into a single queryable structure. It’s industrial-grade but heavy — probably overkill for a solo developer, potentially essential for an enterprise platform.</p><p>The emerging concept of a <strong>Personal Causal Knowledge Graph</strong> (PCKG) is particularly relevant. It encodes causal relationships specific to an individual’s context — habits, events, attributes, and their effects — with continuous adaptation as new events are logged. When data is sparse, it falls back to hypothesis-based generation via LLMs. When queried, it runs goal-directed causal traversals to enumerate plausible explanatory chains in both directions: upstream (what caused this?) and downstream (what will this cause?). This is essentially what I want for a codebase: a personal causal model of how this system behaves, built from accumulated agent experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*3igXXz97pT37VMA-JfcEFQ.png" /></figure><h3>How I’m Thinking About the Loop</h3><p>Based on what exists in research and what I actually need, here’s the pattern I’ve been working through.</p><p><strong>After each session, extract a lesson from the outcome.</strong></p><p>This is the key step, and Reflexion shows it works. An LLM looks at the structured outcome of a session — not just “done” or “failed” but what was attempted, what changed, what the measurable result was — and distills it into a belief. The raw outcome becomes a compressed, reusable lesson. The Memᵖ research suggests two useful formats: fine-grained step-by-step procedures (“when you see X, try Y before Z”) and higher-level abstractions (“timeout issues in this service are usually caused by downstream API latency, not connection pool sizing”).</p><p>But here’s where I diverge from Reflexion’s single-agent approach. Following the MAR insight, I think the extraction step should use a separate model — or better, multiple models. The agent that performed the work has context but also has bias. A fresh model looking at the session outcome can challenge the causal interpretation. Did the connection pool change actually cause the CPU spike, or did something else happen at the same time? This adversarial extraction step is where quality gets built in. Without it, you’re trusting a single model’s post-hoc rationalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kIsPYfx4lytjCJTyt16eSQ.png" /></figure><p><strong>Store beliefs with metadata that makes them useful.</strong></p><p>Each belief carries a confidence score, an evidence trail (which sessions support it), and age information. The evidence trail matters because I want to be able to question any belief — trace it back to the sessions that produced it and decide for myself whether it’s still valid. The confidence score tells the agent how much weight to put on this belief. The age enables decay — old unconfirmed beliefs gradually lose influence.</p><p>For most teams, this starts as a flat store — a JSON file or a SQLite table. If the belief count grows beyond a few hundred and cross-cutting relationships matter (belief about service A is relevant when working on service B that calls A’s API), a graph structure like Mem0g or the CausalKG pattern becomes worth the infrastructure cost. Start flat, go graph when flat breaks.</p><p><strong>Before the next session, relevant beliefs are retrieved.</strong></p><p>This is where you layer the belief store on top of existing memory tools. Mem0 or EchoVault handles fact retrieval and event context. The belief store handles lessons and causal knowledge. The agent sees both: what’s true right now (facts) and what we’ve learned over time (beliefs).</p><p>Letta’s architecture offers a useful pattern here: background consolidation. Instead of running belief extraction during the session (adding latency), run it asynchronously. The agent responds in real time using whatever beliefs already exist. After the session completes, a background process extracts new beliefs, resolves conflicts, and flags anything that needs human review. Next session, the updated beliefs are available. A slight delay before new knowledge is consolidated is a worthwhile trade for zero-latency agent responses.</p><p><strong>After the session, beliefs are updated.</strong></p><p>The outcome either confirms existing beliefs (confidence goes up), contradicts them (confidence goes down), or reveals something new (new belief created). The store evolves with every session. Over time, beliefs that keep getting confirmed become high-confidence foundations. Beliefs that get contradicted fade. New beliefs form as the agent encounters novel situations.</p><h3>Where the Existing Tools Could Go</h3><p>I don’t think the belief store needs to be a separate product. It could be a layer on top of what already exists.</p><p>Mem0 already has an update phase where new facts are compared to existing ones — the LLM decides whether to ADD, UPDATE, DELETE, or NOOP. Their graph variant, Mem0g, already models entity relationships. Adding causal relationships (“X causes Y, confidence 0.8”) and outcome tracking (“this belief was tested in sessions 47, 55, 62”) isn’t a huge conceptual leap from what they’ve already built. It’s an extension of their extraction pipeline.</p><p>Letta’s self-editing memory is perhaps the most natural fit. Their agents already have tools to modify their own memory blocks, insert facts into archival storage, and search past conversations. Adding belief-specific tools — write_belief, update_belief, deprecate_belief — would let the agent manage its own causal knowledge alongside its factual knowledge. Letta&#39;s sleep-time agents could handle belief consolidation asynchronously, running the reflection and extraction steps in the background.</p><p>Claude Code’s auto-memory already writes notes that persist across sessions. If those notes included not just “project uses pytest” but “last three attempts to optimize the query in reports.py made performance worse — investigate caching before trying query optimization,” that’s a belief with implicit confidence (three confirmations) that would actually change the agent’s behavior.</p><p>EchoVault stores observations in SQLite with full-text search. Adding a beliefs table with confidence scores, evidence links, and decay timestamps would give you a belief store alongside the observation store. The observations are the evidence. The beliefs are the compressed knowledge. You need both.</p><p>The pieces are there. What’s missing is the extraction step that goes from “what happened” to “what does it mean,” and the feedback loop that updates those meanings over time.</p><h3>What I’m Worried About</h3><p>I want to be honest about the parts of this that I haven’t figured out.</p><p><strong>Extraction quality is my biggest concern.</strong> The entire pattern depends on an LLM correctly interpreting outcomes and producing accurate causal beliefs. If it extracts the wrong lesson, that wrong lesson compounds across future sessions. I’ve seen it extract “reducing timeouts improves performance” when the actual cause was a concurrent deployment. The belief was wrong, and if it had persisted, the agent would have confidently applied timeout reductions to unrelated problems.</p><p>The MAR research validates this concern. Their core finding is that single-agent reflection systematically produces confirmation bias — the model repeats its own reasoning patterns rather than genuinely correcting them. Their solution is multi-agent critique, which is promising, but introduces its own complexity. My current thinking is to treat every extracted belief as a hypothesis: new beliefs start with low confidence, they need multiple independent confirmations before the agent should rely on them heavily, and anything above a certain confidence threshold gets flagged for human review. The Memᵖ research supports this — they found that the most effective memory update strategy is reflecting on failures to revise existing procedures, not just accumulating successful ones.</p><p><strong>The security angle is real.</strong> ShieldCortex exists because persistent memory creates a new attack surface. If a malicious webpage can inject instructions that get stored as a “memory” and influence future sessions, that’s a serious problem. Beliefs would have the same vulnerability but worse — a poisoned belief with high confidence could steer agent behavior for weeks. The evidence trail helps (you can audit where a belief came from), but it’s not a complete defense. The Palo Alto Unit 42 team demonstrated this exact attack on Amazon Bedrock Agents — injecting instructions through web content that persisted in the agent’s session memory and exfiltrated conversation history in future interactions.</p><p><strong>Belief conflicts are tricky.</strong> Session 31 says X causes Y. Session 58 says X doesn’t cause Y. Both are in the store. The codebase probably changed between the two sessions. But the system doesn’t automatically know which belief reflects the current state. I’m currently thinking about recency weighting (the newer observation gets more influence) and flagging conflicts for human review. It works, but it doesn’t scale well if nobody’s around to review. The Causal Knowledge Graph research offers a more sophisticated approach — counterfactual reasoning that can ask “what changed between session 31 and session 58 that might explain the different outcome?” — but that requires infrastructure I’m not ready to build yet.</p><p><strong>The cold start is real.</strong> A new installation has no beliefs. The first 10–20 sessions don’t benefit from accumulated knowledge. The agent is no better than a stateless one. The existing memory tools (Mem0, auto-memory) don’t have this problem as badly because they store facts from the first session onward. Beliefs require outcomes — you need sessions to complete and results to materialize before there’s anything to learn from. You’re looking at a month before the belief store starts meaningfully improving decision quality. The Memᵖ finding about cross-model transfer partially addresses this — you could potentially seed a belief store from a stronger model’s experience — but codebase-specific beliefs really do need codebase-specific experience to form. Teams need to understand that timeline.</p><h3>What I Think the Landscape Looks Like</h3><p>The agent memory space is moving fast. The survey paper “Memory in the Age of AI Agents” from December 2025 catalogues dozens of papers across episodic memory, experience-driven evolution, and procedural learning — new ones appearing almost weekly. Here’s how I see the layers shaping up:</p><p><strong>Layer 1: Persistence (largely solved).</strong> Store conversation history and events across sessions. Mem0, EchoVault, Claude-Mem, Claude Code auto-memory, Letta, and dozens of MCP servers handle this. The infrastructure is mature enough for production use. Vector databases, SQLite, graph stores — pick your backend.</p><p><strong>Layer 2: Fact extraction (being solved).</strong> Pull out structured facts from conversations. “User prefers TypeScript.” “Project uses PostgreSQL.” “Auth flow was refactored in January.” Mem0 does this well. Their extraction-and-update pipeline is production-grade. Letta’s self-editing memory lets agents manage their own factual knowledge actively. The research is strong — Mem0’s LOCOMO benchmark results show clear improvements over raw context stuffing.</p><p><strong>Layer 3: Belief extraction (still missing).</strong> Learn causal lessons from outcomes. Track confidence. Update beliefs when evidence changes. Decay old beliefs. Surface relevant beliefs before agent decisions. The academic research is converging — Reflexion showed verbal self-reflection works, MAR showed multi-agent critique improves extraction quality, Memᵖ showed procedural memory with dynamic updates improves agent performance over time, and CausalKG showed how to represent causal relationships in queryable structures. But nobody has shipped the full pattern — outcome-based belief extraction with confidence tracking, decay, and cross-session learning — as a production tool for coding agents. Not yet.</p><p>I think Layer 3 is where the next meaningful jump in agent capability comes from. Not because Layers 1 and 2 aren’t valuable — they are. But because the difference between an agent that remembers what happened and an agent that remembers what <em>worked</em> is the difference between an intern reading old meeting notes and a senior engineer who’s been through it before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4R7f6mWsFPPSDZ7aMIG_Zw.png" /></figure><h3>A Practical Path Forward</h3><p>Based on what the research supports and what I actually need, here’s what I’m building. Each phase builds on the last — nothing gets thrown away.</p><p><strong>Phase 1: Reflexion-style verbal reflections, stored persistently.</strong> After each session, run a reflection prompt that takes the structured outcome and produces a plain-language lesson. Store it as a Markdown file alongside the session log. Surface relevant lessons via keyword search before each new session. This is Reflexion extended across sessions instead of across retries within a single task. No databases, no graphs, no infrastructure. Just text files. The research says this alone should improve decision quality — Reflexion showed a 14% accuracy improvement just from verbal self-reflection.</p><p><strong>Phase 2: Procedural distillation with confidence tracking.</strong> Instead of storing raw reflections, distill them into the dual format that Memᵖ found most effective: fine-grained procedures (“when encountering timeout in checkout, check fraud check latency before touching connection pool”) and higher-level abstractions (“timeout issues in payment services are usually downstream, not infrastructure”). Add a simple confidence counter (how many sessions confirmed this) and a last-confirmed timestamp for decay. This is the belief store, minimal version. Flat file or SQLite. The Memᵖ finding about cross-model transfer means these beliefs can follow you if you switch from Claude to GPT or vice versa.</p><p><strong>Phase 3: Multi-agent extraction validation.</strong> Use separate models not for the task itself but for the belief extraction step. One model extracts the belief from the session outcome. A second model challenges the causal interpretation — is this actually causation or just correlation? A third decides whether the belief meets the bar for storage. This directly addresses the confirmation bias that MAR identified as Reflexion’s primary weakness. The models don’t need to agree — disagreement flags the belief for human review rather than storing it with false confidence.</p><p><strong>Phase 4: Graph-based belief store (if scale demands).</strong> If the flat store grows beyond a few hundred beliefs and cross-cutting relationships become important, migrate to a graph structure following the CausalKG or Mem0g pattern. Beliefs become nodes. Causal relationships become edges with effect sizes. The graph supports traversal queries: “what beliefs are relevant to the payment service?” returns not just direct beliefs but beliefs about downstream services it depends on. This is the enterprise path — only build it when the simpler approach hits a real wall.</p><h3>Where I’m Taking This</h3><p>I’m implementing this in my own workflows first. I already have the logging infrastructure (that’s where my symlink system came from). Phase 1 is next — the verbal reflection step after each session, stored as persistent files. Simple enough that I can ship it this week.</p><p>I’ll share what I find. Whether the belief store actually improves decision quality over time or just adds complexity. Whether the extraction step produces useful beliefs or garbage. Whether confidence scores mean anything in practice or are just numbers that make me feel better. Whether multi-agent extraction validation is worth the cost or whether single-agent reflection is good enough. I’m a data engineer, so I want to see the metrics before I declare this works.</p><p>For now, here’s what I’m fairly confident about: the memory problem is real, and it’s being worked on by smart people with real funding. The persistence and fact layers are maturing fast. The academic research on reflection, procedural memory, and causal reasoning is converging on the same gap I keep hitting in practice. But there’s still a gap between remembering what happened and learning from what happened, and that gap is where agents go from tools that know your preferences to systems that build genuine expertise over time.</p><p>The agents are capable enough. The memory tools are getting there. What’s missing is the learning loop that connects outcomes to knowledge. That’s the piece I’m trying to build.</p><p><em>If you’re working on similar problems — giving agents persistent knowledge, building memory layers, making AI systems learn from outcomes — I’d like to hear what you’re finding. The pattern is early enough that the best ideas will come from people hitting walls in different contexts. And if you’re using Mem0, Letta, EchoVault, or any of the other memory tools in production, I’m particularly interested in whether you’ve tried to get them to capture not just what happened but what it meant.</em></p><h3>References and Further Reading</h3><ul><li><strong>Reflexion: Language Agents with Verbal Reinforcement Learning</strong> — Shinn et al., NeurIPS 2023. The foundational paper on verbal self-reflection as a learning mechanism. <a href="https://arxiv.org/abs/2303.11366">arXiv:2303.11366</a></li><li><strong>MAR: Multi-Agent Reflexion</strong> — December 2025. Addresses confirmation bias in single-agent reflection by distributing critique across multiple agents. <a href="https://arxiv.org/html/2512.20845">arXiv:2512.20845</a></li><li><strong>Memᵖ: Exploring Agent Procedural Memory</strong> — Zhejiang University &amp; Alibaba. Dynamic procedural memory with build-retrieve-update loop and cross-model transfer. <a href="https://arxiv.org/html/2508.06433v2">arXiv:2508.06433</a></li><li><strong>Memory in the Age of AI Agents: A Survey</strong> — Comprehensive paper list covering episodic, semantic, and procedural memory research. <a href="https://github.com/Shichun-Liu/Agent-Memory-Paper-List">GitHub</a></li><li><strong>CausalKG: Causal Knowledge Graph Explainability</strong> — Jaimini &amp; Sheth, 2022. Framework for representing causal relationships in knowledge graphs with interventional and counterfactual reasoning. <a href="https://arxiv.org/abs/2201.03647">arXiv:2201.03647</a></li><li><strong>Letta (MemGPT) Research Background</strong> — UC Berkeley. Self-editing memory architecture where agents manage their own persistent state. <a href="https://docs.letta.com/concepts/letta/">docs.letta.com</a></li><li><strong>Mem0</strong> — Production memory layer for AI applications with extraction-update pipeline and graph variant. <a href="https://mem0.ai/">mem0.ai</a></li><li><strong>EchoVault</strong> — MCP-based cross-agent memory using SQLite and Markdown. Works across Claude Code, Cursor, and Codex.</li><li><strong>Remember Me, Refine Me</strong> — Dynamic procedural memory framework for experience-driven agent evolution, December 2025. <a href="https://www.arxiv.org/pdf/2512.10696">arXiv:2512.10696</a></li><li><strong>Fujitsu Causal Knowledge Graph</strong> — Commercial application of causal knowledge graphs combining document extraction with statistical causal discovery. <a href="https://www.fujitsu.com/global/documents/about/research/article/202410-causal-knowledge-graph/">White Paper</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebbf218ae4d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/the-memory-problem-in-ai-agents-is-half-solved-heres-the-other-half-ebbf218ae4d5">The Memory Problem in AI Agents Is Half Solved. Here’s the Other Half.</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Complete Guide to Autonomous Agent Development: Every Concept, Every Principle, From First…]]></title>
            <link>https://medium.com/data-unlocked/the-complete-guide-to-autonomous-agent-development-every-concept-every-principle-from-first-ef9b2342525a?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef9b2342525a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[autonomous]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Fri, 06 Feb 2026 21:12:12 GMT</pubDate>
            <atom:updated>2026-02-10T14:30:59.667Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Complete Guide to Autonomous Agent Development: Every Concept, Every Principle, From First Principles</h3><h4>Yes, the irony of writing a 5,000-word guide about keeping context windows clean is not lost on me.</h4><blockquote>Maybe start with a bit familiarity: <a href="https://www.anthropic.com/engineering/building-c-compiler">https://www.anthropic.com/engineering/building-c-compiler</a></blockquote><h3>PART 1: THE FUNDAMENTAL PROBLEM</h3><h3>1.1 Why This Matters — The Core Tension</h3><p>Software development has always had a fundamental tension: speed vs. quality. You can ship fast and break things, or you can ship slowly and carefully. Every team, every developer, every project sits somewhere on this spectrum.</p><p>What Carlini discovered is that LLM-based coding agents can potentially break this tradeoff. But only if you solve a specific problem first:</p><p><strong>How do you let an autonomous system work fast without it destroying quality?</strong></p><p>The answer, as it turns out, is the same answer that computer science has been giving for decades: you build systems that constrain behavior without limiting capability. Type systems do this for code. Test suites do this for functionality. Git does this for collaboration. Carlini’s contribution was figuring out how to apply these constraints to coding agents specifically.</p><h3>1.2 What Is an “Agent” in This Context?</h3><p>Let’s be precise. When we say “agent” in the context of autonomous coding tools, we mean:</p><p><strong>An LLM instance that has access to a computer (filesystem, terminal, network) and is given a goal rather than a specific instruction.</strong></p><p>This is different from normal LLM usage:</p><ul><li><strong>Normal:</strong> “Write me a function that sorts a list” → LLM writes the function → Done</li><li><strong>Agent:</strong> “Make the test suite pass” → LLM reads the tests, reads the code, identifies what’s wrong, writes fixes, runs the tests, sees new failures, fixes those, runs again, commits → Done (maybe)</li></ul><p>The key difference is <strong>autonomy</strong>. An agent decides what to do next. It observes the environment, makes a plan, acts, observes the result, and adjusts. This is powerful but dangerous — because if the agent’s understanding of “success” is wrong, it will confidently do the wrong thing.</p><h3>1.3 The Session Problem</h3><p>Here’s a constraint most people don’t think about carefully enough:</p><p><strong>Every LLM session starts from zero.</strong></p><p>The LLM has no memory between sessions. When a new session starts, the agent doesn’t know:</p><ul><li>What this project is</li><li>What it was working on last time</li><li>What’s been tried and failed</li><li>What’s currently broken</li><li>What patterns the codebase follows</li></ul><p>A human developer has all this context in their head. They sit down in the morning, remember what they were debugging yesterday, and pick up where they left off. A coding agent can’t do that. Every session is a brand new developer who’s never seen the codebase before.</p><p>This single constraint drives almost every design decision in the entire scaffolding system. If you understand nothing else, understand this: <strong>everything we’re building is compensation for the fact that the agent starts from zero every time.</strong></p><h3>PART 2: CARLINI’S PRINCIPLES — DEEP DIVE</h3><h3>2.1 The Agent Loop</h3><h4>The Concept</h4><p>The simplest possible agent architecture is a loop:</p><pre>while true:<br>    observe the environment<br>    decide what to do<br>    do it<br>    check if it worked<br>    repeat</pre><p>Carlini’s implementation is literally a bash while loop:</p><pre>while true; do<br>    coding-agent -p &quot;$(cat AGENT_PROMPT.md)&quot; --model &lt;model&gt;<br>done</pre><p>Each iteration is one complete agentic coding session. When the session ends (the agent decides it’s done, or it runs out of context), the loop starts a fresh session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*uIz0WfH4aVSp90IslkDqKQ.png" /></figure><h4>Why a Loop and Not a Single Session?</h4><p>Three reasons:</p><ol><li><strong>Context window limits:</strong> LLMs can only hold so much information at once. In a long session, early context gets pushed out. A fresh session starts clean, which can actually be better than a polluted long session.</li><li><strong>Error accumulation:</strong> In a long session, if the agent makes a wrong assumption early on, every subsequent decision is built on that wrong assumption. A fresh session forces re-evaluation from the ground truth (the actual code on disk).</li><li><strong>Resilience:</strong> If a session crashes, gets confused, or goes down a rabbit hole, the loop just starts a new one. The damage is limited to whatever was committed to git.</li></ol><h4>What This Means in Practice</h4><p>Whether you’re running parallel or sequential sessions, the loop concept applies:</p><ol><li>Start an agent session</li><li>The agent orients itself (reads PROGRESS.md, TASKS.md)</li><li>The agent picks a task, does the work, verifies it</li><li>The agent updates tracking files, commits, ends</li><li>A new session begins (or the same one continues on the next task)</li></ol><p>The critical insight: <strong>each session must leave the repo in a state where the NEXT session can immediately be productive.</strong> This is the “relay race” model — the baton handoff must be clean.</p><h3>2.2 Orientation — “Put Yourself in an LLM’s Shoes”</h3><h4>The Concept</h4><p>When a human joins a new team, they spend days or weeks onboarding. They read docs, ask questions, explore the codebase. A coding agent has to do this in seconds, every single session.</p><p>Carlini’s insight: <strong>design the information environment for the agent’s consumption patterns, not yours.</strong></p><p>This means:</p><ul><li><strong>Humans are good at:</strong> browsing, scrolling, visual scanning, remembering context from yesterday, asking colleagues, building mental models over time.</li><li><strong>Agents are good at:</strong> reading text files sequentially, following explicit instructions, pattern matching against examples, processing structured data.</li><li><strong>Agents are bad at:</strong> exploring unfamiliar codebases without guidance, knowing what they don’t know, estimating time, remembering anything from previous sessions.</li></ul><h4>What “Orientation” Actually Means in Practice</h4><p>When a fresh session starts, the agent needs to answer these questions in this order:</p><ol><li><strong>What is this project?</strong> (3 seconds — read the first paragraph of PROGRESS.md)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mP_YSFTJL_Sf3zU_vtFKVA.png" /></figure><ol><li><strong>What’s the current state?</strong> (10 seconds — read the component status table)</li><li><strong>What should I work on?</strong> (10 seconds — read the top task in TASKS.md)</li><li><strong>How do I do it?</strong> (20 seconds — read the relevant section of CONVENTIONS.md)</li><li><strong>How do I verify it worked?</strong> (10 seconds — know which test/check script to run)</li></ol><p>Total: under 60 seconds. Compare this to the agent spending 5–10 minutes exploring the codebase, reading random files, and trying to piece together what’s going on.</p><h4>The Files That Enable Orientation</h4><ul><li><strong>PROGRESS.md</strong> — Answers “where are we?” The most important file because it prevents the agent from re-discovering known problems, retrying failed approaches, or working on something that’s already done.</li><li><strong>ARCHITECTURE.md</strong> — Answers “how does everything connect?” Without this, the agent has to read dozens of config files to understand how services relate to each other.</li><li><strong>CONVENTIONS.md</strong> — Answers “how should I write code here?” Without this, the agent will invent its own patterns, which will be inconsistent with the existing codebase. Consistency matters because the NEXT session needs to read THIS session’s code.</li><li><strong>TASKS.md</strong> — Answers “what should I do right now?” Without this, the agent will either ask you (breaking autonomy) or guess (possibly wrong).</li></ul><h4>Why These Are Separate Files</h4><p>You might think: “Why not put everything in one file?”</p><p>Because of <strong>context window management.</strong> Most agent frameworks auto-load a project-level config file at session start. If that file is 5,000 lines long, that’s 5,000 lines of context consumed before the agent even starts thinking about the task. By splitting into focused files, the agent only reads what’s relevant:</p><ul><li>Working on application code? → Read CONVENTIONS.md</li><li>Working on infrastructure? → Read ARCHITECTURE.md</li><li>Starting any session? → Read PROGRESS.md + TASKS.md (always)</li></ul><p>This is Carlini’s “context window pollution” principle applied to orientation docs.</p><h3>2.3 Tests as Supervisors</h3><h4>The Concept</h4><p>This is Carlini’s most important principle. In his words: “The agent will work autonomously to solve whatever problem I give it. So it’s important that the task verifier is nearly perfect, otherwise the agent will solve the wrong problem.”</p><p>Let’s unpack what this means with an example.</p><p>Say you ask the agent to “implement the user registration endpoint.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*inHXPOZip3jo33-payk7bw.png" /></figure><p><strong>The tests define “correct.”</strong> Without them, “correct” is whatever the agent thinks it means, which may or may not match what you need.</p><h4>The Oracle Concept</h4><p>Carlini used GCC as an “oracle” — a known-correct reference implementation. When his compiler produced output, he could compare it against GCC’s output. If they differed, his compiler was wrong.</p><p>In typical projects, you have several oracles available:</p><ul><li><strong>API specifications (e.g., OpenAPI)</strong> → Oracle for the API. Every endpoint, every request/response schema, every status code is defined. The agent validates its API code against this spec.</li><li><strong>Compile-time type checking</strong> → Oracle for type safety. Strong type systems catch entire categories of bugs before the code runs.</li><li><strong>Linters and static analysis</strong> → Oracle for code quality. Tools like linters catch common mistakes, anti-patterns, and potential bugs.</li><li><strong>Dry-run modes</strong> → Oracle for infrastructure manifests. Before applying changes, you can ask the system “would this be valid?” without actually applying it.</li></ul><p>Each oracle gives the agent a way to verify its work without needing you to look at it. <strong>The more oracles you have, the more autonomous the agent can be.</strong></p><h4>Test-First vs. Test-After</h4><p>There’s a reason the approach emphasizes writing tests BEFORE feature work:</p><p><strong>Test-first</strong> (write tests, then implement):</p><ul><li>Tests define the expected behavior</li><li>The agent has a clear target to hit</li><li>You review the tests (small, readable) instead of the implementation (large, complex)</li><li>If the tests are right and they pass, the implementation is almost certainly right</li></ul><p><strong>Test-after</strong> (implement, then write tests):</p><ul><li>The agent writes tests that validate what it built, not what you wanted</li><li>Tests often just confirm the existing behavior, even if that behavior is wrong</li><li>This is “testing your assumptions, not your requirements”</li></ul><p>For autonomous agent work, <strong>test-first is dramatically more effective.</strong> You spend 20 minutes defining what the endpoint should do (via tests), then the agent spends 30 minutes making the tests pass. You review the tests, not the 500 lines of implementation code.</p><h3>2.4 Context Window Pollution</h3><h4>The Concept</h4><p>An agent’s context window is like a desk. It can hold a certain amount of paper. When the desk is full, adding a new piece of paper means an old one falls off the edge.</p><p>Every piece of text the agent sees — code, test output, error messages, file contents, your instructions — takes up space on this desk. If the desk is covered in useless junk (verbose logs, repeating test names, framework boilerplate), there’s no room for the things that matter (the actual error message, the test that’s failing, the code that needs to change).</p><h4>Carlini’s Rules for Context Hygiene</h4><p><strong>Rule 1: Minimize stdout noise.</strong> If a test suite has 500 tests and 498 pass, the agent doesn’t need to see “PASS” 498 times. It needs to see the 2 failures and a summary line.</p><p><strong>Rule 2: Make errors grep-friendly.</strong> If every error follows the format ERROR: &lt;context&gt; - &lt;reason&gt;, the agent can run grep ERROR and immediately see every problem. If errors are buried in multi-line stack traces with ANSI color codes, the agent wastes context parsing junk.</p><p><strong>Rule 3: Log verbose output to files.</strong> Full stack traces, debug output, timing information — these are valuable but only when needed. Write them to a log file. The agent can cat or grep the log file when it needs to investigate, but the default output stays clean.</p><p><strong>Rule 4: Pre-compute summaries.</strong> Don’t make the agent count things. Instead of printing 500 test results and expecting the agent to figure out the pass rate, print: PASSED: 498/500 (99.6%). The agent instantly knows the situation.</p><p><strong>Rule 5: Use fast-run flags everywhere.</strong> Full test suites take minutes. During active development, the agent should get feedback in seconds. A fast mode that runs a 10–20% sample gives quick signal. Full runs happen before commit.</p><p>For example, a naive health check that prints detailed status for each service might be 100+ lines. The agent reads all of it, most of it is “healthy, healthy, healthy.” That’s wasted context. Instead, the health check should print:</p><pre>PASS: PostgreSQL<br>PASS: Redis<br>FAIL: MinIO - ERROR: connection refused (DNS misconfiguration)<br>PASS: Message Queue<br>...<br>HEALTH: 18/20 services healthy</pre><p>The agent reads 20 lines and knows exactly where the problem is.</p><h3>2.5 Time Blindness</h3><h4>The Concept</h4><p>Humans have an intuitive sense of time. After 20 minutes of debugging, you feel the time passing and might decide to try a different approach. After an hour, you’re definitely aware that you’ve been stuck.</p><p>Agents have no sense of time whatsoever. They process tokens. Whether processing those tokens takes 1 second or 1 hour, the agent has no awareness of the difference. Left alone, an agent will happily:</p><ul><li>Run a full test suite that takes 30 minutes when a 10-second subset would give the same signal</li><li>Spend hours trying to fix a bug through trial and error when a human would have escalated after 15 minutes</li><li>Re-read the same large file multiple times without realizing it’s spending 5 minutes on I/O each time</li></ul><h4>How Carlini Addressed It</h4><ul><li><strong>Default fast mode:</strong> Tests run a small sample (Take note that this refers essentially to tests and functionality, unless you have state of the art security — dont expose any data to the actual model or agent) by default. Full suite requires explicit opt-in.</li><li><strong>Infrequent progress output:</strong> The test harness prints every 100th test, not every test. This serves double duty — it avoids context pollution AND prevents the agent from spending time processing progress updates.</li><li><strong>Time-boxed operations:</strong> Long-running commands have timeouts. If a check hasn’t returned in 30 seconds, it fails rather than hanging forever.</li></ul><h4>Practical Safeguards</h4><ul><li>Fast test scripts should run in under 10 seconds (unit tests only, no DB)</li><li>Every network call in health checks should have a reasonable timeout</li><li>Regression checks run gates in parallel where possible</li><li>If a build takes abnormally long, something is wrong — fail fast</li></ul><h3>2.6 Parallelism and Task Decomposition</h3><h4>The Concept</h4><p>Carlini ran 16 agents in parallel. Each needed independent work to do. If two agents work on the same thing, you don’t get 2x speed — you get merge conflicts and wasted effort.</p><p>The key insight: <strong>parallelism requires independence.</strong> Two tasks can run in parallel only if:</p><ul><li>They don’t modify the same files</li><li>They don’t depend on each other’s output</li><li>Their combined changes don’t conflict</li></ul><h4>How Carlini Achieved It</h4><ul><li><strong>Independent tests:</strong> When there are 500 failing tests, each agent picks a different test. Perfect parallelism.</li><li><strong>File-level oracle:</strong> When compiling the Linux kernel (one monolithic task), he used GCC to compile most files and his agent’s compiler for a random subset. Each agent got a different subset. Parallel debugging.</li><li><strong>Task locking:</strong> Agents claim tasks by writing lock files. Git’s conflict resolution prevents two agents from claiming the same task.</li></ul><h4>The Dependency Graph Concept</h4><p>Not all tasks are independent. Some form a dependency chain:</p><pre>Define types → Write migrations → Implement storage → Build API → Write tests</pre><p>You can’t write migrations before you know the types. You can’t implement storage before the tables exist. This is a <strong>critical path</strong> — the longest chain of sequential dependencies.</p><p>But within each level of the chain, there IS parallelism:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*Vg7YlqL9lmy35nLyqQLduA.png" /></figure><p>Even if you’re not running parallel agents, understanding the dependency graph tells you what order to do things in and where you could run two sessions on different tasks without conflicts.</p><p>A task tracking file (TASKS.md) should encode these dependencies explicitly so that the agent (and you) can make smart sequencing decisions.</p><h3>2.7 Specialization (Agent Roles)</h3><h4>The Concept</h4><p>Not every developer on a team does the same thing. Some write features, some write tests, some do code review, some maintain infrastructure. Specialization works because:</p><ul><li><strong>Focus:</strong> A developer who only writes tests will write better tests than one who’s also trying to ship features.</li><li><strong>Consistency:</strong> The same person/agent maintaining documentation will produce consistent documentation.</li><li><strong>Different priorities:</strong> A feature developer optimizes for “does it work?” A security reviewer optimizes for “is it safe?” These are different lenses on the same code.</li></ul><h4>Carlini’s Agent Roles</h4><ul><li><strong>Builders:</strong> Fix failing tests, implement features, write core functionality</li><li><strong>Deduplicator:</strong> Find and merge duplicate code (LLMs love to re-implement things)</li><li><strong>Performance agent:</strong> Optimize speed</li><li><strong>Code quality agent:</strong> Improve the generated code’s efficiency</li><li><strong>Architecture critic:</strong> Review from an experienced developer’s perspective</li><li><strong>Documentation agent:</strong> Keep docs current</li></ul><h4>Why This Works for LLMs Specifically</h4><p>LLMs have a particular failure mode: they optimize for the immediate goal at the expense of everything else. If you tell the agent “implement this feature,” it will implement the feature. It won’t spontaneously think “I should also update the docs” or “I should check if this duplicates existing code” or “I should consider security implications.”</p><p>By having separate agents with different goals, you ensure that EVERY concern gets attention:</p><ul><li>The builder implements the feature</li><li>The test-writer ensures it’s tested</li><li>The security-reviewer checks for vulnerabilities</li><li>The doc-keeper updates documentation</li><li>The deduplicator consolidates any duplicate code</li></ul><h3>2.8 The Oracle Pattern (Known-Good References)</h3><h4>The Concept in Depth</h4><p>An oracle is something that can tell you whether an answer is correct. The key property of an oracle is that <strong>checking is easier than producing.</strong></p><ul><li>It’s very hard to write a correct C compiler. It’s very easy to check whether compiled code produces the right output — just run it and compare to GCC’s output.</li><li>It’s very hard to implement a correct REST API. It’s very easy to check whether the API matches a spec — just compare the response to the schema.</li></ul><p>This asymmetry is what makes autonomous development possible. The agent does the hard work (producing). The oracle does the easy work (checking). As long as the oracle is trustworthy, the output is trustworthy.</p><h4>Types of Oracles</h4><ul><li><strong>Specification oracles:</strong> A document that defines what correct behavior looks like. An OpenAPI spec, a protocol buffer definition, a formal requirements document.</li><li><strong>Reference implementation oracles:</strong> Another system that does the same thing correctly. Carlini used GCC. You might use existing well-known platforms as reference for expected behavior.</li><li><strong>Property-based oracles:</strong> Instead of checking specific outputs, you check that properties hold. “Every successful POST should make a subsequent GET return that resource.” This doesn’t check exact values — it checks invariants.</li><li><strong>Compile-time oracles:</strong> The compiler itself is an oracle. If strongly-typed code compiles, it’s free of entire categories of bugs (null pointers, type mismatches, etc.). Compile-time query checking (like SQLx for Rust) is a powerful oracle — if queries compile, they’re valid against the real schema.</li><li><strong>Regression oracles:</strong> “The set of passing tests never shrinks.” If 47 tests passed yesterday and only 45 pass today, something broke. You don’t need to know what — you just know there’s a regression.</li></ul><h4>The Oracle Strength Spectrum</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-Ap5xaBKSv0ur6bgn2yWw.png" /></figure><h4>2.9 The Regression Gate</h4><h4>The Concept</h4><p>A regression is when something that used to work stops working. Regressions are the #1 risk of autonomous development. Carlini explicitly called this out: “The agent started to frequently break existing functionality each time it implemented a new feature.”</p><p>A regression gate is a checkpoint that blocks changes from being committed/pushed/deployed if they break existing functionality. It’s the “you must be this tall to ride” sign.</p><h4>How It Works</h4><p>Before any change is accepted:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n2bECQUAe0LXadtibwZHyw.png" /></figure><p>ALL gates must pass. If any one fails, the change is rejected. This is non-negotiable.</p><h4>Why Git Hooks</h4><p>The regression gate is implemented as a git pre-push hook. This means:</p><ul><li>The agent (or you) can make experimental changes freely</li><li>The agent can commit work-in-progress</li><li>But the moment it tries to PUSH (share with the world), the gates activate</li><li>If gates fail, the push is blocked</li></ul><p>This gives the agent freedom to experiment while preventing broken code from reaching the shared repository.</p><h4>The CI Pipeline as a Second Gate</h4><p>Git hooks run locally. CI runs on the server. The CI pipeline should run the SAME checks (and more — like full test suites instead of fast samples). This is defense in depth:</p><ul><li><strong>Git hook:</strong> fast check, catches obvious problems, runs before push</li><li><strong>CI pipeline:</strong> thorough check, catches subtle problems, runs before merge</li></ul><h3>2.10 Task Locking and Handoff</h3><h4>The Concept</h4><p>When multiple agents (or sessions) work on the same codebase, you need to prevent conflicts. Carlini used file-based locks in a current_tasks/ directory.</p><p>For sequential workflows, the problem is slightly different: how does session N+1 know what session N did?</p><h4>The Relay Race Model</h4><p>Think of your sessions as runners in a relay race. The baton is the state of the repository. A clean handoff means:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*-7e1OPYhDEi_wK0rwifyFQ.png" /></figure><p>Now session N+1 starts:</p><ul><li>Reads PROGRESS.md → knows the current state</li><li>Reads TASKS.md → sees the top READY task</li><li>Reads the session log if relevant → knows what was just done</li><li>Picks up the next task → no wasted time, no duplicate work</li></ul><h4>The Session Log</h4><p>A sessions/ directory stores a log for every agent session. Each log records:</p><ul><li>What task was worked on</li><li>What changes were made</li><li>What was learned (important! prevents future sessions from re-discovering things)</li><li>What commits were created</li><li>What should be done next</li></ul><p>This is the equivalent of a developer writing a handoff note before going on vacation. It takes 2 minutes to write and saves 30 minutes for the next person.</p><h3>PART 3: INFRASTRUCTURE CONCEPTS</h3><h3>3.1 GitOps and Why It Matters for Agents</h3><h4>The Concept</h4><p>GitOps is a way of managing infrastructure where git is the single source of truth. Instead of SSHing into a server and running commands, you:</p><ol><li>Define the desired state in git (deployment manifests, infrastructure configs)</li><li>An automated system watches git</li><li>When git changes, the system applies the changes to the environment</li><li>If someone manually changes the environment, the system detects the drift and reverts it</li></ol><h4>Why This Matters for Autonomous Agents</h4><p>GitOps is perfect for agent-driven development because:</p><ul><li><strong>Everything is a file:</strong> Agents work with files. GitOps means infrastructure changes are file changes. The agent can modify a deployment manifest, commit it, and the GitOps tool handles the deployment.</li><li><strong>Audit trail:</strong> Every change is a git commit. You can see exactly what the agent changed, when, and why (via the commit message).</li><li><strong>Rollback:</strong> If the agent breaks something, git revert undoes it. Compare this to SSH changes where there&#39;s no history.</li><li><strong>Validation:</strong> Before applying changes, the CI pipeline validates them. This is your regression gate for infrastructure.</li></ul><h4>The GitOps Flow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*ESvBKvKr4mNKQaL_H_pygQ.png" /></figure><h3>3.2 Base/Overlay Configuration Patterns</h3><h4>The Concept</h4><p>Tools like Kustomize let you define a “base” configuration and then “overlay” environment-specific changes on top.</p><pre>config/service/<br>├── base/<br>│   ├── kustomization.yaml    # Base configuration<br>│   ├── deployment.yaml       # Standard deployment<br>│   ├── service.yaml          # Network configuration<br>│   └── configmap.yaml        # Default settings<br>├── overlays/<br>│   ├── dev/<br>│   │   └── kustomization.yaml   # Dev overrides (1 replica, no TLS)<br>│   ├── staging/<br>│   │   └── kustomization.yaml   # Staging overrides (3 replicas, TLS)<br>│   └── production/<br>│       └── kustomization.yaml   # Prod overrides (5 replicas, TLS, ACLs)</pre><h4>Why This Matters for Agents</h4><p>When the agent needs to change a configuration, it needs to know:</p><ul><li>Base changes affect ALL environments</li><li>Overlay changes affect only one environment</li></ul><p>Without understanding this pattern, the agent might modify the production overlay when it should modify the base. ARCHITECTURE.md should document which configs are bases and which are overlays.</p><h3>3.3 Secrets Management</h3><h4>The Concept</h4><p>Secrets (passwords, API keys, certificates) stored in plain text in git are a security risk. Encrypted secrets solutions (like Sealed Secrets, SOPS, or Vault) solve this by encrypting secrets so they’re safe to commit to version control.</p><pre>Your password: &quot;hunter2&quot;<br>    ↓ (encrypt with public key)<br>Encrypted secret file (safe to commit to git)<br>    ↓ (commit + push)<br>GitOps tool applies encrypted secret<br>    ↓ (controller decrypts with private key)<br>Usable secret (available to applications)</pre><h4>Why Agents Must NOT Touch These</h4><p>Encrypted secret values are opaque to agents. The agent can’t read them, can’t generate them, and shouldn’t try to modify them. This should be documented in your project files because:</p><ul><li>The agent can’t decrypt them (no access to private key)</li><li>If the agent modifies the encrypted blob, it becomes garbage</li><li>Secret management requires human oversight</li></ul><h3>3.4 Modular Code Architecture</h3><h4>The Concept</h4><p>A well-structured project separates concerns into distinct modules or packages. This is a layered architecture where dependencies flow in one direction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*wERoy3ly7fILtah9YUQWFA.png" /></figure><h4>Why This Matters for Agents</h4><p>This structure enables parallel work:</p><ul><li>Agent A modifies core → adds a new type</li><li>Agent B modifies storage → adds a new repository (using existing types)</li><li>These don’t conflict as long as Agent B uses types that already exist</li></ul><p>BUT: if Agent A adds a new type and Agent B needs that new type, there’s a dependency. B must wait for A. Your task tracking should encode these dependencies.</p><p>The modular structure also means running all tests at once catches cross-module breakage. If the agent changes the core module and it breaks the API module, full-project tests catch it immediately.</p><h3>3.5 Compile-Time Verification</h3><h4>The Concept</h4><p>Most languages treat database queries, API contracts, and external interactions as strings or runtime checks. You write a query as a string, send it, and if it’s wrong, you get a runtime error — possibly in production, possibly at 3 AM.</p><p>Compile-time verification tools check these interactions when you build the code. For example, tools like SQLx (Rust) or jOOQ (Java) can verify that SQL queries match the actual database schema at compile time.</p><h4>Why This Is Critical for Autonomous Development</h4><p>Compile-time verification is one of the strongest oracles available. If the agent writes a query that references a non-existent column, the code won’t compile. The agent can’t commit broken queries. The compiler prevents it.</p><h4>The Migration Connection</h4><p>For compile-time query checking to work, there must be a database with the correct schema. This database is set up by running migrations. So:</p><pre>Write migration → Run migration → Database has correct schema →<br>Compiler can verify queries → Compilation succeeds</pre><p>If you add a new table in a migration but forget to run the migration, the compiler can’t verify queries against that table and compilation fails. This is actually a GOOD thing — it forces you to keep migrations and code in sync.</p><h3>PART 4: THE VERIFICATION PHILOSOPHY</h3><h3>4.1 Defense in Depth</h3><h4>The Concept</h4><p>Defense in depth means having multiple independent layers of protection. If one layer fails, the next catches the problem. Military analogy: a castle has a moat, then a wall, then archers, then a keep. Each layer makes the attacker’s job harder.</p><h4>Your Defense Layers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qBIavtmkd88ydWyLjFXIA.png" /></figure><p>A bug has to get through ALL eleven layers to reach production and cause harm. Each layer the agent’s code must pass through makes the output more trustworthy.</p><h3>4.2 The Testing Pyramid</h3><h4>The Concept</h4><p>The testing pyramid says you should have many fast, small tests and fewer slow, large tests:</p><pre>      / E2E  \           Few, slow, expensive<br>     / Integr.\          Some, medium speed<br>    /Unit Tests\         Many, fast, cheap</pre><h4>Applied to Agent Development</h4><p><strong>Unit tests (many, fast):</strong> Test individual functions in isolation. These run in milliseconds, need no external services, and catch logic errors.</p><p><strong>Integration tests (some, medium):</strong> Test components working together. These need a database or other services, take seconds, and catch interaction bugs.</p><p><strong>End-to-end tests (few, slow):</strong> Test the entire system. These take minutes, need the full environment, and catch system-level bugs.</p><p>For autonomous development, the agent should run unit tests constantly (after every change), integration tests before every commit, and E2E tests before every push. This matches the fast / medium / full tier system.</p><h3>4.3 Deterministic vs. Non-Deterministic Tests</h3><h4>The Concept</h4><p>A deterministic test gives the same result every time. A non-deterministic (flaky) test sometimes passes, sometimes fails, without any code change.</p><p><strong>Flaky tests are POISON for autonomous development.</strong> Here’s why:</p><ol><li>Agent runs tests, one fails</li><li>Agent tries to fix it</li><li>Runs tests again, it passes (it was flaky)</li><li>Agent thinks it fixed the problem</li><li>Agent commits a “fix” that changed nothing meaningful</li><li>Wasted time, wasted context, potentially introduced a new bug</li></ol><h4>How to Prevent Flaky Tests</h4><ul><li>No sleep() in tests — use explicit wait conditions</li><li>No reliance on system time — inject fake clocks</li><li>No shared test state — each test gets its own database/context</li><li>No ordering dependencies — tests run in any order</li><li>Deterministic random seeds — if randomness is needed, fix the seed</li></ul><h3>PART 5: THE ECONOMICS AND STRATEGY</h3><h3>5.1 The Cost Equation</h3><p>Carlini spent $20,000 on 2 billion input tokens and 140 million output tokens. That’s for 16 parallel agents over two weeks. Your economics will differ based on your setup, but the principles are the same:</p><ul><li><strong>Cost of a wasted session:</strong> If the agent spends a session re-discovering a known issue, working on the wrong task, or producing code that breaks existing functionality, that’s 30–60 minutes of your time wasted (reviewing, reverting, re-doing).</li><li><strong>Cost of a bug in production:</strong> For any serious project, a production bug costs more in recovery time and reputation than any tooling investment.</li><li><strong>The scaffolding investment:</strong> Setting up the orientation files, regression gates, and verification scripts might take 2–3 hours. But it prevents wasted sessions for the entire life of the project. If it saves 15 minutes per session and you run 200 sessions, that’s 50 hours saved.</li></ul><h3>5.2 Solo Developer + Agent Team = Force Multiplier</h3><h4>What YOU Should Do (Humans Are Better At)</h4><ul><li>Architectural decisions (“should we add this feature?”)</li><li>Security decisions (“is this configuration safe?”)</li><li>Reviewing test definitions (“are we testing the right thing?”)</li><li>Customer interactions (“what does the market need?”)</li><li>Strategic prioritization (“what do we build first?”)</li></ul><h4>What AGENTS Should Do (LLMs Are Good Enough For)</h4><ul><li>Implementing well-defined features with clear tests</li><li>Writing boilerplate (CRUD endpoints, database repositories)</li><li>Refactoring and deduplication</li><li>Documentation maintenance</li><li>Infrastructure manifest generation</li><li>Running repetitive verification tasks</li></ul><h4>The Optimal Workflow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*BQ3WeXEKJ3TmXqoUUhdXuQ.png" /></figure><p>Total: 70 minutes for a feature that might take 4+ hours of solo coding. The leverage comes from the fact that <strong>defining “what to build” (tests, specs) takes much less time than actually building it.</strong></p><h3>5.3 When NOT to Use Agents</h3><p>Agents are not always the right tool:</p><ul><li><strong>Don’t use agents for architectural decisions:</strong> If you’re deciding between a message queue vs. direct HTTP calls, that’s a thinking problem, not a coding problem. Talk to the LLM conversationally, not through an agent.</li><li><strong>Don’t use agents for one-off investigations:</strong> If you’re debugging a production issue, you want interactive exploration, not autonomous execution. Use the coding tool interactively — ask it to look at logs, explore the codebase, form hypotheses.</li><li><strong>Don’t use agents for security-critical code on the first pass:</strong> Authentication middleware, permission checks, data isolation logic — these should be written carefully with human review at every step. Use agents to WRITE these, but review every line before merging.</li><li><strong>Don’t use agents before the scaffolding is in place:</strong> Running an agent without tests, without PROGRESS.md, without CONVENTIONS.md is like hiring a contractor without a blueprint. They’ll build something, but it probably won’t be what you wanted.</li></ul><h3>PART 6: CONNECTING EVERYTHING TOGETHER</h3><h3>6.1 The Complete Session Lifecycle</h3><p>Here’s how every concept connects in a single session:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dTI4oLiRbOBXbcM1ChUdAg.png" /></figure><h3>6.2 Why Each Piece Exists</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xDOxh6LxeJHiP5GUCXRiA.png" /></figure><h3>6.3 The Feedback Loops</h3><p>There are three nested feedback loops in this system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*yJXxcJAeea6BtjjPEEz0gw.png" /></figure><h3>PART 7: THE MENTAL MODELS</h3><h3>7.1 Think of the Agent as a Brilliant but Amnesiac Contractor</h3><p>Coding agents are extremely capable — they can write code, manage infrastructure, design APIs, debug complex issues. But they have amnesia. Every session they wake up and don’t remember yesterday.</p><p>Your job is to build the world’s best onboarding packet. So good that a brilliant developer with amnesia can read it in 60 seconds and be productive immediately.</p><h3>7.2 Think of Tests as Blueprints, Not Inspections</h3><p>Most people think of tests as quality checks that happen after the work. For autonomous agents, flip this: <strong>tests are BLUEPRINTS that happen before the work.</strong></p><p>Write the test first. The test says “this is what I want.” Then the agent implements it. The test isn’t checking the agent’s work — it’s directing it.</p><h3>7.3 Think of the Repo as the Brain</h3><p>Since agents have no persistent memory, the repository IS the memory. Everything that matters — current state, conventions, failed approaches, task priorities — must be in files that the agent can read.</p><p>If it’s not in the repo, it doesn’t exist for the agent. No verbal agreements, no “remember when we discussed,” no assumed context. <strong>Files or it didn’t happen.</strong></p><h3>7.4 Think of Automation as Trust</h3><p>Each piece of automation you build increases how much you can trust autonomous output:</p><pre>NO automation:       Trust nothing, review everything<br>Type safety:         Trust that it won&#39;t crash from type errors<br>Schema checking:     Trust that queries are valid<br>Unit tests:          Trust that individual functions work<br>Integration tests:   Trust that components work together<br>CI pipeline:         Trust that main branch is always healthy<br>Full stack:          Trust enough to deploy without reading every line</pre><p>You’re building toward that last level. Each piece of scaffolding moves you closer.</p><h3>SUMMARY: THE 10 COMMANDMENTS OF AGENT DEVELOPMENT</h3><ol><li><strong>Every session starts from zero.</strong> Build orientation materials so good that zero is fine.</li><li><strong>Tests define correct.</strong> Without tests, “correct” is whatever the agent guesses. With tests, “correct” is whatever you specified.</li><li><strong>Keep the context clean.</strong> Minimal stdout, grep-friendly errors, verbose logs in files.</li><li><strong>Oracles are your safety net.</strong> API specs, compile-time checks, linters, health checks — each one prevents a category of bugs automatically.</li><li><strong>Gate the regressions.</strong> Never let broken code reach the repository. Pre-push hooks and CI pipelines are non-negotiable.</li><li><strong>Decompose by independence.</strong> Tasks that touch different files can run in parallel. Tasks that share dependencies must be sequenced. Map this explicitly.</li><li><strong>Specialize your agents.</strong> Builders build, testers test, reviewers review. Don’t ask one agent to do everything.</li><li><strong>Compensate for time blindness.</strong> Fast test modes, timeouts, and progress summaries keep the agent from wasting hours on minutes-long tasks.</li><li><strong>Design for handoff.</strong> Every session must leave the repo ready for the next session. Progress files, session logs, and task tracking are the baton.</li><li><strong>Automate trust.</strong> Every layer of verification you add is a layer of review you can skip. Build toward autonomous deployment, one oracle at a time.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef9b2342525a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/the-complete-guide-to-autonomous-agent-development-every-concept-every-principle-from-first-ef9b2342525a">The Complete Guide to Autonomous Agent Development: Every Concept, Every Principle, From First…</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Microsoft Fabric Actually Stores Your Data: A Production Reality Check]]></title>
            <link>https://medium.com/data-unlocked/how-microsoft-fabric-actually-stores-your-data-a-production-reality-check-c409a574173e?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c409a574173e</guid>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 18:14:37 GMT</pubDate>
            <atom:updated>2025-09-28T18:20:05.840Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 1 of the series exploring what really matters when building data platforms</em></p><p>Picture this: You’re staring at a dashboard where the same query that ran in 3 seconds yesterday is taking 45 seconds today. No code changes. No new data. Just… slower.</p><p>This scenario plays out in data teams everywhere, and it always leads to the same rabbit hole of discovery about how modern storage actually works. Here’s what that journey looks like, and why understanding these patterns matters for what we’re all building.</p><h3>The Column Storage Revelation That Started Everything</h3><p>Most of us know about “columnar storage” from the documentation, but the real understanding comes when you see the dramatic differences it creates in practice.</p><p>Consider two identical tables with the same data and 10 million rows. One takes 47GB of storage. The other takes 8GB.</p><p>This isn’t about compression settings or data types. It’s about something more fundamental that explains many performance mysteries we encounter.</p><p>The breakthrough comes when you realize that modern analytical systems don’t store rows together like a spreadsheet. They split everything by columns, which completely changes the performance characteristics:</p><p>Here’s the fundamental difference:</p><pre>How I Thought Data Was Stored (Row Storage):<br>Record 1: [ID: 1, Name: &quot;Alice&quot;, Age: 30, Salary: 70000]<br>Record 2: [ID: 2, Name: &quot;Bob&quot;, Age: 25, Salary: 60000]<br>Record 3: [ID: 3, Name: &quot;Carol&quot;, Age: 35, Salary: 80000]</pre><pre>What&#39;s Actually Happening (Column Storage):<br>IDs:      [1, 2, 3]<br>Names:    [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Carol&quot;]<br>Ages:     [30, 25, 35]<br>Salaries: [70000, 60000, 80000]</pre><p>This completely changes how queries work. When calculating average salary in a row-based system, you read every complete employee record, extract just the salary, then discard everything else. With 10 million employees, that means reading gigabytes of data to get one column of numbers.</p><p>In column storage, you read the Salaries column directly. One clean chunk of 10 million numbers. No waste.</p><p>This pattern explains many of those mysterious performance differences. Queries that touch few columns fly on columnar systems. Queries that need complete records can be slower than traditional row storage.</p><h3>The Hidden Transaction Log That Controls Everything</h3><p>Understanding columnar storage is just the beginning. Every Delta table has a hidden control system that most people never examine: the _delta_log folder.</p><p>This folder contains the complete history of everything that’s ever happened to your table. It’s not just metadata — it’s the brain that makes ACID transactions, time travel, and concurrent writes possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e853C7QOel-wpNgQ.png" /><figcaption>Image from <a href="https://vutr.substack.com/">https://vutr.substack.com/</a></figcaption></figure><p>The JSON files contained records of every single thing that had ever happened to the table:</p><pre>// The very first file - table creation<br>{<br>  &quot;add&quot;: {<br>    &quot;path&quot;: &quot;part-00000-tid-1234.parquet&quot;,<br>    &quot;size&quot;: 134217728,<br>    &quot;modificationTime&quot;: 1704067200000,<br>    &quot;stats&quot;: &quot;{\&quot;numRecords\&quot;:1000000,\&quot;minValues\&quot;:{\&quot;date\&quot;:\&quot;2024-01-01\&quot;},\&quot;maxValues\&quot;:{\&quot;date\&quot;:\&quot;2024-01-31\&quot;}}&quot;<br>  }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*Gq90aImGfFMcGwn1.png" /><figcaption>Image from <a href="https://vutr.substack.com/">https://vutr.substack.com/</a></figcaption></figure><pre><br>// A later file - someone deleted data<br>{<br>  &quot;remove&quot;: {<br>    &quot;path&quot;: &quot;part-00000-tid-1234.parquet&quot;,<br>    &quot;deletionTimestamp&quot;: 1704068000000<br>  }<br>}<br>{<br>  &quot;add&quot;: {<br>    &quot;path&quot;: &quot;part-00002-tid-9101.parquet&quot;,<br>    &quot;size&quot;: 134219000<br>  }<br>}</pre><p>This was like finding the diary of our database. Every addition, every deletion, every update — all recorded in chronological order. But the really clever part was in those stats. When I ran a query like WHERE date = &#39;2024-06-15&#39;, the system would read these statistics first. If a file&#39;s maximum date was 2024-01-31, it would skip the entire file without even opening it.</p><p>This explained another mystery: why some of our date-range queries were lightning fast while others crawled. The fast ones were hitting files with good statistics. The slow ones were opening files that shouldn’t have contained relevant data.</p><h3>The Checkpoint Discovery That Changes Query Performance</h3><p>Here’s a pattern that puzzles many teams: queries that suddenly become much faster for no apparent reason. A report that takes 2 minutes one day takes 30 seconds the next, with no changes to code or data.</p><p>The explanation lies in those checkpoint files in the transaction logs.</p><p>Here’s what’s actually happening: after every 10 commits to a table, Delta Lake creates a checkpoint — a snapshot of the complete table state at that moment. These aren’t just backup files — they’re performance optimizations.</p><p>Without checkpoints, reading a table that’s had 1000 changes means reading 1000 individual JSON files sequentially and reconstructing the current state in memory. That could take several minutes for a heavily-modified table.</p><pre>The Old Way (what I thought was happening):<br>1. Read JSON file 1: &quot;Add file A&quot;<br>2. Read JSON file 2: &quot;Remove file A, add file B&quot;  <br>3. Read JSON file 3: &quot;Add file C&quot;<br>... repeat 997 more times<br>4. Finally know what files actually contain current data<br>Time: Minutes</pre><pre>The Checkpoint Way (what actually happens):<br>1. Read _last_checkpoint: &quot;Current state is at commit 1000&quot;<br>2. Read checkpoint file: Complete table state instantly<br>3. Read any JSON files after 1000 (usually none)<br>Time: Seconds</pre><p>Those mysterious performance improvements aren’t random at all. They happen exactly when tables cross checkpoint boundaries. Understanding this pattern helps predict when queries will improve and allows for strategic checkpoint triggering during maintenance windows.</p><h3>The Update Strategy That Can Break Your System</h3><p>Here’s a scenario that catches many teams off guard: simple customer address updates — changing just a few fields for a handful of customers — suddenly take 45 minutes and consume hundreds of gigabytes of temporary storage.</p><p>The culprit is often Copy-on-Write behavior that teams don’t expect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g4S4OaAT6bNpmm_6.png" /><figcaption>Image from <a href="https://vutr.substack.com/">https://vutr.substack.com/</a></figcaption></figure><p>When you update a single customer record, here’s what actually happens behind the scenes:</p><pre>Intent: Change John Smith&#39;s address</pre><pre>What You Might Expect:<br>1. Find John&#39;s record<br>2. Update his address  <br>3. Save the change<br>4. Done</pre><pre>What Was Actually Happening (Copy-on-Write):<br>1. Read the entire 50GB customer file into memory<br>2. Find John&#39;s record among 80 million customers<br>3. Change his address<br>4. Write a completely new 50GB file with the updated data<br>5. Mark the old 50GB file as deleted<br>6. Update transaction log</pre><pre>Cost: 100GB of I/O to change one address<br>Time: 45 minutes<br>Storage: 100GB temporarily (both files exist during transition)</pre><p>No wonder updates can crush systems. You’re unknowingly rewriting gigabytes of data to change a few bytes.</p><p>The solution is Merge-on-Read, a completely different approach:</p><pre>The Same Update with Merge-on-Read:<br>1. Write a tiny &quot;deletion vector&quot;: &quot;Customer ID 5,234,567 is outdated&quot;<br>2. Write a small new file with just John&#39;s updated record  <br>3. Update transaction log to reference both<br>4. Readers automatically merge the view at query time</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rDh3fYDjaeJY9Cwu.png" /><figcaption>Image from <a href="https://vutr.substack.com/">https://vutr.substack.com/</a></figcaption></figure><pre>Cost: A few kilobytes of I/O<br>Time: Seconds<br>Storage impact: Minimal</pre><p>Switching to Merge-on-Read can transform update performance from 45 minutes to 30 seconds. But there’s a trade-off: reads become slightly more complex. Instead of reading one clean file, queries now check deletion vectors and merge multiple small files with the main data.</p><h3>The Small Files Disaster Nobody Warns You About</h3><p>Here’s a common scenario: your streaming pipeline runs smoothly for months, writing new data every 30 seconds. Updates are fast. Queries are responsive. Then suddenly, everything slows to a crawl.</p><p>The data volume is the same. Query patterns are identical. But every single query takes 10x longer than before.</p><p>The problem is often hiding in plain sight, in your storage directory:</p><pre>What You Expect to See:<br>/sales-data/<br>  ├── file_001.parquet (128MB)<br>  ├── file_002.parquet (128MB)<br>  ├── file_003.parquet (128MB)<br>  └── ... maybe 100 more files</pre><pre>What You Actually Find:<br>/sales-data/<br>  ├── file_001.parquet (500KB)<br>  ├── file_002.parquet (450KB)<br>  ├── file_003.parquet (520KB)<br>  ├── file_004.parquet (480KB)<br>  └── ... 47,000 more tiny files</pre><p>Writing data every 30 seconds for two months creates nearly 50,000 tiny files. Every query has to check metadata for every single one, even if it only needs data from a few.</p><p>The overhead becomes crushing:</p><ol><li>Open file handle: 50,000 times</li><li>Read compression headers: 50,000 times</li><li>Initialize decompression: 50,000 times</li><li>Read actual data: Finally</li></ol><p>That’s where OPTIMIZE comes in — Delta Lake’s file compaction command:</p><pre>Before OPTIMIZE:<br>47,000 files × 500KB average = 23GB across thousands of tiny files<br>Query time: 3 minutes (mostly overhead)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C-piRibuUx6diQG1.png" /><figcaption>Image from <a href="https://vutr.substack.com/">https://vutr.substack.com/</a></figcaption></figure><pre>After OPTIMIZE:  <br>180 files × 128MB each = 23GB in properly-sized chunks<br>Query time: 8 seconds (actual data processing)</pre><p>The same data, the same queries, but organized efficiently. This makes scheduling OPTIMIZE operations essential, not optional.</p><h3>The Concurrency Pattern That Prevents Data Corruption</h3><p>Here’s what seems impossible: multiple data pipelines writing simultaneously to the same tables, with real-time updates and constant analytics queries, yet nothing corrupts. No locks, no blocking, no lost data.</p><p>This works through optimistic concurrency control. Here’s how it plays out in practice:</p><pre>11:30:15 AM - Writer A starts updating inventory<br>11:30:15 AM - Writer B starts updating inventory  <br>(Both read current version: 847)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tgTA8JUO80fdxQ6d.png" /><figcaption>Image from <a href="https://vutr.substack.com/">https://vutr.substack.com/</a></figcaption></figure><pre>11:30:22 AM - Writer A finishes, tries to commit<br>- Attempts to create: 00000000000000000848.json<br>- Atomic operation succeeds<br>- Version 848 is now official</pre><pre>11:30:23 AM - Writer B finishes, tries to commit  <br>- Attempts to create: 00000000000000000848.json<br>- Atomic operation FAILS (file already exists)<br>- System says: &quot;Someone beat you to version 848&quot;</pre><pre>11:30:24 AM - Writer B automatically retries<br>- Reads current state (now version 848)<br>- Checks if its changes conflict with version 848<br>- No conflicts found<br>- Creates: 00000000000000000849.json<br>- Success!</pre><p>Both writes succeeded, neither corrupted the other, and it all happened automatically. The atomic file creation operation — available on every major cloud storage system — was the key to the entire ACID guarantee.</p><h3>What Every Team Should Know From Day One</h3><p>Here’s what would save weeks of debugging for most data teams:</p><p><strong>File sizes control everything.</strong> Not just performance — whether your system works at all. Too small, and metadata overhead kills you. Too large, and memory usage explodes. The sweet spot: 100–200MB files.</p><p><strong>Choose your update strategy before you need it.</strong> Copy-on-write for tables that change completely. Merge-on-read for selective updates. Switching later requires rewriting everything.</p><p><strong>OPTIMIZE and VACUUM aren’t features — they’re requirements.</strong> Schedule them from day one. Monitor them like you monitor your application. Alert if they fail.</p><p><strong>Streaming needs batching, even with exactly-once guarantees.</strong> Batching for at least 5 minutes prevents the small files problem. The exactly-once semantics work through the transaction log — each streaming job gets an ID, and the log tracks exactly which offsets have been processed.</p><p><strong>Those sudden performance improvements aren’t magic — they’re checkpoints.</strong> You can predict them, trigger them manually, and use them to plan maintenance windows.</p><h3>What’s Next</h3><p>Understanding these storage patterns is critical — it’s the difference between a system that works and one that fails under load. But these foundational Delta Lake concepts are just the beginning when working with Microsoft Fabric.</p><p>Later this week, we’ll delve deeper into the application of these patterns in Direct Lake, OneLake and V-Order. We’ll explore how the file size recommendations change when Direct Lake’s memory limits come into play, why perfectly optimized Delta tables sometimes still struggle in Power BI, and how V-Order optimization creates different trade-offs than traditional Z-ORDER clustering.</p><p>What storage patterns have you encountered in your Fabric implementations? How have Direct Lake memory limits affected your table design decisions?</p><p><em>This is Part 1 of a series sharing real production experiences with data platforms. Catch up on Part 2 later in the week.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c409a574173e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/how-microsoft-fabric-actually-stores-your-data-a-production-reality-check-c409a574173e">How Microsoft Fabric Actually Stores Your Data: A Production Reality Check</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Governor’s Decree: A Practitioner’s Guide to Implementing Zero Trust]]></title>
            <link>https://medium.com/data-unlocked/the-governors-decree-a-practitioner-s-guide-to-implementing-zero-trust-d5b7d0c8456b?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5b7d0c8456b</guid>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Fri, 18 Jul 2025 10:02:43 GMT</pubDate>
            <atom:updated>2025-07-18T10:02:43.042Z</atom:updated>
            <content:encoded><![CDATA[<p>In our journey architecting the Contoso Platform, we have laid down two powerful blueprints. First, in <strong>“From Digital Wild West to Cloud Metropolis,”</strong> we designed our network — a planned city with secure traffic flows. Next, with our <strong>“Data Refinery,”</strong> we established the Medallion Architecture — a systematic process for creating trusted, high-quality data. We have the physical plans for a state-of-the-art digital nation.</p><p>But even the most well-designed city is vulnerable without laws, a system of identification, and a vigilant civil guard. An architecture alone is static; it needs a dynamic, operational philosophy to bring it to life and keep it secure. For any modern data platform, that governing philosophy is <strong>Zero Trust</strong>.</p><p>This is not a product you can buy or a feature you can simply enable. It is a fundamental shift in security strategy, a mindset that permeates every architectural choice we make. This guide will walk you through the core tenets of Zero Trust, not as abstract theory, but as the living, breathing legal system that governs every interaction within our platform, specifically focusing on how we protect our most valuable assets: the data in our Storage Lake and the secrets in our Key Vault.</p><h3>The End of the Fortress: Why the Old World Failed</h3><p>For decades, the foundation of cybersecurity was the “castle-and-moat” model. We built an imposing digital fortress with a powerful perimeter firewall and assumed that any user or service operating “inside” the walls was trustworthy. This model is now fundamentally broken.</p><p>The modern enterprise has no perimeter. Our data is in the cloud, our users are working from home on personal devices, and our applications are interconnected via APIs. The “moat” has evaporated. Attackers no longer need to storm the castle walls; they can steal the credentials of a trusted user through a phishing email and walk right through the front gate. Once inside, the castle-and-moat model offers little resistance, allowing them to move laterally through “trusted” internal networks to find the crown jewels.</p><p>Zero Trust begins by accepting this new reality. It operates on a single, powerful assumption: <strong>the network is always hostile, and a breach is not a matter of <em>if</em>, but <em>when</em>.</strong> This forces us to discard the outdated concept of a trusted internal network and build a system where trust is never granted implicitly, but is earned and explicitly verified for every single request.</p><p>This strategy is built upon three core principles:</p><ol><li><strong>Verify Explicitly:</strong> We must always authenticate and authorize based on all available data points. This means we challenge every request to prove its identity and its right to be there, looking at signals like user identity, device health, geographic location, and the data being accessed.</li><li><strong>Use Least Privilege Access:</strong> We grant users and services only the absolute minimum permissions they need to perform their role. This access should be “just-enough-access” (JEA) and “just-in-time” (JIT), ensuring permissions are granted for the shortest possible duration, minimizing the window of opportunity for an attacker.</li><li><strong>Assume Breach:</strong> We design our entire system with the expectation that individual components will be compromised. This forces us to build for resilience by segmenting networks to limit the “blast radius,” encrypting all traffic, and ensuring we have end-to-end visibility to detect threats.</li></ol><h3>Zero Trust in Action: Protecting the Vault and its Keys</h3><p>For the Contoso Platform, these principles are not just ideals; they are concrete engineering choices embedded in how we designed our Azure Storage and Key Vault infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KFMPdnZXd4SzZsgi.jpeg" /></figure><h4>Securing the Vault: Our Data Lake</h4><p>Our data, stored in <strong>Azure Data Lake Storage (ADLS Gen2)</strong>, is the treasure. We architect its defense by applying Zero Trust at every layer. We begin by <strong>assuming breach</strong> and making our data lake “disappear” from the public internet. By setting its default network rule to Deny, we eliminate the most common attack vector. The only way to reach the data is via a <strong>Private Endpoint</strong>—a secure, private tunnel that exists only within our &quot;Cloud Metropolis&quot; network. This network segmentation ensures that an attacker on the internet cannot even discover that our storage account exists, let alone attack it.</p><p>Then, we <strong>verify explicitly</strong> at this private doorway. An application cannot access the storage just because it’s on the right network. It must present a verified identity credential (a Managed Identity or Service Principal token) to the storage endpoint. This is the first of many identity checks.</p><p>Finally, we enforce <strong>least privilege access</strong> with surgical precision using Role-Based Access Control (RBAC). A BI report’s service identity may be granted the Storage Blob Data Reader role on the Gold data container, but it has no permissions whatsoever on the Silver or Bronze containers. This is reinforced by our <strong>Domain-Driven Design</strong>, where the Finance container has a completely separate set of permissions from the Marketing container. We ensure no single identity has the keys to the entire kingdom.</p><h4>Securing the Keys to the Kingdom: Our Key Vault</h4><p>If the data is the treasure, then the secrets — the connection strings, passwords, and API keys — are the keys to the vault. Storing these in code or configuration files is a cardinal sin of security. We manage them in our digital fortress, <strong>Azure Key Vault</strong>, which is itself a masterpiece of Zero Trust design.</p><p>We again <strong>assume breach</strong> by making the Key Vault itself fully private, locking it down with its own <strong>Private Endpoint</strong>. The keys to the kingdom are not left hanging by the public-facing front door; they are kept in a separate, fortified room within our already secure facility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*-5Jgh5tnezdoHJb4.png" /></figure><p>We <strong>verify explicitly</strong> by using modern RBAC for authorization. When an application requests a secret, Key Vault doesn’t just check a simple access key; it validates the application’s entire identity against Azure AD and ensures it has been assigned a specific role.</p><p>This allows us to enforce <strong>least privilege access</strong> on the secrets themselves. A data pipeline’s identity may be granted the “Key Vault Secrets User” role, allowing it to <em>get</em> secrets, but not the power to list all secrets, modify them, or delete them. An administrator’s identity has more privilege, but that access is heavily audited. This granular control ensures that even if an application is compromised, its blast radius is limited; it cannot be used to destroy the very secrets it has permission to read.</p><h3>The Secure Handshake: A Zero Trust Journey</h3><p>Let’s trace a single request to see this symphony of verification in action. A data processing job in Databricks needs to read from a Silver table and write to a Gold table.</p><ol><li><strong>Identity Verification:</strong> The job starts, and its Managed Identity is authenticated with Azure Active Directory, which issues a short-lived access token. <em>This is the first verification.</em></li><li><strong>Secret Request:</strong> The job needs the connection string for a source system. It makes a call to the <strong>Key Vault’s Private Endpoint</strong>. The request travels securely within our Hub-and-Spoke network, invisible to the outside world.</li><li><strong>Secret Authorization:</strong> Azure Key Vault receives the request and the access token. It validates the token and checks its RBAC policies. “Does this Databricks job’s identity have the ‘Key Vault Secrets User’ role?” If yes, it returns the requested secret over the secure channel. <em>This is the second verification.</em></li><li><strong>Data Access Request:</strong> Now armed with the necessary credentials, the job connects to the <strong>ADLS Private Endpoint</strong>.</li><li><strong>Data Authorization:</strong> The Azure Storage service receives the request. It validates the identity and checks its RBAC assignments. “Does this identity have read access on the Silver/Finance container and write access on the Gold/Finance container?&quot; <em>This is the third, and most granular, verification.</em></li></ol><p>Only after passing through this multi-stage gauntlet of checks is the job allowed to proceed. At every step, trust was challenged, identity was proven, and permissions were constrained. This is the power of a true Zero Trust architecture — a holistic strategy, woven into the fabric of our network, storage, and secret management, that ensures the Contoso Platform is secure by design, not by chance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5b7d0c8456b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/the-governors-decree-a-practitioner-s-guide-to-implementing-zero-trust-d5b7d0c8456b">The Governor’s Decree: A Practitioner’s Guide to Implementing Zero Trust</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Medallion Architecture: A Methodology to Prevent Your Lakehouse from Becoming a Data Swamp]]></title>
            <link>https://medium.com/data-unlocked/the-medallion-architecture-a-methodology-to-prevent-your-lakehouse-from-becoming-a-data-swamp-28974efb4b97?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/28974efb4b97</guid>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 12:37:19 GMT</pubDate>
            <atom:updated>2025-07-14T12:37:19.397Z</atom:updated>
            <content:encoded><![CDATA[<p>For years, data teams have celebrated the arrival of the Lakehouse as the end of the great data wars. We finally had a unified platform that promised the scale and flexibility for AI workloads alongside the reliability and performance required for business intelligence. No more shuttling data between rigid data warehouses and chaotic data lakes. The Lakehouse was our promised land.</p><p>But here’s the uncomfortable truth: having the infrastructure is only half the battle. Without a systematic approach to data management, even the most sophisticated Lakehouse can quickly devolve into the same data swamp we sought to escape. Raw data floods in from dozens of sources, transformations multiply without governance, and data quality becomes a perpetual firefight. Teams end up with terabytes of storage but can’t trust the data enough to make critical business decisions.</p><p>Enter the <strong>Medallion Architecture</strong>. This isn’t just another data pattern — it’s the constitutional framework that transforms your Lakehouse from a mere storage repository into a systematically governed, enterprise-grade data refinery. It provides the methodology to ensure that every piece of data entering your kingdom follows a disciplined path from raw chaos to trusted business intelligence, creating the single source of truth that the Lakehouse promised but couldn’t deliver alone.</p><h3>A Quick Look Back: The Great Data Infrastructure Evolution</h3><p>To truly appreciate why the Medallion Architecture is essential, let’s examine what led us here — the evolution from data warehouses to data lakes to the Lakehouse, and why each step forward created new challenges.</p><h3>The Data Warehouse Era: Rigid Perfection</h3><p>Traditional data warehouses were marvels of engineering discipline. Every table was carefully designed, every ETL process meticulously planned, and data quality was enforced through rigid schemas and expensive validation processes. The result was pristine, highly performant data that business users could trust completely.</p><p>But this perfection came at a crushing cost. Data warehouses required extensive upfront modeling, couldn’t handle unstructured data, and were prohibitively expensive to scale. Worse, they created bottlenecks — every new data source required months of schema design and ETL development. In the age of big data and machine learning, this approach simply couldn’t keep pace.</p><h3>The Data Lake Revolution: Unlimited Chaos</h3><p>Data lakes solved the warehouse’s scalability problem by embracing a fundamentally different philosophy: store everything first, figure out the structure later. Raw data could be dumped directly into cheap cloud storage in its native format. Machine learning teams could access vast datasets without waiting for data modeling committees. It was data democracy at its finest.</p><p>But democracy without governance often leads to anarchy. Data lakes became notorious “data swamps” where:</p><ul><li><strong>Data quality was a constant unknown</strong> — no systematic validation meant corrupted data could propagate for months</li><li><strong>Discoverability became impossible</strong> — thousands of datasets with cryptic names and no documentation</li><li><strong>Lineage was lost</strong> — understanding how data was transformed required archaeological detective work</li><li><strong>Compliance became a nightmare</strong> — proving data quality for regulatory audits was nearly impossible</li></ul><h3>The Lakehouse Promise: Best of Both Worlds</h3><p>The Lakehouse emerged as the synthesis, promising warehouse-quality governance with lake-scale flexibility. Technologies like Delta Lake, Apache Iceberg, and Apache Hudi brought ACID transactions, schema evolution, and time travel to data lakes. The infrastructure was finally in place to have it all.</p><p>But infrastructure alone doesn’t create governance. The Lakehouse provided the <em>capability</em> for data quality, but not the <em>methodology</em>. Teams still needed a systematic approach to organize, validate, and refine their data. Without it, Lakehouses often became just more sophisticated data swamps.</p><h3>The Medallion Advantage: From Storage to Systematic Refinement</h3><p>This is where the Medallion Architecture revolutionizes the landscape, providing the missing methodology that transforms Lakehouse capability into business value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*u97oNT52gPVJpFvfOP-bWw.png" /></figure><h3>How is Medallion Different from Traditional ETL? Systematic Quality Progression</h3><p>Traditional ETL processes often resembled complex spider webs — data moved chaotically between systems, transformations were buried in proprietary code, and understanding the full data lifecycle required extensive institutional knowledge. The Medallion Architecture replaces this chaos with a systematic, three-stage refinement process:</p><p><strong>Bronze Layer: The Archive of Truth</strong></p><ul><li><strong>Pure ingestion</strong>: Data arrives in its raw, unmodified form</li><li><strong>Complete auditability</strong>: Every transformation can be traced back to its source</li><li><strong>Schema resilience</strong>: Changes in source systems don’t break downstream processes</li><li><strong>Compliance foundation</strong>: Raw data provides the legal basis for regulatory audits</li></ul><p><strong>Silver Layer: The Industrial Heart</strong></p><ul><li><strong>Systematic cleansing</strong>: Data quality rules are applied consistently across all sources</li><li><strong>Cross-domain integration</strong>: Related data from different systems is joined and enriched</li><li><strong>Standardized formats</strong>: Common schemas enable reliable downstream consumption</li><li><strong>Quality enforcement</strong>: Bad data is rejected before it can propagate</li></ul><p><strong>Gold Layer: The Business Interface</strong></p><ul><li><strong>Performance optimization</strong>: Data is structured for sub-second query performance</li><li><strong>Business semantics</strong>: Tables reflect business concepts, not technical artifacts</li><li><strong>Self-service analytics</strong>: Analysts can trust the data without understanding its origins</li><li><strong>Scalable consumption</strong>: Multiple teams can access the same data without conflicts</li></ul><h3>How is Medallion Different from Schema-on-Read? Graduated Validation</h3><p>Many data lake approaches embrace “schema-on-read” — leave data unstructured and apply validation only when it’s consumed. This creates a false economy where data quality problems are discovered late, often in production dashboards or critical reports.</p><p>The Medallion Architecture implements “graduated validation” — data quality improves systematically at each layer:</p><ul><li><strong>Bronze validation</strong>: Minimal checks to ensure data can be stored and tracked</li><li><strong>Silver validation</strong>: Comprehensive business rule validation and cross-reference checks</li><li><strong>Gold validation</strong>: Performance and semantic validation for business consumption</li></ul><p>This graduated approach catches data quality issues early while allowing rapid ingestion of new data sources.</p><h3>Technology Agnostic Implementation</h3><p>The Medallion Architecture is fundamentally a methodology, not a technology mandate. While we’ll use specific examples, the principles apply across the entire modern data stack:</p><p>Layer Delta Lake Apache Iceberg Apache Hudi Snowflake BigQuery <strong>Bronze</strong> Schema evolution, ACID Snapshot isolation Timeline server External stages Raw datasets <strong>Silver</strong> MERGE/UPDATE/DELETE Row-level operations Copy-on-write Streams/Tasks DML operations <strong>Gold</strong> Liquid clustering Partition evolution Compaction Clustering keys Partitioning</p><h3>Real-World Impact: Governance, Performance, and Team Productivity</h3><h4>Empowering Teams, Ensuring Quality, Accelerating Delivery</h4><p>From a <strong>data engineering perspective</strong>, the Medallion Architecture eliminates the constant firefighting that plagued traditional approaches. Instead of debugging complex data lineage issues, engineers work within a clear framework where each layer has defined responsibilities. New data sources follow predictable patterns, making onboarding faster and more reliable.</p><p><strong>Example transformation flow:</strong></p><pre>-- Bronze: Preserve raw structure<br>CREATE TABLE bronze.customer_events AS<br>SELECT * FROM source_stream<br>WHERE _ingestion_time &gt; current_timestamp() - INTERVAL 1 DAY;</pre><pre>-- Silver: Clean and conform<br>CREATE TABLE silver.customers AS<br>SELECT <br>    customer_id,<br>    UPPER(TRIM(customer_name)) as customer_name,<br>    LOWER(email) as email,<br>    CAST(signup_date AS DATE) as signup_date<br>FROM bronze.customer_events<br>WHERE customer_id IS NOT NULL;</pre><pre>-- Gold: Business-ready aggregations<br>CREATE TABLE gold.customer_summary AS<br>SELECT <br>    customer_id,<br>    customer_name,<br>    COUNT(*) as total_orders,<br>    SUM(order_value) as lifetime_value<br>FROM silver.customers c<br>JOIN silver.orders o USING (customer_id)<br>GROUP BY 1, 2;</pre><p>From an <strong>analytics perspective</strong>, the Medallion Architecture provides unprecedented data trust. Analysts know that Gold layer data has been systematically validated and is optimized for their queries. They can focus on deriving insights rather than debugging data quality issues.</p><p>From a <strong>governance perspective</strong>, the architecture provides natural audit trails. Regulators can see exactly how sensitive data flows from raw ingestion through business reporting. Data lineage is built into the architecture rather than retrofitted through external tools.</p><h3>Cost Efficiency and Performance Optimization</h3><p>Architecturally, the Medallion approach optimizes costs through <strong>intelligent data placement</strong>:</p><ul><li><strong>Bronze layer</strong>: Uses cheap object storage with compression</li><li><strong>Silver layer</strong>: Balances storage cost with query performance</li><li><strong>Gold layer</strong>: Optimizes for query speed with techniques like clustering and caching</li></ul><p>Recent implementations show <strong>60–80% cost reduction</strong> compared to traditional data warehouse approaches while maintaining sub-second query performance for business intelligence workloads.</p><h3>🔧 Implementation Spotlight: Getting Started</h3><p><strong>Week 1: Foundation Setup</strong></p><pre># Create storage structure<br>/lakehouse/<br>├── bronze/source_system/table/year/month/day/<br>├── silver/domain/table/<br>└── gold/business_area/metric/</pre><p><strong>Week 2: First Data Source</strong></p><ol><li>Identify one critical business data source</li><li>Implement Bronze ingestion with change tracking</li><li>Create basic Silver transformation with quality checks</li><li>Build simple Gold aggregation for immediate business value</li></ol><p><strong>Week 3: Monitoring and Quality</strong></p><ol><li>Add data quality metrics at each layer boundary</li><li>Implement alerting for failed transformations</li><li>Create data lineage documentation</li><li>Set up performance monitoring</li></ol><p><strong>Week 4: Scale and Iterate</strong></p><ol><li>Add second data source following established patterns</li><li>Implement incremental processing for large datasets</li><li>Create self-service tools for business users</li><li>Document lessons learned and refine processes</li></ol><h3>Medallion and Modern Data Architecture: Perfect Synergy</h3><h3>The Data Mesh Connection</h3><p>The Medallion Architecture aligns perfectly with <strong>Data Mesh principles</strong>, providing the systematic governance that domain-oriented data ownership requires:</p><p><strong>Domain-Specific Implementation</strong>: Each business domain can implement its own Medallion layers while following consistent patterns:</p><ul><li><strong>Sales Domain</strong>: sales_bronze → sales_silver → sales_gold</li><li><strong>Marketing Domain</strong>: marketing_bronze → marketing_silver → marketing_gold</li><li><strong>Product Domain</strong>: product_bronze → product_silver → product_gold</li></ul><p><strong>Data as a Product</strong>: The Gold layer naturally creates high-quality data products that other domains can consume with confidence.</p><p><strong>Self-Serve Infrastructure</strong>: The standardized layer approach enables platform teams to provide tooling and governance that works across all domains.</p><p><strong>Federated Governance</strong>: Each domain owns its data transformation logic while following enterprise-wide quality standards.</p><h3>Future-Proofing Your Architecture</h3><p>As new technologies emerge, the Medallion Architecture provides stability. Whether you adopt streaming architectures, edge computing, or next-generation query engines, the systematic refinement principles remain constant. Your data governance investment is protected because it’s built on methodology, not technology.</p><h3>Beyond Architecture: Building Data Culture</h3><p>The Medallion Architecture isn’t just a technical pattern — it’s a cultural transformation that finally brings engineering discipline to data work. It replaces the “move fast and break things” mentality that led to data swamps with a systematic approach that enables sustainable velocity.</p><p><strong>For data engineers</strong>, it provides clear career progression from Bronze (ingestion expertise) to Silver (transformation mastery) to Gold (business intelligence optimization).</p><p><strong>For business stakeholders</strong>, it creates predictable data delivery timelines and trustworthy data products.</p><p><strong>For executives</strong>, it transforms data from a cost center into a strategic asset with measurable quality metrics and clear business value.</p><p>The Medallion Architecture represents the maturation of data engineering from art to science. It’s about moving beyond the old compromises between speed and quality, embracing a systematic approach that delivers both reliability and agility.</p><p>Your Lakehouse provided the foundation. The Medallion Architecture provides the blueprint for building something extraordinary on top of it.</p><p><strong>The methodology awaits. Begin refining.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28974efb4b97" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/the-medallion-architecture-a-methodology-to-prevent-your-lakehouse-from-becoming-a-data-swamp-28974efb4b97">The Medallion Architecture: A Methodology to Prevent Your Lakehouse from Becoming a Data Swamp</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Digital Wild West to Cloud Metropolis: A Practitioner’s Guide to Azure Networking]]></title>
            <link>https://medium.com/data-unlocked/from-digital-wild-west-to-cloud-metropolis-a-practitioners-guide-to-azure-networking-1ed4fe14ad6d?source=rss-eeb195852e1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ed4fe14ad6d</guid>
            <dc:creator><![CDATA[Moses Njau]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 11:59:42 GMT</pubDate>
            <atom:updated>2025-07-08T11:59:42.510Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://github.com/Morse2580/contoso-platform/blob/master/infra/modules/network/README.md">Contoso-Platform</a></h4><h3>Stop Building Digital Shacks. Let’s Architect a Cloud Metropolis.</h3><p>Forget everything you’ve been told about subnets, CIDR blocks, and routing tables for a moment. At its heart, cloud networking is simple. Imagine the cloud is a vast, open frontier of digital land. When you start a new project, you just need a small plot to build on.</p><p>The problem is, most people just start building shacks. Then more shacks. Soon, they have a sprawling, messy, and lawless settlement.</p><p>This guide isn’t about building another shack. It’s about giving you the blueprint to build a modern, secure, and magnificent cloud metropolis from day one. We’re going to walk you through the common pitfalls and show you how to make the right architectural choices so your platform can grow without crumbling under its own weight.</p><h3>The First Mistake: The Digital Suburb</h3><p>When you start out, the easiest thing to do is give every team or project its own plot of land, fenced off with a <strong>Virtual Network (VNet)</strong>. This is your basic digital suburb. The marketing team gets a VNet, the finance team gets a VNet, and the developers get three.</p><p>It seems logical. But what happens when the marketing application needs to get data from the finance database? You build a small private road between them called <strong>VNet Peering</strong>. Then the developers need access to both, so you build more roads.</p><p>Soon, you’re not a city planner; you’re a frantic road builder in a rapidly growing suburb with no master plan. This was the first model we considered for the Contoso Platform: a <strong>Flat VNet-per-Team Model</strong>.</p><ul><li><strong>The Sobering Reality:</strong> The “roads” (peerings) exploded into a tangled mess. Trying to figure out who could talk to whom became a nightmare, and enforcing a single security policy was impossible. We were creating digital spaghetti.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FE4HrpsWYCasmKvotHf3Pg.png" /></figure><p>Some architects try to solve this with a <strong>Full Mesh</strong>, building a direct highway from every VNet to every other VNet. The problem? The number of highways you have to manage grows exponentially. For just 20 VNets, you’d be managing 190 peerings. It’s an administrative nightmare waiting to happen.</p><h3>The Second Mistake: The Giant Open-Plan Office</h3><p>Realizing the road system was a disaster, the next logical step was to bring everyone into one giant building — a <strong>Transit VNet</strong> that hosts all the important shared services. This is a huge improvement! Now, instead of dozens of security guards, you have one main entrance with a single, highly-competent security team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*-46g4QNRpbxo9uuVYZOniw.png" /></figure><p>This was another model we evaluated. It centralized security, but it created a new problem.</p><ul><li><strong>The Sobering Reality:</strong> While it put everyone in the same building, it was like a giant, chaotic open-plan office. There were no walls. The development team’s experimental project was in the same general space as the critical production database. The risk of one team’s work accidentally interfering with another’s (like using the same internal addresses) was far too high. We had centralization, but we had lost <strong>isolation</strong>.</li></ul><h3>The Blueprint for Success: A Hub-and-Spoke Metropolis</h3><p>After seeing the flaws in these common approaches, the correct path became clear. We needed a plan that gave us the best of both worlds: the <strong>centralized services</strong> of the open-plan office and the <strong>secure isolation</strong> of the suburbs.</p><p>We needed to become true city planners. We needed the <strong>Hub-and-Spoke</strong> model.</p><h4>The Hub: Your City’s Central Core 🏢</h4><p>The <strong>Hub</strong> is a dedicated VNet that serves as the industrial and security core of your city. Nothing lives here except the essential, shared services that the entire metropolis depends on:</p><ul><li><strong>The Guardian at the Gates:</strong> Our central <strong>Azure Firewall</strong>.</li><li><strong>The Bridge to the Physical World:</strong> Our <strong>VPN Gateway</strong> to connect back to on-premise offices.</li><li><strong>The Central Post Office:</strong> Shared services like <strong>DNS servers</strong>.</li></ul><p>By placing these here, you build them once, you pay for them once, and you secure them once.</p><h4>The Spokes: Your Secure Neighborhoods 🏡</h4><p>The <strong>Spokes</strong> are other VNets, but they are designed to be isolated neighborhoods. This is where your applications live. You’ll have a spoke for development, a spoke for production, and maybe another for your data analytics workloads.</p><p>The crucial design feature is that spokes cannot talk to each other directly. They are islands. Their only connection is a private highway (<strong>VNet Peering</strong>) back to the central Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xN-0x_ZZwBeikF28.png" /></figure><h4>The Guardian: A Deeper Look at Azure Firewall 🔥</h4><p>That private highway from every spoke leads directly to our guardian: the <strong>Azure Firewall</strong> sitting in the Hub. By using <strong>Route Tables</strong> (think of them as digital GPS systems), we command that any traffic trying to leave its own neighborhood <em>must</em> pass through the firewall for inspection.</p><p>This single checkpoint is incredibly powerful.</p><ul><li><strong>It’s a Bouncer:</strong> It checks every request against a list of rules. You can create rules like, “Allow the production servers to access GitHub, but block them from accessing any other website.”</li><li><strong>It’s an Intelligence Agent:</strong> It’s constantly updated by Microsoft with a list of known malicious websites and attackers, automatically blocking threats before they even get close to your applications.</li><li><strong>It’s a Meticulous Record-Keeper:</strong> Every single connection attempt is logged in one central place. This is a dream for security audits and for understanding what’s truly happening within your network.</li></ul><h3>What About the Future? Planning for a Global Empire</h3><p>The Hub-and-Spoke model is the perfect architecture for building a thriving metropolis in your primary region. But what happens when your company expands globally? Do you build another, separate metropolis in Asia and another in Europe?</p><p>This is where the final evolution comes in: <strong>Azure Virtual WAN (vWAN)</strong>.</p><p>Think of vWAN as Microsoft saying, “Your city has become so successful you now need a national highway and airport system. Let us manage that for you.”</p><p>For an early-stage project like Contoso, building this global transit system from day one would be <strong>overkill</strong>. It’s the right tool, but at the wrong time. Our Hub-and-Spoke design is the perfect, cost-effective starting point that gives us a clear, logical path to adopt vWAN when our scale demands it.</p><p>By deliberately choosing the Hub-and-Spoke model, we rejected the chaos of the digital suburbs and the false economy of the open-plan office. We chose a blueprint that provides <strong>managed complexity</strong>, <strong>strong isolation</strong>, <strong>centralized governance</strong>, and <strong>cost-effective scalability</strong>. We chose to build a metropolis.</p><blockquote><a href="https://learn.microsoft.com/en-us/azure/architecture/networking/architecture/hub-spoke"><strong>Learn More about the Hub-spoke network topology in Azure</strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ed4fe14ad6d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-unlocked/from-digital-wild-west-to-cloud-metropolis-a-practitioners-guide-to-azure-networking-1ed4fe14ad6d">From Digital Wild West to Cloud Metropolis: A Practitioner’s Guide to Azure Networking</a> was originally published in <a href="https://medium.com/data-unlocked">Data Unlocked</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>