<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matheus Palhares on Medium]]></title>
        <description><![CDATA[Stories by Matheus Palhares on Medium]]></description>
        <link>https://medium.com/@matheusjerico?source=rss-7b3af2736fca------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wsuq6Gl8EOLpEsrt3kBT0A.png</url>
            <title>Stories by Matheus Palhares on Medium</title>
            <link>https://medium.com/@matheusjerico?source=rss-7b3af2736fca------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:29:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matheusjerico/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Took Myself Out of the Implementation Loop using Grok Bot]]></title>
            <link>https://medium.com/codetodeploy/how-i-took-myself-out-of-the-implementation-loop-using-grok-bot-51fb4caa3295?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/51fb4caa3295</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[grok]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[grok-bot]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 19:07:57 GMT</pubDate>
            <atom:updated>2026-10-02T19:07:57.272Z</atom:updated>
            <content:encoded><![CDATA[<h4>I still define the work and approve the merge. Grok Bots coordinate implementation, verification, recovery, and deployment.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-eeGidYBP2GXOpGcQgRmvw.jpeg" /><figcaption>AI Maturity Curve, from the Grok Bot Galaxy event.</figcaption></figure><blockquote><em>💥 </em><strong><em>Master AI &amp; Tech Skills</em></strong><br> 🎓 Get Up to 50% OFF Premium Courses<br> ⏰ Limited-Time Offer<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITogvtC6mF2BUF35Qsd9pg.png" /></figure><h3>The engineering work I want to focus on</h3><p>My goal as a software engineer is to spend more of my time understanding the problem we need to solve and defining the requirements, while the Agents carry more of the implementation forward. I want the business requirements document (BRD) to explain who needs the change and what a useful outcome looks like, with the constraints clear before work starts.</p><p>I started moving in this direction with spec-driven development (SDD). In <a href="https://matheusjerico.medium.com/ai-needs-specs-not-just-prompts-846b4f033191">AI Needs Specs, Not Just Prompts</a>, I wrote about defining scope, shared data models, and acceptance criteria before implementation. Those decisions need to survive between conversations. My Grok Bot workflow continues that approach: specifications still guide the work, and the Agents coordinate its execution.</p><p>I agree with the direction of Augusto Galego’s <a href="https://www.youtube.com/watch?v=QyVFkGuswl4">Se afaste do código o mais rápido possível</a>. Spending less time in the implementation loop makes me think harder about which problems deserve that implementation. <a href="https://www.pathtostaff.com/p/business-sense-for-engineers-the">Business Sense for Engineers</a> argues for understanding how the business works alongside product judgment. I want to connect requirements to the people affected and to a business outcome, including whether a proposed change is worth the effort.</p><p>That also gives me a reason to practice skills that can be easy to neglect when I am focused on implementation. <a href="https://formation.dev/blog/5-nontechnical-skills-that-matter-for-software-engineers-in-the-age-of-ai">Formation’s discussion of nontechnical skills</a> covers product thinking, communication, problem decomposition, prioritization, and curiosity. In this workflow, those skills help me explain a requirement, separate a large problem into work that can be verified, and decide what should be done first.</p><p>I still need technical depth to evaluate the result. <a href="https://github.blog/news-insights/octoverse/the-new-identity-of-a-developer-what-changes-and-what-doesnt-in-the-ai-era/">GitHub’s research on developer identity</a> describes developers directing AI work and verifying its output. <a href="https://www.ibm.com/think/perspectives/what-every-future-software-engineer-must-know">IBM’s perspective on engineering skills</a> emphasizes systems thinking, data fluency, and validation. Delegating implementation leaves me responsible for understanding the proposed architecture and judging whether the evidence supports the change.</p><p>For AI engineers and data engineers, a pipeline change might need acceptance checks for data freshness and quality. A data contract could require schema compatibility with existing consumers. For a change to an inference service, agreed test cases can define the expected model behavior.</p><p>My next step with SDD is to let the Agents carry the agreed specifications through implementation and verification without needing a message from me at every handoff. I want my attention concentrated on requirements and the BRD. In my current setup, I also agree the technical direction, take part when scope changes, and approve the merge and production release.</p><h3>The workflow I use</h3><p>In my previous coding-agent workflow, I had to keep each conversation moving. I guided the implementation, carried the specification into the next step, and decided what to do when the agent got stuck. Even with a spec-driven process, I was still steering the work through chat.</p><p>I wanted to agree on the work and return to a change that had been exercised and reviewed, without sending another message at each step. I built a workflow with Grok Bot and Cursor cloud agents for my day-to-day engineering work. I spend time with intake on requirements and technical direction, then the Bots coordinate implementation and route failures. Chief Agent brings me the pull request for approval and merge; intake comes back to me if the agreed scope needs to change.</p><figure><img alt="Conceptual illustration of a human defining work and approving a result while connected agent modules coordinate execution." src="https://cdn-images-1.medium.com/max/1024/1*20KesDilI4M4ySrA8Hc7Nw.png" /><figcaption>An editorial illustration of the operating model; it does not depict the computer architecture.</figcaption></figure><h3>Agreeing on the work</h3><p>Brainstorm Agent interviews me about what the system should do and turns our discussion into a business requirements document and a technical specification in Notion, linked from the Linear issue. The requirements describe the problem and intended outcome; the specification sets the implementation direction. Ticket Ready Agent works with me on acceptance criteria: the behavior we will check to decide whether the ticket has been delivered. These artifacts give the Bots the agreed scope without requiring me to join every implementation conversation.</p><p>If either document is missing, Ticket Ready Agent stops and requests Brainstorm Agent. It works through unresolved questions with me and finalizes the acceptance criteria once the documents are ready. It does not write the missing business document or change the issue’s status. It sends Engineering Manager Agent the card and a readiness signal.</p><p>Meeting Analyst Agent also automatically analyzes team-meeting transcripts and feeds relevant work into Brainstorm Agent or Ticket Ready Agent for refinement. I still participate when requirements, technical direction, or acceptance criteria need agreement. I want those questions settled during intake, before ambiguity reaches implementation.</p><h3>From agreed scope to live verification</h3><p>Engineering Manager Agent can move the ready issue to In Progress and hand it to Planner Agent. The issue stays assigned to me. The fleet works on issues assigned to me in repositories I have approved; another repository requires my explicit approval.</p><p>Planner Agent turns the agreed work into an implementation plan. Engineer Agent uses Cursor cloud agents to implement it and opens a draft pull request. Review Agent examines and approves the code and its alignment with the acceptance criteria. That approval does not make the change ready to merge: the sandbox checks still have to follow.</p><figure><img alt="Delivery flow: author and intake agree scope; Engineering Manager and Planner coordinate; Engineer opens a draft PR; Review approves; required sandbox tests pass; Infra Manager and optional IaC deploy the candidate to sandbox; Eval verifies acceptance criteria on that live revision; Chief asks the author to merge and authorize production; infrastructure deploys through GitHub Actions; Release performs smoke checks and produces the evidence pack; Deploy Watch observes production." src="https://cdn-images-1.medium.com/max/1024/1*x8Fecq62Pr_Dlv7TwuEZ8Q.png" /><figcaption>The normal delivery path in my setup. Agreeing scope happens during intake; approving and merging the reviewed PR is the human decision that authorizes production delivery.</figcaption></figure><p>After Review Agent approves, the required sandbox test suite must pass. Infra Manager Agent, with IaC Agent when needed, then deploys the candidate revision into the sandbox environment. The Linear card reaches In Review at this stage, not Done.</p><p>Eval Agent then follows verification procedures to check the acceptance criteria against that live sandbox revision and attaches evidence to the pull request and Linear. A passing unit-test or CI run alone cannot provide that sign-off. Sandbox behavior checks, screenshots, and pull-request review supply complementary evidence: reviewed code, passing tests, a deployed candidate, and verified behavior tell me different things about the change.</p><h3>Routing failures</h3><p>Engineering Manager Agent routes sandbox and evaluation failures according to their cause. Code or implementation problems go to Engineer Agent or Review Agent for correction within the agreed requirements. Environment problems go through Infra Manager Agent to the infrastructure specialists. An application check can fail because the environment is wrong; changing application code would not necessarily fix it. The Bots handle those handoffs without me relaying the failed check to its owner.</p><p>Problems with the requirements, technical specification, or acceptance criteria return to intake. If an agreed artifact needs to change, intake asks me. I remain responsible for the requirements and technical direction, including decisions that need revisiting. The Bots must not silently rewrite an acceptance criterion to make a failed check pass.</p><figure><img alt="Failure routing: a sandbox or evaluation failure goes to Engineering Manager; code problems go to Engineer or Review; environment problems go to Infra Manager and infrastructure specialists; scope problems go to Brainstorm or Ticket Ready, who involve the author when agreed requirements, specifications, or criteria must change. Repaired candidates re-enter the applicable delivery checks, including live Eval." src="https://cdn-images-1.medium.com/max/1024/1*vrAQDqeGk9dvBrfkBCzCBw.png" /><figcaption>Implementation can be repaired within the agreed scope. Changes to that scope bring me back into the conversation.</figcaption></figure><p>Repaired candidates return to the applicable verification steps, including checks by Eval Agent against the live sandbox revision. Routing a fix does not make the change ready to merge.</p><h3>My merge authorizes production</h3><p>Chief Agent asks me once per card to merge the pull request into the main branch and authorize production. The request comes after Review Agent approval, passing the required sandbox suite, candidate sandbox deployment, and Eval Agent’s sign-off on the deployed sandbox revision, unless I record an explicit written waiver. I can consider the implementation and its evidence at this point.</p><p>My approval and merge are the single production decision for the card. They authorize Infra Manager Agent and IaC Agent to trigger the GitHub deployment and monitor GitHub Actions for successful deployment. Scope agreement during intake is a separate responsibility.</p><p>After production deployment, Release Agent owns the production smoke checks and evidence pack. Deploy Watch Agent follows by observing runtime behavior, logs, HTTP responses, errors, and alerts. A successful deployment job reports on the deployment process; these observations concern the running application. For the repositories in this workflow, that observation is informational, not an additional approval or hard completion gate.</p><h3>The platform underneath the roles</h3><p>The roles and handoffs are my design. Grok Bot’s <a href="https://x.ai/bot/guides/grok-bot-for-engineering">engineering guide</a> describes capabilities that support this workflow: Bots can launch cloud agents, read their transcripts, inspect artifacts, and send follow-ups.</p><p>The Grok Bots in an account share one cloud computer, including files, browser sessions, and logins. Each Bot has its own screen, conversation, and learned context. Different user accounts are isolated; adding Bots within an account does not give them separate computers. The <a href="https://docs.x.ai/grok-bot/overview">official overview</a> explains this model.</p><p>Grok Bot has a <a href="https://docs.x.ai/grok-bot/computer-and-apps">filesystem and terminal</a> and can edit files. I assign implementation to Cursor cloud agents by choice.</p><p>A Bot’s role defines ownership and handoffs. Skills describe how to perform tasks and can include when to use them. Private skills live in a shared library. A routine has an owning Bot and runs on a schedule or a supported, configured event. The <a href="https://docs.x.ai/grok-bot/skills-routines-and-automations">skills and routines documentation</a> covers these mechanics.</p><p>Instructions about ownership and review do not, by themselves, guarantee access control. The approval boundary is our operating policy; the actual controls depend on the tools and permissions configured around it.</p><h3>What reaches me</h3><p>Chief Agent is my sole point of contact for operational interruptions. Specialists escalate through their managers and Chief Agent instead of opening parallel private conversations with me. Actionable handoffs take priority, and routine status is batched. Intake still works with me on scope decisions.</p><p>The Active Board can show the current owner, a readable status, the next gate, and the pull request, so I can follow the work without piecing together several chats.</p><h3>References and further reading</h3><ul><li><a href="https://docs.x.ai/grok-bot/overview">Grok Bot overview</a>: Persistent Bots, per-Bot context, and the account’s shared cloud computer.</li><li><a href="https://docs.x.ai/grok-bot/bots">Create and manage Bots</a>: Defining focused jobs, ownership boundaries, and specialist teams.</li><li><a href="https://docs.x.ai/grok-bot/chat-and-collaboration">Message and collaborate</a>: Direct messages, group chats, and asynchronous Bot-to-Bot handoffs.</li><li><a href="https://docs.x.ai/grok-bot/skills-routines-and-automations">Skills and routines</a>: Reusable instructions, shared private skills, routine ownership, schedules, and supported event triggers.</li><li><a href="https://docs.x.ai/grok-bot/computer-and-apps">Use the computer and apps</a>: Browser sessions, filesystem and terminal access, and shared resources.</li><li><a href="https://x.ai/bot/guides/grok-bot-for-engineering">Grok Bot for Engineering</a>: Practical examples of coordinating Cursor cloud agents and checking transcripts, screenshots, and other artifacts.</li><li><a href="https://x.ai/bot/guides/grok-bot-for-pms">Grok Bot for PMs</a>: Examples of product research, engineering management, and specialist Bot teams.</li><li><a href="https://matheusjerico.medium.com/ai-needs-specs-not-just-prompts-846b4f033191">AI Needs Specs, Not Just Prompts</a>: My earlier article on scope, shared models, and acceptance criteria in AI-assisted development.</li><li><a href="https://www.youtube.com/watch?v=QyVFkGuswl4">Se afaste do código o mais rápido possível</a>: Augusto Galego’s reflection on the engineering role beyond writing code.</li><li><a href="https://www.pathtostaff.com/p/business-sense-for-engineers-the">Business Sense for Engineers: The Missing Half of Product Sense</a>: Connecting engineering decisions to business priorities.</li><li><a href="https://formation.dev/blog/5-nontechnical-skills-that-matter-for-software-engineers-in-the-age-of-ai">5 nontechnical skills that matter for software engineers in the age of AI</a>: Product thinking, communication, problem decomposition, prioritization, and curiosity.</li><li><a href="https://github.blog/news-insights/octoverse/the-new-identity-of-a-developer-what-changes-and-what-doesnt-in-the-ai-era/">The new identity of a developer: What changes and what doesn’t in the AI era</a>: GitHub’s qualitative research on experienced AI users directing and verifying work.</li><li><a href="https://www.ibm.com/think/perspectives/what-every-future-software-engineer-must-know">Beyond Code: How AI redefines what it means to be an engineer</a>: IBM’s perspective on systems thinking, data fluency, and validation.</li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51fb4caa3295" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/how-i-took-myself-out-of-the-implementation-loop-using-grok-bot-51fb4caa3295">How I Took Myself Out of the Implementation Loop using Grok Bot</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Agent Can Already Read the Code. It Still Needs a Knowledge Base.]]></title>
            <link>https://matheusjerico.medium.com/your-agent-can-already-read-the-code-it-still-needs-a-knowledge-base-506e3cc6af2d?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/506e3cc6af2d</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[pydantic]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 21:19:33 GMT</pubDate>
            <atom:updated>2026-09-29T21:37:55.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>A field guide to retrieving and evaluating evidence for coding agents.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qfA-FJ2xjPyg2CQef7BmnQ.png" /><figcaption>A thin base with task-specific retrieval is a useful starting point. Compare it with full-context prompting for a small, bounded corpus.</figcaption></figure><p>I work on agents for PR review, incident triage, and CI analysis. They can inspect a repository, follow a failure, and produce a plausible explanation. The harder questions often concern something outside the working tree. Why does a deployment exception exist? Which remediation is approved? Answering may require the last similar incident or a record of the trade-off a service owner accepted.</p><p>Those tasks need maintained, discoverable evidence with sources and ownership. That is what I mean by a knowledge base. A separate service or vector database is optional, and the repository may be all a code-only task needs. A decision nobody recorded remains unavailable as verified fact.</p><p>Search can still return incomplete, stale, or irrelevant material that makes an answer look well supported. For a small, bounded corpus, I would also test supplying the full material in context. Loading an ever-growing document collection uses context without showing that the extra material helps. Before choosing a vector database, I want to know which claims need external evidence and what the agent should do when it cannot find it.</p><h3>When to retrieve</h3><p>Pre-inference retrieval selects evidence before the task’s first generation and can use the user’s question to do so. This differs from preloading a stable base of repository instructions, team conventions, or tool contracts. Retrieving upfront may avoid an extra agent tool round trip, although search, reranking, and prompt length still contribute to total latency.</p><p>With just-in-time retrieval, the agent forms a query during the task, inspects the results, and can search again using what it has learned. That selectivity costs another model turn, a tool call, and retrieval latency. There is also a behavioral failure to account for: the agent may skip a search that policy requires.</p><p>I start with a hybrid: preload the small amount of material that is consistently useful and give the agent retrieval tools for task-dependent evidence. I treat that as a design to test. Anthropic’s <a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents">context-engineering guide</a> describes Claude Code combining upfront CLAUDE.md context with on-demand navigation.</p><p>My rule is to retrieve before making claims that need internal, mutable, private, high-risk, or explicitly cited evidence. Stable general knowledge may need no search at all. If expected organizational evidence is missing, the answer should say what is missing.</p><p>Instructions alone cannot guarantee that behavior. Trusted application logic must check the required evidence path before releasing an answer. The backend also has to apply the authenticated caller’s tenant and read permissions before restricted content reaches the model. A prompt or model-supplied identity cannot establish those permissions.</p><p>Google’s <a href="https://research.google/blog/unlocking-dependable-responses-with-gemini-enterprise-agent-platforms-agentic-rag/">June 2026 Agentic RAG report</a> reports up to a 34% accuracy improvement for its complete architecture on FramesQA: 824 questions over 2,676 PDFs, with LLM-judged answers. The experiment concerns multi-hop question answering. It does not measure coding-agent productivity or isolate the contribution of any one retrieval component.</p><h4>A Pydantic AI tool sketch</h4><p>This Pydantic AI sketch shows how to bound an input, return structured evidence, and instruct the agent about when to search. The trusted runtime and retrieval backend are left out.</p><pre>from datetime import datetime<br>from typing import Annotated<br><br>from pydantic import BaseModel, Field<br>from pydantic_ai import Agent<br><br><br>class KnowledgeHit(BaseModel):<br>    content: str<br>    source: str<br>    updated_at: datetime<br>    relevance: float<br><br><br>agent = Agent(<br>    &quot;anthropic:claude-sonnet-4-6&quot;,<br>    instructions=(<br>        &quot;Retrieve before making claims about internal, mutable, private, &quot;<br>        &quot;high-risk, or citation-required information. If expected evidence &quot;<br>        &quot;is missing, state the gap instead of guessing.&quot;<br>    ),<br>)<br><br><br>@agent.tool_plain<br>async def search_knowledge(<br>    query: str,<br>    k: Annotated[int, Field(ge=1, le=10)] = 6,<br>) -&gt; list[KnowledgeHit]:<br>    &quot;&quot;&quot;Search runbooks, incidents, review outcomes, and decisions.&quot;&quot;&quot;<br>    return await retrieve_and_rerank(query=query, limit=k)</pre><p>You would supply the application-specific retrieve_and_rerank backend. <a href="https://pydantic.dev/docs/ai/tools-toolsets/tools/">Pydantic AI&#39;s function-tool documentation</a> explains how signatures and docstrings produce tool schemas, and its <a href="https://pydantic.dev/docs/ai/models/anthropic/">Anthropic documentation</a> supports the model identifier used here. The bound on k validates the argument. The backend must still limit the results and their token count; that bound says nothing about output size or relevance.</p><p>A real implementation also needs trusted runtime context to supply the caller’s access scope, plus stable source/version identifiers and applicability information in the results. The model must not choose its own access scope. Treat relevance as a ranking signal, never as the probability that a claim is true. Likewise, updated_at tells you when someone edited a record, without establishing that the procedure is approved or still effective.</p><h3>Choosing a retrieval pipeline</h3><p>Lexical retrieval is useful for exact identifiers, error strings, product names, and policy language. Dense retrieval can find related passages whose wording differs from the query. When both return candidates, fusion can combine their lists. A reranker applies a more precise relevance estimate to reorder candidates before the final evidence is returned.</p><p>The useful combination depends on the corpus. Exact operational terms may favor lexical search; other material may benefit from dense retrieval or a mix of both. None of these stages is mandatory. Evaluate the candidate pool size and final k as well, instead of inheriting them from a framework default or a diagram.</p><p>Contextual retrieval adds a short explanation of a chunk’s place in its source before embedding or indexing it. That extra context can help with passages that are ambiguous on their own. It also adds work at ingestion and another artifact to check.</p><figure><img alt="Anthropic’s reported retrieval failure rate, one minus recall at 20: baseline 5.7%, contextual embeddings plus contextual BM25 2.9%, and with reranking 1.9%; the relative reductions are 49% and 67%." src="https://cdn-images-1.medium.com/max/1024/1*c8iWe_vRJPHFSCN7cIVh-w.png" /><figcaption>Retrieval failure rate (1 − recall@20), lower is better. <a href="https://www.anthropic.com/engineering/contextual-retrieval">Anthropic’s September 2024 evaluation</a> measured retrieval in its tested domains, not answer accuracy.</figcaption></figure><p>Anthropic reported a baseline top-20 retrieval failure rate of 5.7%. Contextual embeddings combined with contextual BM25 brought it to 2.9%, a 49% relative reduction. With reranking, it fell to 1.9%, a 67% relative reduction. These percentages describe failures to retrieve relevant context. They do not measure gains in answer accuracy or tell us to send 20 chunks to every model call.</p><p>Use a defined corpus and metric to test the change your failures suggest. Missed exact error strings are a reason to try lexical retrieval. Fusion or reranking may help when relevant results arrive but rank poorly, while contextualization addresses chunks that lose meaning outside their parent document.</p><figure><img alt="An optional retrieval pipeline: a query selects lexical retrieval, dense retrieval, or both; fusion and reranking may be added before evidence is returned. Tune counts and evidence size through evaluation." src="https://cdn-images-1.medium.com/max/1024/1*NNW39AOll5mQ6TNnyJrBbA.png" /><figcaption>Candidate generation, fusion, reranking, and final selection are options to evaluate. Keep the stages that help on your tasks.</figcaption></figure><h3>What the repository can tell you</h3><p>Ordinary repository search and navigation give an agent access to files, tests, configuration, and recorded history near the checked-out state. That is enough for many questions about the code. A plausible explanation still cannot establish a rationale that nobody recorded.</p><p>Structural parsing, dependency graphs, call graphs, or semantic indexes can reduce search cost or reveal relationships across files. I would add them when evaluation shows that ordinary file, lexical, and symbol search falls short. Code being embeddable is not sufficient reason to build an index, and any index needs to identify the revision it describes.</p><p>Runbooks, incident records, prior review outcomes, architectural decisions, exceptions, and ownership history provide organizational evidence. Some already live with the code: <a href="https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions">Nygard’s original ADR proposal</a> keeps decisions in the project repository. Existing document and incident systems can hold the rest.</p><p>Implementation evidence explains what a revision does. Organizational evidence helps establish which decision or procedure applies, who owns it, and whether it remains valid. A task may need either or both, through one search system or several. Shared storage still has to preserve provenance, freshness, and access rules for each source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yjc3hL7kQWXflHiA466sdQ.png" /><figcaption>Either evidence path can be used first or on its own. They may share a repository or document system, with explicit permissions and provenance.</figcaption></figure><h3>An illustrative CI failure</h3><p>Suppose a CI agent sees PERMISSION_DENIED after an infrastructure change. Repository search finds the failing call and its configuration path, but may leave the applicable organizational procedure unresolved. This is an illustrative scenario, not a measured result from my system.</p><p>The diagnosis now needs operational evidence, so the policy requires search_knowledge. The query includes the error, environment, service, and recent change. A trusted backend restricts the search to sources the caller may read. For this example, assume it returns a current runbook and a prior incident, both with source/version and applicability metadata.</p><p>I would want the ranking to favor the affected environment over documents that merely repeat the error string. Evaluation has to show whether the reranker does that. The answer should cite the failing call as an observation from the repository, the possible cause as an interpretation supported by the incident, and the diagnostic recommendation as guidance from the runbook.</p><p>If authoritative, applicable evidence is missing or contradictory, the agent should report the gap. Permission to act needs a separate check against the current caller and workflow, including any required approval. Retrieving a runbook grants no permission to execute its procedure. Pydantic AI distinguishes <a href="https://pydantic.dev/docs/ai/tools-toolsets/deferred-tools/">human approval from authorization</a>. After resolution, a proposed lesson belongs in write-back staging before it can become a shared operating rule.</p><figure><img alt="An agent proposes an evidence-backed shared lesson, stages it with sources and scope, and passes it through governed promotion. Accepted records feed indexes and caches for later retrieval. Versions, revocations, and supersession propagate; scratch notes are not automatically shared policy." src="https://cdn-images-1.medium.com/max/1024/1*kEcqAJ1W5_1IIGFE_AeFvg.png" /><figcaption>Validate proposed shared lessons before promotion, then propagate version and access changes to retrieval. Temporary notes have a different status from organizational policy.</figcaption></figure><p>Trusted application controls must enforce the evidence checks, permitted-source filters, and action approvals in this design. Drawing those boundaries or describing them in the model’s instructions does not enforce them.</p><p>Treat retrieved text as untrusted input, including material from internal systems. <a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">OWASP’s prompt-injection guidance</a> covers malicious instructions carried by external content. Separate that content from control instructions and restrict the actions available to the agent. Test adversarial documents too: RAG and citations do not remove the prompt-injection risk.</p><h3>Checking the result</h3><p>Turn a failure like the CI example into an evaluation case. Record the question, expected evidence, permitted sources, required retrieval behavior, and the claims an acceptable answer must include or avoid. Use the simplest path as a baseline, then build a small held-out set of representative tasks. Include missing, stale, superseded, contradictory, and unauthorized-source cases before drawing conclusions beyond that first example.</p><p>For retrieval, Source-ID Recall@k is the fraction of all labeled relevant sources found in the top k results. Hit rate asks whether at least one appears; the two coincide only when there is one relevant source. MRR or nDCG provides ranking information. Ragas documents <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/">context precision</a> and <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/">context recall</a>, with reference-answer-based and source-ID variants. Coverage of a reference answer is useful, but cannot establish exhaustive corpus recall.</p><p>The answer needs its own checks for correctness, completeness, and useful abstention. Ragas <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/">faithfulness</a> measures support from the supplied context. The context can still be false, outdated, incomplete, or unauthorized. Calibrate imperfect LLM judges against human-labeled examples before relying on their scores.</p><p>Follow each citation to the version of the source the agent used and check whether it supports the adjacent claim. A relevant document can still fail to support that particular sentence.</p><p>Check how the agent searched, too. Did it retrieve when required, skip unnecessary searches, reformulate a weak query, and stop when evidence stayed insufficient? Set limits on search rounds and evidence size to prevent an unresolved question from becoming an unlimited loop. Retrieval metrics alone will miss these behavioral failures.</p><p>Track latency, cost, the number of searches, and how often the agent encounters stale or conflicting evidence. Those measurements tell you whether an accuracy improvement is affordable for your workload. A benchmark from someone else’s system cannot settle that.</p><h3>A first implementation</h3><p>I would start with one real agent failure and one knowledge source. Identify the missing evidence and label the source that should have supplied it. Establish a baseline with ordinary search, or full-context prompting if the corpus is suitably bounded, then build the simplest path that returns cited evidence. Measure the final answer before adding hybrid retrieval, reranking, contextualization, a code graph, or routing to address a measured failure.</p><p>For each added stage, specify the improvement you expect: recovering a missing source, improving its rank, reducing irrelevant context, controlling latency, or changing tool behavior.</p><p>The <a href="https://arxiv.org/abs/2602.11988v2">June 2026 revision of Evaluating AGENTS.md</a> found no significant aggregate task-success benefit from repository context files in its tested settings, while inference costs increased. Its subject was instruction files on coding benchmarks, so the result does not establish how enterprise knowledge bases or task-retrieved evidence perform. I take it as a reason to test whether added context helps, rather than count the documentation we can load.</p><figure><img alt="Five-step build order that puts evaluation before retrieval optimizations" src="https://cdn-images-1.medium.com/max/1024/1*agDEO9HSv_8orOsbWZsStg.png" /><figcaption>Choose a source and a failure, label the evidence, and measure a baseline. Add complexity in response to measured errors.</figcaption></figure><h3>Other retrieval approaches</h3><p><a href="https://microsoft.github.io/graphrag/query/overview/">Microsoft GraphRAG</a> has local search that combines graph information with source text, and global search that uses generated community reports. These can help with relationships or corpus-wide themes when an evaluation case calls for them. They are optional, and the generated graphs and reports need provenance and a way to stay up to date.</p><p><a href="https://aclanthology.org/2024.naacl-long.389/">Adaptive-RAG</a> predicts question complexity to choose among no retrieval, single-step retrieval, and iterative retrieval. It was evaluated on open-domain question answering, separate from this article’s CI scenario. I would consider routing once there are distinct, evaluated paths to choose from.</p><p><a href="https://github.com/VectifyAI/PageIndex">PageIndex</a> is a vectorless, reasoning-based RAG approach. It builds a hierarchical document index and uses an LLM to navigate toward relevant sections. I would consider it for long, structured reports or manuals where the document’s organization helps locate evidence. It still follows the RAG pattern of retrieving material to support an answer.</p><p>VectifyAI <a href="https://github.com/VectifyAI/Mafin2.5-FinanceBench">reports 98.7% answer accuracy</a> for its complete Mafin 2.5 financial QA system on the 150 public <a href="https://github.com/patronus-ai/financebench">FinanceBench questions</a>. That result does not isolate PageIndex’s contribution or establish a general advantage over vector or hybrid retrieval.</p><p>For a coding agent, I would test tree-based retrieval against the existing search baseline on representative questions. Check whether the cited sources support the answer, along with latency, cost, and behavior after document updates. The <a href="https://docs.pageindex.ai/sdk/agents">PageIndex SDK documentation</a> also says document and folder context guide the agent without restricting tool access. Permission checks still belong in trusted application code.</p><h3>Updating shared knowledge</h3><p>An agent’s proposed shared lesson needs validation before it becomes accepted organizational knowledge. I would stage it with its task, supporting evidence, summary, authoring agent, and provenance. Before promotion, validate and deduplicate it, preserve ownership and access controls, and define its version and validity behavior. Keep snapshots and a rollback path.</p><p><a href="https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/">OWASP’s vector and embedding guidance</a> supports permission-aware retrieval and source validation. Staging and rollback are my proposed implementation choices; OWASP does not prescribe that architecture. Check permissions at read time and propagate revocations, deletions, and superseded versions to indexes and caches. Retain historical evidence with its status so that a recent edit does not automatically overrule it.</p><p>Automatic scratch notes and personal memory can support continuity at a different trust level. An inferred lesson needs deliberate promotion before it becomes organizational policy. Keep observed events, tentative interpretations, and approved operating rules distinguishable.</p><h3>References and further reading</h3><p>These sources describe the mechanisms and experiments discussed here. The reported results apply to their tested settings, without establishing a design that works for every agent.</p><ul><li><a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents">Anthropic: Effective context engineering for AI agents</a>. Hybrid context and on-demand navigation.</li><li><a href="https://www.anthropic.com/engineering/contextual-retrieval">Anthropic: Contextual Retrieval</a>. The September 2024 retrieval experiment, its metric, and its scope.</li><li><a href="https://research.google/blog/unlocking-dependable-responses-with-gemini-enterprise-agent-platforms-agentic-rag/">Google Research: Agentic RAG</a>. A complete architecture evaluated on multi-hop question answering.</li><li><a href="https://pydantic.dev/docs/ai/tools-toolsets/tools/">Pydantic AI: Function Tools</a> and <a href="https://pydantic.dev/docs/ai/models/anthropic/">Anthropic models</a>. The interfaces used in the intentionally incomplete tool sketch.</li><li><a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/">Ragas: Context precision</a>, <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/">context recall</a>, and <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/">faithfulness</a>. Distinct retrieval and answer-support measurements.</li><li><a href="https://arxiv.org/abs/2602.11988v2">Evaluating AGENTS.md, June 2026 revision</a>. Evidence about repository context files in the tested coding-agent settings.</li><li><a href="https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions">Documenting Architecture Decisions</a>. The original proposal for keeping decisions with a project.</li><li><a href="https://microsoft.github.io/graphrag/query/overview/">Microsoft GraphRAG query overview</a> and <a href="https://aclanthology.org/2024.naacl-long.389/">Adaptive-RAG</a>. Graph-based and complexity-routed retrieval approaches.</li><li><a href="https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/">OWASP: Vector and Embedding Weaknesses</a> and <a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">Prompt Injection</a>. Corpus, access-control, and untrusted-content risks.</li><li><a href="https://github.com/VectifyAI/PageIndex">PageIndex</a>, <a href="https://github.com/VectifyAI/Mafin2.5-FinanceBench">Mafin 2.5 evaluation</a>, <a href="https://github.com/patronus-ai/financebench">FinanceBench</a>, and <a href="https://docs.pageindex.ai/sdk/agents">SDK agent documentation</a>. Tree-based retrieval, the reported financial QA result, and access-scope caveats.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=506e3cc6af2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Garmin to Telegram: Building My Own AI Training Coach]]></title>
            <link>https://matheusjerico.medium.com/from-garmin-to-telegram-building-my-own-ai-training-coach-6f82b0405775?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f82b0405775</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[garmin]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 12:01:01 GMT</pubDate>
            <atom:updated>2026-09-07T12:01:01.665Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An engineering walkthrough of the data pipeline, agent memory, structured planning, and production failures behind Garmin Coach.</em></p><p>My Garmin already collected the data: activities, sleep, heart rate, heart rate variability, and recovery indicators. What I wanted was continuity between those measurements and the decisions around my training.</p><p>A conversation should remember the plan we discussed. A morning briefing should use the sleep data that actually arrived. A weekly review should compare what was planned with what happened. And changing a workout should update the system’s records as well as the message on my phone.</p><p>That became Garmin Coach, a personal application with Telegram as its interface, PostgreSQL as its durable state, and language models handling conversation and structured planning.</p><p>For a software engineer, the interesting part is how these pieces fit together. The project took me through data modeling, asynchronous delivery, tool permissions, model output validation, memory, and the awkward boundary between a database transaction and an external API.</p><p>This is a walkthrough of the implementation. The scheduled coach and Telegram webhook are currently paused; the separate MCP connector remains available. The examples below describe the system I built, without publishing personal health records or credentials.</p><h3>The product starts with a daily loop</h3><p>When enabled, the system is configured to collect Garmin data several times a day. A morning job evaluates the available recovery signals and prepares a briefing. During the day, I can ask questions or record a check-in through Telegram. An evening digest compares the plan with the activities that were recorded.</p><p>The weekly cycle adds a review, a new plan, and synchronization of supported workouts to Garmin Connect. The watch receives those workouts through Garmin’s own synchronization.</p><p>An example interaction might be: “I cannot train tomorrow. What needs to change in this week’s plan?” That sentence needs more than a text response. The application must locate the current week, inspect the sessions already stored, apply the relevant constraints, and keep subsequent answers consistent with the change.</p><p>I designed the system around one athlete and one set of integrations. There is no tenant isolation or account onboarding flow. That scope made it possible to explore the product without first building a platform around it.</p><h3>Four application components, with PostgreSQL between them</h3><figure><img alt="Garmin Connect feeds an ingestion job, which writes to PostgreSQL. Telegram connects to the chat agent, and scheduled workflows read and write application state. A separate synchronization workflow sends supported workouts to Garmin Connect." src="https://cdn-images-1.medium.com/max/1024/1*B3LzxAwKIzanKNHLFP6m1A.png" /><figcaption>The core runtime paths. Arrows show data and request flow; supporting cloud services are omitted. Diagram by the author.</figcaption></figure><p>The repository separates the application into four components:</p><ul><li><strong>ingestion</strong> fetches Garmin data, stores raw responses, transforms them, and runs the deterministic daily briefing and digest.</li><li><strong>agent</strong> contains the Telegram webhook, PydanticAI agents, tools, weekly workflows, and workout synchronization.</li><li><strong>shared</strong> owns database migrations and common domain behavior, including calendar rules, planned sessions, and the running protocol.</li><li><strong>mcp-server</strong> exposes a separate authenticated Garmin connector for external AI clients. The Telegram coach does not route all its work through that remote endpoint.</li></ul><p>The infrastructure is declared in Terraform. Cloud Run services handle HTTP requests; Cloud Run jobs execute work that has a beginning and an end. Cloud Scheduler starts the recurring jobs. Supabase hosts PostgreSQL, OpenRouter provides access to language models, Vertex AI supplies embeddings, and Langfuse records model traces. The distinction between services and jobs follows <a href="https://docs.cloud.google.com/run/docs/overview/what-is-cloud-run">Cloud Run’s execution model</a>.</p><p>There is a useful deployment boundary here: a failure in a weekly planning job does not require the conversational service to restart. The components still share a database and domain code, so they are not independently evolving microservices. They are separate runtimes for a small application.</p><h3>Give the data somewhere to live before asking questions about it</h3><p>The ingestion pipeline uses three PostgreSQL schemas.</p><p><strong>bronze preserves the raw Garmin responses.</strong> A row is identified by an endpoint and a date. The payload is JSONB, accompanied by a hash.</p><p><strong>silver gives those responses typed structure and useful meaning.</strong> Activities, sleep, heart rate variability, and other measurements become tables and views that downstream code can query consistently.</p><p><strong>app holds the product&#39;s state.</strong> Plans, planned sessions, briefings, check-ins, conversations, memories, rules, and execution ledgers belong here.</p><p>The raw upsert is small enough to show in full:</p><pre>INSERT INTO bronze.raw (endpoint, day, payload, payload_hash, fetched_at)<br>VALUES (%(endpoint)s, %(day)s, %(payload)s::jsonb, %(hash)s, now())<br>ON CONFLICT (endpoint, day) DO UPDATE<br>  SET payload = EXCLUDED.payload,<br>      payload_hash = EXCLUDED.payload_hash,<br>      fetched_at = now()<br>  WHERE bronze.raw.payload_hash IS DISTINCT FROM EXCLUDED.payload_hash</pre><p>The Python code computes the hash from JSON serialized with sorted keys. Re-fetching an unchanged payload avoids rewriting the row. A changed response updates the stored payload. This uses PostgreSQL’s <a href="https://www.postgresql.org/docs/current/sql-insert.html">conditional conflict handling</a>.</p><p>There are two limits worth making explicit. Bronze retains the latest payload for each key; it is not an append-only history of every response. And fetched_at changes only when the payload changes, so it cannot double as a timestamp for the last successful polling attempt.</p><p>The normal ingestion range includes yesterday and today, allowing a later run to pick up data that arrived after an earlier synchronization. Historical backfills use the same pipeline. Current capability snapshots, such as personal records, are fetched separately: copying today’s snapshot into every date of a backfill would manufacture a history that never existed.</p><p>Preserving raw payloads paid off when I needed to change what a metric meant.</p><figure><img alt="Raw Garmin payloads are stored in bronze, transformed into typed silver data, and exposed through shared views. Body Battery samples and a recorded sleep end define distinct midnight, wake-time, and latest values for the briefing and chat tools." src="https://cdn-images-1.medium.com/max/1024/1*HQLmjn5X8uEuC6-sav3X1Q.png" /><figcaption>Raw payloads make corrections reproducible. A shared semantic view gives every consumer the same definition of a measurement. Diagram by the author.</figcaption></figure><h3>A plausible number can still be the wrong fact</h3><p>One of the most instructive bugs involved Garmin’s Body Battery data.</p><p>The application had daily aggregates such as the starting, ending, minimum, and maximum level. Those values looked sufficient for a morning briefing. But the first reading of a calendar day is around midnight. It does not tell you the level when the athlete woke up. The maximum can also occur at another time of day.</p><p>The system could retrieve a valid number, label it “at wake-up,” and build an explanation around the wrong event. A better prompt would still be working with an ambiguous field.</p><p>The correction changed the data model. Intraday samples became timestamped rows. Sleep records retained their end time. A shared view now exposes distinct fields:</p><ul><li>midnight_level identifies the beginning-of-day aggregate.</li><li>wake_level and wake_at identify the sample nearest the recorded wake time.</li><li>last_level and last_at identify the most recent stored sample.</li></ul><p>Without a recorded wake time, the wake-specific value is NULL. The nearest sample is also an approximation tied to an explicit timestamp, rather than a claim that the device measured at the exact instant of waking.</p><p>Both the briefing and the chat tools read that view. Existing raw payloads supplied the backfill.</p><p>This is the kind of work I now look for early in an AI application: which fields force the model to guess what a number represents? Making that meaning explicit improves every consumer of the data.</p><h3>The morning verdict is ordinary, testable Python</h3><p>The daily readiness verdict comes from a deterministic function. It receives a feature dictionary and a threshold configuration, then returns a color, a penalty total, reasons, and a prediction.</p><p>The inputs include sleep, heart rate variability, resting heart rate, Body Battery, and stress, compared with personal baselines. Missing signals are skipped. The thresholds are configurable, and the reasons are produced by the same branches that calculate the score.</p><p>That makes the behavior reproducible: the same inputs and configuration produce the same verdict. I can test a threshold boundary without calling a model, and a model-provider change cannot silently alter that calculation.</p><p>These remain personal heuristics. Determinism makes the implementation inspectable; it does not establish that a threshold is clinically valid or that a training recommendation will improve an outcome.</p><p>Language models enter where the application needs to interpret a request, explain context, extract structure, or draft a plan. The calculations and the generated narrative have different owners.</p><h3>A Telegram message has a lifecycle</h3><p>The Telegram integration is a FastAPI webhook. It checks the webhook secret and the allowed chat, claims the incoming update ID in PostgreSQL, and passes accepted messages to ChatService. Voice messages are transcribed before entering the same conversation path. Telegram documents the webhook secret header and delivery behavior in its <a href="https://core.telegram.org/bots/api#setwebhook">Bot API</a>.</p><figure><img alt="A Telegram message passes through webhook checks and update deduplication. ChatService loads recent messages and current application state, invokes the agent with tools, stores the exchange, and returns a reply through Telegram." src="https://cdn-images-1.medium.com/max/1024/1*FO2RqDoAHYuU9T8yUXFPeg.png" /><figcaption>One chat turn crosses several state boundaries. History persistence is best-effort, and sending a reply is a separate external operation. Diagram by the author.</figcaption></figure><p>ChatService loads recent conversation history and constructs a current-state block: the local date and time, the stored weekly plan, today&#39;s briefing, and recent execution context. It then runs the agent with access to tools.</p><p>Those tools have different responsibilities. Fixed warehouse queries answer common questions. A more flexible analytics tool accepts SQL, subject to validation, a read-only transaction, an eight-second statement timeout, and a maximum of 200 returned rows. Garmin tools provide on-demand access to data available in Garmin Connect. Domain tools manage application concepts such as check-ins and planned sessions.</p><p>An on-demand Garmin query still depends on the watch having synchronized. Likewise, the warehouse’s most recent sample may be hours old. The tool responses include timing information so the model can describe the data it actually has.</p><p>Keeping orchestration in ChatService also keeps model concerns out of the HTTP handler. Tests can replace the chat service without constructing a real agent or calling an external provider.</p><h3>Memory started with remembering the previous reply</h3><p>An early version could propose a change and then fail to remember its own proposal on the next request. Each HTTP request created another model run, and the conversation had not been restored.</p><p>The first useful memory feature was therefore straightforward persistence.</p><p>The application stores role and content as plain database fields. Before a model run, it loads up to 30 stored messages from the previous 48 hours and rebuilds them as PydanticAI messages. Both user and assistant messages count toward that limit; it is not 30 full exchanges.</p><p>Keeping the stored representation independent of the framework’s internal message format gives the database schema a longer life than any particular agent-library version.</p><p>The current-state block is rebuilt for each turn. Only the original user message and the resulting assistant text enter conversation history, so old copies of “today’s plan” do not accumulate as if they were still current.</p><p>Longer-term memory adds embeddings, retrieval of older conversations, and a weekly consolidation workflow. That workflow reads unconsumed check-ins, memories, and bounded excerpts of chat, then produces at most three durable memory entries. It inserts those entries and marks their sources consumed within a database transaction.</p><figure><img alt="Each chat turn combines up to 30 recent messages from the previous 48 hours with current application state. Older conversations are retrieved through a tool. Weekly consolidation creates up to three durable memories and marks their sources consumed in one database transaction." src="https://cdn-images-1.medium.com/max/1024/1*0BUG6UrJilzvmPna0fjlGQ.png" /><figcaption>Conversation history, current state, and durable memories have different lifecycles. Rule proposals require approval before the planner uses them. Diagram by the author.</figcaption></figure><p>A reflection step can propose rules based on accumulated context. Rules have separate proposed, approved, and rejected states; planning reads the approved set. This is a lifecycle for application state. It does not mean the model has retrained itself or established causality from one person&#39;s history.</p><h3>A weekly plan is a typed object with consequences</h3><p>The planner gathers warehouse context, invokes a model, and receives a structured WeekPlan. PydanticAI&#39;s <a href="https://pydantic.dev/docs/ai/core-concepts/output/">structured output support</a> lets the application receive a validated object instead of parsing a paragraph.</p><p>Here is the actual output model, with its docstring omitted:</p><pre>Bullet = Annotated[str, Field(max_length=180)]<br> <br>class WeekPlan(BaseModel):<br>    week_start: date<br>    sessions: list[Session] = Field(min_length=7, max_length=14)<br>    headline: str = Field(max_length=160)<br>    why: list[Bullet] = Field(min_length=2, max_length=4)<br>    targets: list[Bullet] = Field(min_length=2, max_length=4)</pre><p>This is an excerpt, not a standalone program: imports and the Session model are omitted. Each session carries fields such as date, modality, intensity, duration, title, and rationale, with additional domain validation.</p><p>The length limits came from a delivery problem. A free-form weekly opening grew into a long dump of evidence, and messages exceeded what the Telegram path could deliver. Short structured fields give the renderer control over presentation. The application also splits long messages and falls back to plain text when Telegram rejects the markup.</p><p>Schema validation is only the first check. The planner evaluates domain constraints, including session interference and the running protocol, and requests one regeneration if it finds violations. Running sessions that remain outside the protocol are removed after that retry.</p><p>The implementation has a limitation here: some remaining non-running violations are reported rather than blocking the entire plan. A typed response should not be mistaken for universal enforcement of every planning rule.</p><p>The distinction matters because the next step has side effects.</p><h3>The application owns workout synchronization</h3><p>The chat agent’s direct Garmin toolset is restricted to reads. Supported workout writes go through application workflows that construct the Garmin payload and call the upload and scheduling tools.</p><p>This boundary followed a concrete mistake. Giving the model a generic upload_workout tool gave it the ability to submit arbitrary workout JSON. Removing convenience builders did not remove that capability. An upload could bypass the application records that later workflows relied on.</p><p>The current synchronization code builds payloads from deterministic templates. A batch uses one MCP subprocess session and one Garmin login for multiple operations. The third-party <a href="https://github.com/Taxuspt/garmin_mcp/tree/2974244bfda1595b00836b3f942f579ec2d6f7d6">garmin_mcp project</a> supplies the Garmin tools; the application supplies the workflow around them.</p><p>Bike and swim sessions can enter the weekly synchronization batch. Running follows a separate path: a protocol-qualified session starts as proposed, requires explicit confirmation through the application flow, and is checked again before synchronization. Pending synchronization is represented in stored state so reconciliation has something to retry.</p><p>The weekly planner still contains an important distributed-systems compromise: it performs the external synchronization batch before inserting the replacement session rows. A database rollback cannot undo a workout that Garmin has already accepted.</p><p>That ordering is part of the current implementation, not an idealized transaction diagram. Moving toward durable operation records and explicit reconciliation would make failures between those steps easier to recover from.</p><figure><img alt="Weekly planning validates a WeekPlan, synchronizes eligible bike and swim workouts, replaces stored sessions, and sends a summary. Running proposals require later athlete confirmation and a protocol check; synchronization succeeds as planned or remains confirmed_pending_sync for revalidation and retry." src="https://cdn-images-1.medium.com/max/1024/1*arnY94izCpPEydCLzo_2rg.png" /><figcaption>The weekly batch and running confirmation are separate workflows. The diagram preserves the current ordering of external writes and database updates. Diagram by the author.</figcaption></figure><h3>Retries exposed the boundaries of the design</h3><p>One planning run persisted its work and then failed while sending the summary. Retrying generated another plan and duplicated sessions and workouts. The repair added cleanup of the previous week’s generated state before replacing it, along with tracking for workouts whose deletion failed.</p><p>The broader lesson was that “retry the job” was too coarse. Fetching data, generating a new plan, synchronizing a workout, and sending a message have different replay semantics.</p><p>The morning briefing uses a simpler approach. Several scheduled attempts check whether the required sleep data has arrived. An attempt exits if the day has already been handled. A final attempt records skipped_no_sleep if the window closes without the data, with a best-effort notice to the athlete.</p><p>Those ledger rows are useful for deduplication and operations, but they do not create end-to-end exactly-once delivery. Sending a Telegram message and committing a PostgreSQL row are separate operations. A crash between them can still produce a gap or a duplicate, depending on the path. The webhook has the converse concern: it claims an update before processing it, so a crash after the claim can prevent a retried update from being processed again.</p><p>For an engineering article, that distinction is worth keeping visible. A unique key protects a database operation. External side effects need their own recovery strategy.</p><figure><img alt="A briefing can be duplicated if the process crashes after Telegram accepts it but before the sent ledger is committed. A webhook reply can be lost if the process crashes after claiming the update ID; a redelivered update is then skipped as a duplicate." src="https://cdn-images-1.medium.com/max/1024/1*bB90VfuNAXhMQ8nQwoxJHQ.png" /><figcaption>The crash window determines the retry outcome. Committing a deduplication record and completing an external side effect are separate events. Diagram by the author.</figcaption></figure><h3>Make failures visible outside the failing channel</h3><p>The application records traces for model runs, logs workflow failures, and maintains ledgers that show whether scheduled work ran. Ingestion also checks for missing recent ledger activity and Telegram webhook problems.</p><p>There is a difference between “the job ran and had no usable sleep data” and “the job never ran.” A skipped_no_sleep record proves the first. No recent execution record points toward the second.</p><p>I also added cloud monitoring alerts through email. Depending entirely on Telegram for operational alerts created a circular dependency: when the Telegram integration broke, it could silence the message explaining the breakage.</p><p>Deployment follows a similarly explicit path. A push triggers image builds for affected components. Images receive content-derived tags, and deployment resolves an image digest before updating Cloud Run. CI runs linting, tests, coverage gates, and Terraform checks. Shared-code changes rebuild the consumers that package that code.</p><p>The system includes an adaptive engine as well, but it starts in an observation mode: it records evaluations and decisions for comparison. The implemented promotion path tops out at suggestions, with eligibility checks and an explicit approval step. It is not an unrestricted feedback loop rewriting the plan.</p><h3>The version I would build first</h3><p>The repository grew to include capabilities beyond the core loop, including nutrition-document workflows and a Notion cockpit. Those integrations are useful extensions, but the core product can be evaluated with a much smaller slice.</p><p>If I were rebuilding it, I would start with raw ingestion, a handful of well-defined views, one deterministic briefing, and a Telegram chat that remembers the previous exchange. Then I would add a persisted weekly plan and compare it with actual activities before enabling workout writes.</p><p>That sequence produces useful checkpoints. Can I replay a transform? Can I explain the timestamp behind a metric? Does the second message remember the first? Can I distinguish a proposed change from one that was applied? Can I recover when an API accepts a request but the process dies before recording the result?</p><p>Garmin Coach made those questions concrete. The language model gave me a flexible interface to the system. Durable records, explicit meanings, constrained workflows, and recovery paths made that interface worth returning to.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f82b0405775" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Converse com seu Garmin: MCP no Claude Code e no Codex]]></title>
            <link>https://matheusjerico.medium.com/converse-com-seu-garmin-mcp-no-claude-code-e-no-codex-2272c9eecea0?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/2272c9eecea0</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[garmin]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 20:01:05 GMT</pubDate>
            <atom:updated>2026-09-04T20:01:05.052Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Da consulta de sono e atividades ao deploy no Cloud Run: como construí a integração do Garmin Coach, como reproduzir a conexão e o que ainda falta para usá-la no ChatGPT.</em></p><figure><img alt="Ilustração de um relógio esportivo e um celular conectados a balões de conversa por elementos abstratos de dados." src="https://cdn-images-1.medium.com/max/1024/1*POO8Dwf3lllOXRGiNO8P_A.png" /><figcaption>Ilustração conceitual criada com IA para este artigo; não representa uma tela ou métricas reais.</figcaption></figure><p>O Garmin registra o treino, o sono, a frequência cardíaca e vários sinais de recuperação. Depois de algumas semanas, você já tem material para perguntas interessantes:</p><blockquote><em>Como meu sono mudou nas semanas em que aumentei o volume de treino?</em></blockquote><blockquote><em>Minhas últimas atividades foram realmente leves ou passei mais tempo em intensidade alta do que imaginava?</em></blockquote><blockquote><em>O que mudou no meu histórico desde o último mês?</em></blockquote><p>Responder costuma exigir abrir telas diferentes, escolher períodos e cruzar informações. Eu queria fazer essas perguntas em uma conversa, deixando a IA consultar os dados necessários.</p><p>A peça que permite isso é um <strong>servidor MCP conectado ao Garmin Connect</strong>. Construí essa integração no Garmin Coach, um app pessoal que também tem ingestão de dados e planejamento de treinos. O conector MCP funciona como um componente independente: dá para consultar o Garmin sem instalar o restante do coach.</p><p>Neste artigo, parto do código desse componente para mostrar a conexão local no Claude Code e no Codex e um deploy remoto no Cloud Run. A experiência registrada em produção é com o conector do Claude. Ao final, explico o ajuste que ainda seria necessário para conectar o ChatGPT; ele não está implementado no proxy apresentado aqui.</p><p>Você pode seguir apenas a parte local e já começar a consultar seus dados. O deploy na nuvem é uma segunda etapa, útil para acessar a integração sem manter seu computador ligado.</p><h3>O que muda quando a IA pode consultar seu Garmin</h3><p>Sem a integração, o modelo responde com o conhecimento geral que possui e com as informações que você colocou na conversa. Com a integração, ele pode buscar atividades e métricas da sua conta para fundamentar a resposta.</p><p>O MCP, ou <em>Model Context Protocol</em>, padroniza essa comunicação. O servidor apresenta ferramentas com nomes, descrições e parâmetros. O cliente de IA escolhe uma ferramenta, faz a chamada e recebe o resultado.</p><p>O fluxo fica assim:</p><figure><img alt="O relógio sincroniza com Garmin Connect. Depois, o cliente de IA consulta o servidor MCP, que busca os dados no Garmin Connect e devolve a resposta." src="https://cdn-images-1.medium.com/max/1024/1*Wm7aKRk054IThZGoBcllfw.png" /><figcaption>Do relógio ao chat: a consulta usa os dados já sincronizados no Garmin Connect. Diagrama do autor.</figcaption></figure><p>Quem interpreta os dados é o modelo do cliente que você está usando. O servidor MCP faz a ponte com o Garmin; ele não precisa de uma chave da OpenAI ou da Anthropic para executar essa função. No app, esse caminho também não passa pelo banco do coach, pelo Telegram nem pelo pipeline de ingestão.</p><p>Também vale ajustar o significado de “tempo real”: a consulta acontece quando a ferramenta é chamada, sobre os dados disponíveis no <strong>Garmin Connect</strong>. O relógio precisa ter sincronizado. Essa integração não transmite seus batimentos diretamente do pulso, e algumas ferramentas retornam resumos em vez de todas as amostras.</p><p>A disponibilidade de cada métrica depende do dispositivo, dos registros da conta e das ferramentas implementadas. Por isso, uma boa análise também precisa dizer quando faltam dados.</p><h3>O que vamos usar</h3><p>A base é o projeto aberto <a href="https://github.com/Taxuspt/garmin_mcp/tree/2974244bfda1595b00836b3f942f579ec2d6f7d6">garmin_mcp, de Taxuspt</a>, que expõe consultas e operações do Garmin Connect como ferramentas MCP. Trata-se de uma integração de terceiros, sem garantia de estabilidade da Garmin.</p><p>Vamos usar duas formas de executá-lo:</p><p><strong>Local, por </strong><strong>stdio:</strong> o servidor roda no seu computador, iniciado pelo Claude Code ou pelo Codex. É o caminho para começar.</p><p><strong>Remoto, por HTTPS:</strong> o servidor roda em um container no Cloud Run. É o caminho do conector Claude utilizado no app, acessível sem manter seu computador ligado.</p><p>Na versão remota, um proxy FastMCP adiciona login Google e controle de acesso. Cada instalação usa os tokens de <strong>uma conta Garmin</strong>. Adicionar outro e-mail à lista de acesso permite que aquela pessoa consulte essa mesma conta; não conecta automaticamente o Garmin dela.</p><p>Para reproduzir com seus dados, use seus próprios tokens e sua própria instalação. O código do proxy e o Dockerfile no apêndice são os mesmos do Garmin Coach. O Terraform foi reduzido para uma instalação independente; essa adaptação está identificada na seção de deploy.</p><h3>Primeiro: conecte sua conta Garmin</h3><p>Os comandos abaixo usam um terminal Bash ou Zsh, como no macOS ou Linux; no Windows, você pode usar WSL. Tenha Git, o <a href="https://docs.astral.sh/uv/getting-started/installation/">uv instalado</a> e o Claude Code ou Codex já funcionando.</p><p>O uvx executa programas Python em um ambiente gerenciado. Para manter os exemplos na mesma revisão de código, vamos fixar a versão do servidor por commit:</p><pre>GARMIN_MCP_SOURCE=&quot;git+https://github.com/Taxuspt/garmin_mcp@2974244bfda1595b00836b3f942f579ec2d6f7d6&quot;<br> <br>uvx --python 3.12 --from &quot;$GARMIN_MCP_SOURCE&quot; garmin-mcp-auth</pre><p>Siga o login no terminal, incluindo o código de autenticação, se solicitado. O processo salva tokens em ~/.garminconnect. Você não precisa colocar sua senha Garmin no chat nem na configuração do cliente. O fluxo está implementado no <a href="https://github.com/Taxuspt/garmin_mcp/blob/2974244bfda1595b00836b3f942f579ec2d6f7d6/src/garmin_mcp/auth_cli.py">comando de autenticação do projeto</a>.</p><p>Esse diretório contém credenciais de acesso. Mantenha-o fora de repositórios, imagens Docker e arquivos que você pretende compartilhar.</p><p>Para conferir a autenticação:</p><pre>uvx --python 3.12 --from &quot;$GARMIN_MCP_SOURCE&quot; garmin-mcp-auth --verify</pre><p>Vamos usar as mesmas 16 ferramentas de leitura declaradas para o conector remoto do Garmin Coach:</p><pre>GARMIN_READ_TOOLS=&quot;get_activities,get_activity,get_sleep_summary,get_sleep_data,get_stress_summary,get_body_battery,get_stats,get_rhr_day,get_heart_rates_summary,get_training_readiness,get_morning_training_readiness,get_hrv_data,get_hrv_trend,get_training_status,get_training_load_trend,get_vo2max_trend&quot;</pre><p>A variável GARMIN_ENABLED_TOOLS restringe as ferramentas registradas pelo servidor. Essa lista cobre atividades, sono, estresse, Body Battery, frequência cardíaca, HRV e indicadores de treinamento. Prefira get_sleep_summary nas primeiras consultas; reserve get_sleep_data para investigar detalhes. Não a deixe vazia para representar “nenhuma ferramenta”: no servidor utilizado aqui, a ausência de filtro expõe o conjunto completo. Esse comportamento está no <a href="https://github.com/Taxuspt/garmin_mcp/blob/2974244bfda1595b00836b3f942f579ec2d6f7d6/src/garmin_mcp/__init__.py">registro de ferramentas do servidor</a>.</p><p>Mantenha esse terminal aberto para os próximos comandos. Se abrir outro, defina novamente as duas variáveis acima.</p><h3>Conectando ao Claude Code</h3><p>Execute:</p><pre>claude mcp add --scope user --transport stdio \<br>  --env &quot;GARMIN_MCP_TRANSPORT=stdio&quot; \<br>  --env &quot;GARMINTOKENS=$HOME/.garminconnect&quot; \<br>  --env &quot;GARMIN_ENABLED_TOOLS=$GARMIN_READ_TOOLS&quot; \<br>  garmin -- \<br>  uvx --python 3.12 --from &quot;$GARMIN_MCP_SOURCE&quot; garmin-mcp</pre><p>Abra uma nova sessão do Claude Code e use /mcp para verificar a conexão. O escopo user torna essa configuração disponível para seu usuário nos diferentes projetos da máquina. O comando e o fluxo de autenticação dos servidores remotos estão descritos na <a href="https://code.claude.com/docs/en/mcp">documentação do Claude Code</a>.</p><h3>Conectando ao Codex</h3><p>No mesmo terminal:</p><pre>codex mcp add garmin \<br>  --env &quot;GARMIN_MCP_TRANSPORT=stdio&quot; \<br>  --env &quot;GARMINTOKENS=$HOME/.garminconnect&quot; \<br>  --env &quot;GARMIN_ENABLED_TOOLS=$GARMIN_READ_TOOLS&quot; \<br>  -- uvx --python 3.12 --from &quot;$GARMIN_MCP_SOURCE&quot; garmin-mcp</pre><p>Abra uma nova sessão e confira /mcp. O Codex suporta servidores locais e HTTP com OAuth; a configuração fica, por padrão, em ~/.codex/config.toml. Se a primeira inicialização demorar para preparar dependências, é possível aumentar startup_timeout_sec na entrada do servidor. Veja a <a href="https://learn.chatgpt.com/docs/extend/mcp?surface=cli">documentação oficial de MCP</a>.</p><p>Configurar o Codex localmente não registra automaticamente um conector no ChatGPT web. São clientes e configurações diferentes. Os comandos dos CLIs foram conferidos, mas a evidência de uso em produção deste projeto é do conector Claude, não de uma sessão Codex.</p><h3>Sua primeira análise</h3><p>Comece com algo simples de conferir no aplicativo Garmin:</p><blockquote><em>Consulte minhas três atividades mais recentes no Garmin. Mostre data, modalidade, duração e distância, quando disponíveis. Informe as unidades e não preencha campos ausentes por estimativa.</em></blockquote><p>Compare o resultado com o Garmin Connect. Esse primeiro teste confirma que a ferramenta está consultando a conta e que os dados chegaram corretamente à conversa.</p><p>Depois, avance para perguntas que cruzam períodos:</p><blockquote><em>Consulte meu sono e minha HRV nos últimos sete dias. Mostre os valores por data, identifique dias sem informação e compare com o período anterior, se os dados estiverem disponíveis. Diferencie observação de hipótese.</em></blockquote><p>Ou para uma atividade específica:</p><blockquote><em>Analise minha última atividade de bicicleta. Primeiro apresente os dados retornados pela ferramenta. Depois explique o que eles permitem concluir sobre duração, intensidade e consistência do esforço. Diga também o que não é possível avaliar com esse retorno.</em></blockquote><p>Esses são exemplos de perguntas, não resultados de uma execução. A resposta muda conforme seu histórico e as informações que o Garmin disponibiliza.</p><p>Para comparações maiores, peça que o cliente organize os dados em uma tabela e calcule os agregados com uma ferramenta de cálculo disponível. Isso ajuda a tornar a análise verificável. A existência do MCP não garante, por si só, que uma média, uma conversão de unidade ou uma interpretação esteja correta.</p><h3>Dos dados para uma proposta de treino</h3><p>O histórico também pode dar contexto ao planejamento. Você pode informar objetivo, disponibilidade, modalidades e restrições, e pedir uma proposta:</p><blockquote><em>Consulte minhas atividades recentes e os dados de recuperação disponíveis. Considere o objetivo e a disponibilidade que informei nesta conversa. Proponha uma semana de treino, explique quais dados sustentam cada escolha e liste o que você ainda precisa saber. Apresente a proposta em uma tabela, sem criar ou agendar treinos.</em></blockquote><p>A configuração deste artigo permite consultar os dados e usá-los para elaborar essa proposta. <strong>Ela não publica treinos no Garmin.</strong> O modelo do cliente produz a análise; o MCP fornece as informações.</p><p>No Garmin Coach, o planejamento e o envio ao relógio são outra parte da arquitetura. O chat também tem acesso direto apenas a consultas Garmin. Quando o app precisa enviar um treino, um workflow abre sua própria sessão MCP local, monta o conteúdo com templates e chama upload_workout, seguido de schedule_workout.</p><p>Para corrida, há regras adicionais no código: protocolo versionado, proposta registrada, confirmação explícita do atleta e nova validação antes da sincronização. Se o envio falhar, a aprovação fica registrada como pendente para o reconciliador tentar novamente. Esse comportamento pertence ao app; instalar o conector em outro cliente não transfere essas regras para ele.</p><p>Essa separação veio de uma falha concreta de desenho: liberar upload_workout ao modelo permite enviar um JSON arbitrário, inclusive de uma modalidade que deveria passar por validação. Retirar apenas os builders create_* não retira a capacidade de escrita. Por isso, o tutorial mantém a mesma fronteira de leitura do conector do projeto.</p><p>Para quem usa Garmin, o resultado imediato é poder discutir um planejamento com seu histórico à mão. Uma proposta gerada pela IA pode apoiar a conversa com seu treinador; métricas do relógio, sozinhas, não constituem liberação médica para lesão ou retorno ao exercício.</p><figure><img alt="No tutorial, o conector consulta métricas, analisa o histórico e apoia uma proposta de treino. No Garmin Coach, sessões são registradas, passam pelas regras do app e são enviadas por um workflow separado. Corrida exige protocolo, confirmação e revalidação." src="https://cdn-images-1.medium.com/max/1024/1*SRIQoMHsOx8LSljHd4dkOw.png" /><figcaption>Analisar e propor no chat não equivale a publicar um treino. A escrita pertence aos workflows do coach. Diagrama do autor.</figcaption></figure><h3>Quando vale levar o servidor para a nuvem</h3><p>No modo local, o cliente inicia o servidor na sua máquina. Para um cliente remoto acessar os dados com o computador desligado, precisamos de um endereço HTTPS.</p><p>A versão que vou mostrar usa:</p><ul><li><strong>Cloud Run</strong> para executar o container;</li><li><strong>Secret Manager</strong> para guardar os tokens Garmin e o segredo OAuth;</li><li><strong>Google OAuth</strong> para identificar quem está acessando;</li><li><strong>Firestore</strong> para preservar o estado da autenticação;</li><li><strong>Terraform</strong> para declarar os recursos.</li></ul><p>O fluxo remoto é:</p><figure><img alt="Cliente MCP acessa o proxy FastMCP no Cloud Run por HTTPS. Google OAuth identifica o usuário, Firestore guarda estado OAuth cifrado e Secret Manager fornece credenciais. O proxy usa garmin-mcp por stdio para consultar Garmin Connect." src="https://cdn-images-1.medium.com/max/1024/1*IGfHA8L579OqPvCu2zGHKA.png" /><figcaption>Arquitetura do mcp-server: autenticação no proxy, estado OAuth persistente e consultas diretas ao Garmin. Diagrama baseado no código do Garmin Coach.</figcaption></figure><p>O login Google protege o acesso ao servidor. O login Garmin feito no terminal permite que o servidor consulte sua conta. São autenticações distintas.</p><p>O servidor Garmin original não autentica seu endpoint HTTP. Por isso, neste desenho, ele roda como subprocesso stdio, atrás do proxy. O <a href="https://gofastmcp.com/integrations/google">GoogleProvider do FastMCP</a> cuida da ponte OAuth.</p><p>Também precisamos conservar o estado OAuth fora do container. Se registros de clientes e tokens ficarem apenas no disco efêmero do Cloud Run, uma nova instância pode perder esse estado. O código do app usa Firestore, valores cifrados, chaves de documento normalizadas por hash e campos de expiração compatíveis com TTL.</p><p>O MCP não exige um warehouse de dados de saúde: o Firestore deste tutorial guarda estado de autenticação. As métricas continuam sendo consultadas no Garmin. Os resultados das ferramentas, porém, são enviados ao cliente de IA para compor a resposta; considere as configurações de privacidade da conta que utilizar.</p><h3>O que aprendemos colocando o conector em produção</h3><p>Dois problemas do Garmin Coach explicam detalhes que poderiam parecer excessivos em um tutorial.</p><p>Primeiro, o estado OAuth ficava no disco da instância. Quando o Cloud Run descartava a instância, perdíamos os registros necessários para reconhecer clientes e tokens. A correção foi mover esse estado para um database Firestore nomeado, cifrar os valores e restringir o acesso da identidade do serviço àquele database.</p><p>Segundo, um client_id recebido do Claude podia ser uma URL. Usá-lo diretamente como identificador de documento no Firestore quebrava a autorização por causa das barras. O proxy passou a transformar todas as chaves em hashes determinísticos. Outro ajuste mantém expires_at como Timestamp, pois a expiração automática do Firestore não funciona sobre uma string de data.</p><p>O registro de validação da versão implantada em 24 de agosto de 2026 acompanhou chamadas autenticadas do Claude nos três dias seguintes, com renovação de tokens e sem um novo fluxo de autorização. Também confirmou que um cliente registrado continuava reconhecido após o serviço subir em uma nova instância. É essa experiência que sustenta o desenho apresentado aqui.</p><p>Persistir OAuth não elimina a demora de inicialização. Naquele acompanhamento, os dois cold starts medidos foram de 12,7 e 14,2 segundos, mesmo com o boost de CPU. Também houve respostas 400 e 404 do transporte, registradas no acompanhamento; a retomada observada pelo cliente não significa ausência de falhas nem resposta instantânea.</p><h3>Fazendo o deploy</h3><p>Esta etapa pressupõe um <strong>projeto Google Cloud próprio, com faturamento habilitado</strong>, Google Cloud CLI, Terraform e Docker com Buildx. Sua identidade de provisionamento precisa poder habilitar APIs, criar os recursos e configurar IAM. A identidade usada pelo container terá permissões menores.</p><p>Há cobrança potencial de execução, armazenamento de imagens, segredos e Firestore. Escalar a zero reduz o tempo ocioso de computação, mas não torna toda a solução gratuita.</p><p>No Garmin Coach, o Cloud Build constrói a imagem após mudanças no repositório. A tag deriva do conteúdo dos arquivos, e o Terraform resolve a imagem publicada para um digest imutável antes de atualizar o Cloud Run. O pipeline de deploy espera a imagem existir; ele não faz o build durante o apply.</p><p>Para reproduzir somente o conector, crie uma pasta nova e copie os quatro arquivos do apêndice: proxy.py, Dockerfile, .dockerignore e main.tf. <strong>O proxy e o Dockerfile são cópias do app; o Terraform é uma adaptação para instalação independente.</strong> Ela preserva OAuth, lista de e-mails, ferramentas de leitura, segredos e Firestore, usando build local e duas etapas de provisionamento no lugar do pipeline completo. Foi validada estaticamente, mas não implantada em um projeto novo nesta revisão.</p><p>Autentique as ferramentas:</p><pre>gcloud auth login<br>gcloud auth application-default login</pre><p>Crie terraform.tfvars na pasta do tutorial:</p><pre>project_id             = &quot;SEU_PROJECT_ID&quot;<br>project_number         = &quot;SEU_PROJECT_NUMBER&quot;<br>region                 = &quot;us-central1&quot;<br>allowed_emails         = [&quot;voce@example.com&quot;]<br>google_oauth_client_id = &quot;&quot;<br>deploy_service         = false</pre><p>Você encontra o número do projeto no console ou com:</p><pre>gcloud projects describe SEU_PROJECT_ID --format=&#39;value(projectNumber)&#39;</pre><p>Inicialize e crie a infraestrutura de apoio:</p><pre>terraform init<br>terraform plan -out=bootstrap.tfplan<br>terraform apply bootstrap.tfplan<br>terraform output -raw google_redirect_uri</pre><p>O serviço ainda não será criado. Essa separação permite cadastrar os segredos e publicar a imagem antes de iniciar o container.</p><p>No Google Cloud, configure a tela de consentimento OAuth e crie um cliente do tipo <strong>Web application</strong>. Se usar o modo de testes, inclua sua conta entre os usuários de teste. Cadastre como redirect URI o endereço impresso pelo Terraform, terminado em /auth/callback.</p><p>A URL usa o formato determinístico do Cloud Run, com nome curto de serviço, número do projeto e região, conforme a <a href="https://docs.cloud.google.com/run/docs/triggering/https-request">documentação do Google Cloud</a>.</p><p>Coloque o client ID em google_oauth_client_id. Salve o client secret no Secret Manager; não o coloque no .tfvars, pois valores gerenciados por Terraform podem chegar ao arquivo de estado. Faça a inclusão interativa sem imprimir o segredo:</p><pre>python3 -c &#39;<br>import getpass<br>import subprocess<br> <br>project = input(&quot;Project ID: &quot;).strip()<br>secret = getpass.getpass(&quot;Google OAuth client secret: &quot;)<br>subprocess.run(<br>    [&quot;gcloud&quot;, &quot;secrets&quot;, &quot;versions&quot;, &quot;add&quot;, &quot;garmin-mcp-oauth-client-secret&quot;,<br>     &quot;--project&quot;, project, &quot;--data-file=-&quot;],<br>    input=secret.encode(),<br>    check=True,<br>)<br>&#39;</pre><p>Envie os tokens Garmin já obtidos na etapa local:</p><pre>gcloud secrets versions add garmin-tokens \<br>  --project=SEU_PROJECT_ID \<br>  --data-file=&quot;$HOME/.garminconnect/garmin_tokens.json&quot;</pre><p>Publique a imagem com as credenciais da sua identidade de provisionamento:</p><pre>gcloud auth configure-docker us-central1-docker.pkg.dev<br> <br>GARMIN_IMAGE_REPOSITORY=&quot;$(terraform output -raw image_repository)&quot;<br>GARMIN_IMAGE_TAG=&quot;$GARMIN_IMAGE_REPOSITORY:v1&quot;<br> <br>docker buildx build --platform linux/amd64 \<br>  --tag &quot;$GARMIN_IMAGE_TAG&quot; --push .<br> <br>GARMIN_IMAGE_DIGEST=&quot;$(gcloud artifacts docker images describe &quot;$GARMIN_IMAGE_TAG&quot; \<br>  --project=SEU_PROJECT_ID --format=&#39;value(image_summary.digest)&#39;)&quot;<br>printf &#39;%s@%s\n&#39; &quot;$GARMIN_IMAGE_REPOSITORY&quot; &quot;$GARMIN_IMAGE_DIGEST&quot;</pre><p>Se escolheu outra região, ajuste também o hostname passado a configure-docker.</p><p>No .tfvars, copie a referência com @sha256:... impressa acima e altere <strong>de forma persistente</strong> deploy_service para true. Substitua DIGEST_IMPRESSO pelo hash completo, sem repetir o prefixo sha256::</p><pre>image          = &quot;us-central1-docker.pkg.dev/SEU_PROJECT_ID/garmin-mcp/garmin-mcp@sha256:DIGEST_IMPRESSO&quot;<br>deploy_service = true</pre><p>Mantenha apenas uma atribuição para cada variável. Depois:</p><pre>terraform plan -out=deploy.tfplan<br>terraform apply deploy.tfplan<br>terraform output -raw connector_url</pre><p>O serviço aceita requisições públicas na camada IAM porque os clientes MCP não apresentam identidade IAM do Google Cloud. <strong>As ferramentas continuam protegidas pelo OAuth e pela lista de e-mails dentro do proxy.</strong></p><p>Para atualizações, publique uma nova tag, obtenha o novo digest, altere image no .tfvars e aplique o novo plano. Para renovar tokens, faça novamente a autenticação local e adicione uma versão ao segredo. Uma instância já em execução pode continuar usando credenciais carregadas anteriormente; altere também revision_marker no .tfvars para provocar uma nova revisão. Essa variável é uma conveniência da adaptação do tutorial; não existe no módulo original. Um terraform apply sem mudanças não reinicia o serviço.</p><h3>Conectando à versão remota</h3><p>Use outro nome para preservar a configuração local.</p><p>No Claude Code:</p><pre>claude mcp add --scope user --transport http \<br>  garmin-cloud https://SEU_SERVICO.run.app/mcp</pre><p>Abra /mcp, selecione o servidor e conclua o login Google no navegador.</p><p>No Codex:</p><pre>codex mcp add garmin-cloud --url https://SEU_SERVICO.run.app/mcp<br>codex mcp login garmin-cloud</pre><p>O client secret do Google pertence ao servidor. Ele não precisa ser copiado para esses comandos.</p><h3>E no ChatGPT?</h3><p>O ChatGPT também pode funcionar como cliente de um servidor MCP remoto, conforme a disponibilidade da conta e do workspace. Porém, há uma diferença entre suportar o protocolo e estar configurado para este servidor.</p><p><strong>O proxy atual do Garmin Coach não inclui callbacks do ChatGPT.</strong> Ele permite https://claude.ai/api/mcp/auth_callback e callbacks locais em localhost e 127.0.0.1. Portanto, cadastrar a URL /mcp no ChatGPT não basta para reproduzir uma conexão autenticada com o código deste artigo.</p><p>Para acrescentar esse cliente, seria necessário obter o redirect URI exato na configuração da conexão, adicioná-lo a allowed_client_redirect_uris, testar o fluxo OAuth, publicar a nova imagem e validar uma consulta real. O callback do Google continuaria sendo /auth/callback no servidor; o callback permitido no proxy é o retorno ao cliente de IA.</p><p>A documentação oficial descreve <a href="https://developers.openai.com/plugins/deploy/connect-chatgpt">como conectar um servidor ao ChatGPT</a> e os <a href="https://developers.openai.com/plugins/build/auth">formatos de callback OAuth</a>. Essa extensão fica identificada como próximo passo: o repositório ainda não traz sua implementação nem evidência de execução.</p><h3>Como saber se a integração está funcionando</h3><p>Teste em camadas:</p><ol><li><strong>Garmin:</strong> garmin-mcp-auth --verify confirma a autenticação upstream.</li><li><strong>OAuth:</strong> uma requisição sem token ao endpoint deve receber uma resposta de autenticação, enquanto a descoberta OAuth deve estar acessível.</li><li><strong>Dados:</strong> uma chamada autenticada deve retornar uma atividade que você consegue conferir no Garmin Connect.</li><li><strong>Persistência:</strong> após publicar uma nova revisão, confirme que uma conexão existente continua utilizável sem cadastrar tudo novamente. O cliente pode precisar reinicializar a sessão de transporte; isso é diferente de perder o cadastro OAuth.</li><li><strong>Ferramentas:</strong> confira se o cliente descobriu exatamente as 16 ferramentas da lista. Um endpoint autenticado com ferramentas extras ainda está configurado de forma diferente da pretendida.</li></ol><p>Para a segunda camada:</p><pre>GARMIN_BASE_URL=&quot;$(terraform output -raw base_url)&quot;<br>curl -i &quot;$GARMIN_BASE_URL/mcp&quot;<br>curl -fsS &quot;$GARMIN_BASE_URL/.well-known/oauth-protected-resource/mcp&quot;</pre><p>Um 401 sem token é esperado nessa primeira chamada. Ele não comprova que a consulta ao Garmin funciona; isso exige o terceiro teste.</p><p>No app, esses dois sinais também compõem o smoke test: acesso sem token recusado e descoberta OAuth disponível. A suíte do proxy cobre autorização por e-mail, cifragem, expiração, chaves em formato de URL e associação entre código OAuth e cliente. Em 4 de setembro de 2026, os 18 testes passaram com cobertura de linhas e ramificações de 100% do arquivo proxy.py. Isso valida esses contratos; não mede a qualidade de uma análise de treino.</p><p>Se o login funcionar e as ferramentas falharem, investigue tokens Garmin, métricas indisponíveis e limites da API. Se o login não terminar, confira o e-mail autorizado, os dois callbacks e o acesso do serviço ao Firestore e aos segredos. O <a href="https://modelcontextprotocol.io/docs/tools/inspector">MCP Inspector</a> também permite inspecionar e chamar ferramentas sem depender de uma conversa com o modelo.</p><h3>Uma interface nova para o histórico que você já tem</h3><p>O ganho aparece quando você consegue transformar uma dúvida em uma consulta verificável: escolher o período, recuperar os registros, comparar os resultados e fazer uma nova pergunta com esse contexto.</p><p>Comece por uma atividade. Confira os números. Depois explore o sono, a recuperação e a distribuição dos treinos. Quando chegar ao planejamento, peça uma proposta com dados, contexto e justificativas. Publicar treinos exige um fluxo de escrita separado, como o que construímos no coach.</p><p>O histórico já está sendo construído a cada sincronização. O MCP acrescenta uma maneira de conversar com ele.</p><h3>Apêndice: os arquivos da versão remota</h3><p>O proxy e o Dockerfile abaixo reproduzem os arquivos do Garmin Coach. O main.tf foi adaptado para criar apenas o conector em outro projeto; .dockerignore restringe o contexto do build. O conjunto mantém o endpoint em leitura. Não inclui os workflows de planejamento e escrita, o pipeline de CI/CD nem os alertas do restante do app.</p><p>Use uma pasta exclusiva para estes arquivos. Mantenha terraform.tfvars, *.tfstate*, *.tfplan, .terraform/ e credenciais fora do Git. Guarde o estado Terraform: ele permite administrar os recursos criados. Em equipe, configure um backend remoto com controle de acesso.</p><h3>proxy.py</h3><pre>&quot;&quot;&quot;OAuth-authenticated MCP proxy in front of garmin_mcp (stdio subprocess).<br> <br>Claude connectors (mobile/web/desktop) speak MCP OAuth; garmin_mcp has no auth<br>of its own. This proxy terminates OAuth (Google IdP, single-user email<br>allowlist) and forwards MCP traffic to the garmin-mcp subprocess over stdio,<br>so the unauthenticated server never touches the network.<br> <br>The OAuth state (DCR clients, authorization codes, consent transactions, JTI<br>mappings, upstream tokens) lives in a named Firestore database when<br>FIRESTORE_DATABASE is set. FastMCP&#39;s default is a file store inside the<br>instance, which Cloud Run discards on every cold start and redeploy — every<br>issued token died with it.<br>&quot;&quot;&quot;<br> <br>import os<br>from collections.abc import Callable, Mapping<br>from typing import Any<br> <br>from cryptography.fernet import Fernet<br>from fastmcp.client.transports import StdioTransport<br>from fastmcp.exceptions import ToolError<br>from fastmcp.server import create_proxy<br>from fastmcp.server.auth.jwt_issuer import derive_jwt_key<br>from fastmcp.server.auth.providers.google import GoogleProvider<br>from fastmcp.server.dependencies import get_access_token<br>from fastmcp.server.middleware import Middleware, MiddlewareContext<br>from key_value.aio._utils.sanitization import AlwaysHashStrategy<br>from key_value.aio._utils.serialization import BasicSerializationAdapter<br>from key_value.aio.protocols import AsyncKeyValue<br>from key_value.aio.stores.firestore import FirestoreStore<br>from key_value.aio.wrappers.encryption import FernetEncryptionWrapper<br> <br>STORAGE_ENCRYPTION_SALT = &quot;fastmcp-storage-encryption-key&quot;<br>DEFAULT_FIRESTORE_DATABASE = &quot;(default)&quot;<br> <br> <br>class TimestampFirestoreStore(FirestoreStore):<br>    &quot;&quot;&quot;FirestoreStore that writes created_at/expires_at as Timestamps.<br> <br>    The upstream store serialises dates as ISO strings; Firestore TTL policies<br>    only act on Timestamp fields. Reading is symmetric, so documents written<br>    here load back unchanged. Keys are always hashed: Firestore document ids<br>    cannot contain &quot;/&quot;, and CIMD client_ids are URLs (the 2026-08-24 reconnect<br>    500). py-key-value-aio declares this store&#39;s API unstable and the adapter<br>    is a private attribute: keep the exact pin and re-run the TTL and<br>    sanitization tests on every bump.<br>    &quot;&quot;&quot;<br> <br>    def __init__(self, **kwargs: Any):<br>        kwargs.setdefault(&quot;key_sanitization_strategy&quot;, AlwaysHashStrategy())<br>        super().__init__(**kwargs)<br>        self._serialization_adapter = BasicSerializationAdapter(<br>            date_format=&quot;datetime&quot;, value_format=&quot;string&quot;<br>        )<br> <br> <br>def oauth_client_storage(<br>    env: Mapping[str, str],<br>    store_factory: Callable[..., AsyncKeyValue] | None = None,<br>) -&gt; AsyncKeyValue | None:<br>    &quot;&quot;&quot;Durable, encrypted OAuth state store — or None to keep FastMCP&#39;s default.<br> <br>    The Fernet key is an HKDF of the Google client secret with FastMCP&#39;s<br>    storage salt (one step; FastMCP&#39;s own file store derives through its JWT<br>    key first — the two stores never coexist, so parity is irrelevant). No<br>    new secret is needed and upstream tokens stay encrypted at rest. A<br>    rotated secret turns old documents into cache misses (re-registration),<br>    never into errors.<br>    &quot;&quot;&quot;<br>    database = env.get(&quot;FIRESTORE_DATABASE&quot;, &quot;&quot;)<br>    if not database:<br>        return None<br>    if database == DEFAULT_FIRESTORE_DATABASE:<br>        raise ValueError(&quot;FIRESTORE_DATABASE must be a named database, never &#39;(default)&#39;&quot;)<br>    build_store = store_factory or TimestampFirestoreStore<br>    key = derive_jwt_key(<br>        high_entropy_material=env[&quot;GOOGLE_OAUTH_CLIENT_SECRET&quot;],<br>        salt=STORAGE_ENCRYPTION_SALT,<br>    )<br>    return FernetEncryptionWrapper(<br>        key_value=build_store(project=env.get(&quot;GCP_PROJECT&quot;) or None, database=database),<br>        fernet=Fernet(key=key),<br>        raise_on_decryption_error=False,<br>    )<br> <br> <br>def parse_allowed_emails(raw: str) -&gt; set[str]:<br>    &quot;&quot;&quot;Normalize the comma-separated email allowlist from configuration.&quot;&quot;&quot;<br>    return {email.strip().lower() for email in raw.split(&quot;,&quot;) if email.strip()}<br> <br> <br>class EmailAllowlist(Middleware):<br>    &quot;&quot;&quot;Restrict access to the configured email allowlist.<br> <br>    Google authenticates anyone with a Google account; this narrows to ours.<br>    Runs on every MCP request, after OAuth token validation.<br>    &quot;&quot;&quot;<br> <br>    def __init__(self, allowed_emails: set[str]):<br>        self.allowed_emails = allowed_emails<br> <br>    async def on_request(self, context: MiddlewareContext, call_next):<br>        &quot;&quot;&quot;Reject the request unless the token&#39;s email claim is allowlisted.<br> <br>        Args:<br>            context: The middleware context for the in-flight MCP request.<br>            call_next: Continuation that forwards the request downstream.<br> <br>        Returns:<br>            The downstream response when the caller&#39;s email is allowlisted.<br> <br>        Raises:<br>            ToolError: If the access token is missing or its email claim is<br>                not in the configured allowlist.<br>        &quot;&quot;&quot;<br>        token = get_access_token()<br>        email = (token.claims.get(&quot;email&quot;) or &quot;&quot;).lower() if token else &quot;&quot;<br>        if email not in self.allowed_emails:<br>            raise ToolError(&quot;Account not authorized for this server.&quot;)<br>        return await call_next(context)<br> <br> <br>def build_proxy(env: Mapping[str, str]):<br>    &quot;&quot;&quot;Compose the authenticated proxy from its explicit environment.&quot;&quot;&quot;<br>    auth = GoogleProvider(<br>        client_id=env[&quot;GOOGLE_OAUTH_CLIENT_ID&quot;],<br>        client_secret=env[&quot;GOOGLE_OAUTH_CLIENT_SECRET&quot;],<br>        base_url=env[&quot;BASE_URL&quot;],<br>        required_scopes=[&quot;openid&quot;, &quot;https://www.googleapis.com/auth/userinfo.email&quot;],<br>        allowed_client_redirect_uris=[<br>            &quot;https://claude.ai/api/mcp/auth_callback&quot;,<br>            &quot;http://localhost:*&quot;,<br>            &quot;http://127.0.0.1:*&quot;,<br>        ],<br>        client_storage=oauth_client_storage(env),<br>    )<br>    backend = StdioTransport(<br>        command=&quot;garmin-mcp&quot;,<br>        args=[],<br>        env={<br>            &quot;GARMIN_MCP_TRANSPORT&quot;: &quot;stdio&quot;,<br>            &quot;GARMINTOKENS&quot;: env[&quot;GARMINTOKENS&quot;],<br>            &quot;GARMIN_ENABLED_TOOLS&quot;: env.get(&quot;GARMIN_ENABLED_TOOLS&quot;, &quot;&quot;),<br>            &quot;HOME&quot;: env.get(&quot;HOME&quot;, &quot;/home/mcp&quot;),<br>            &quot;PATH&quot;: env[&quot;PATH&quot;],<br>        },<br>    )<br>    app = create_proxy(backend, name=&quot;Garmin Coach&quot;, auth=auth)<br>    app.add_middleware(EmailAllowlist(parse_allowed_emails(env[&quot;ALLOWED_EMAILS&quot;])))<br>    return app<br> <br> <br>def main(env: Mapping[str, str] = os.environ) -&gt; None:<br>    &quot;&quot;&quot;Run the proxy over HTTP for the container entrypoint.&quot;&quot;&quot;<br>    build_proxy(env).run(<br>        transport=&quot;http&quot;,<br>        host=&quot;0.0.0.0&quot;,<br>        port=int(env.get(&quot;PORT&quot;, &quot;8080&quot;)),<br>    )<br> <br> <br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>O adaptador de serialização usa uma API interna de py-key-value-aio; mantenha o pin indicado e execute os testes de TTL e chaves ao atualizar. Configure FIRESTORE_DATABASE: sem ela, o proxy mantém o armazenamento local padrão do FastMCP, inadequado para persistência no Cloud Run. O código recusa o database (default). A chave de cifragem deriva do client secret Google; sua rotação torna os dados antigos ilegíveis e exige nova autenticação.</p><h3>Dockerfile</h3><pre># Garmin MCP server behind a FastMCP OAuth proxy — Stage 0 connector.<br># garmin_mcp pinned to the KB-validated commit (2026-07-18); it runs as a stdio<br># subprocess of proxy.py, which terminates MCP OAuth (Google IdP).<br># Runtime config is injected by Terraform: infrastructure/cloud-run-mcp.tf<br>#<br># Supply-chain pins (security review 2026-07-18): base image by digest, uv,<br># fastmcp and py-key-value-aio by version. Refresh deliberately, not implicitly.<br># fastmcp 3.4.4 is the version the test suite runs against (make coverage-mcp);<br># py-key-value-aio[firestore] backs the OAuth state store (proxy.py).<br>FROM python:3.12-slim@sha256:57cd7c3a7a273101a6485ba99423ee568157882804b1124b4dd04266317710de<br> <br>COPY --from=ghcr.io/astral-sh/uv:0.11.29 /uv /usr/local/bin/uv<br> <br>ENV PYTHONUNBUFFERED=1 \<br>    UV_SYSTEM_PYTHON=1<br> <br># Tarball install avoids needing git in the image.<br>RUN uv pip install \<br>    &quot;fastmcp==3.4.4&quot; \<br>    &quot;py-key-value-aio[firestore]==0.4.5&quot; \<br>    &quot;garmin-mcp @ https://github.com/Taxuspt/garmin_mcp/archive/2974244bfda1595b00836b3f942f579ec2d6f7d6.tar.gz&quot;<br> <br>COPY proxy.py /app/proxy.py<br> <br># Non-root runtime user. Token dir comes from GARMINTOKENS (absolute path,<br># read-only secret volume); HOME is set so any &quot;~&quot; expansion stays writable.<br>RUN useradd --create-home --uid 1000 mcp<br>USER mcp<br>ENV HOME=/home/mcp<br> <br>ENTRYPOINT [&quot;python&quot;, &quot;/app/proxy.py&quot;]</pre><h3>.dockerignore</h3><pre>**<br>!Dockerfile<br>!proxy.py</pre><h3>main.tf</h3><pre>terraform {<br>  required_version = &quot;&gt;= 1.7&quot;<br>  required_providers {<br>    google = {<br>      source  = &quot;hashicorp/google&quot;<br>      version = &quot;~&gt; 6.0&quot;<br>    }<br>  }<br>}<br> <br>variable &quot;project_id&quot; {<br>  type = string<br>}<br>variable &quot;project_number&quot; {<br>  type = string<br>}<br>variable &quot;region&quot; {<br>  type    = string<br>  default = &quot;us-central1&quot;<br>}<br>variable &quot;allowed_emails&quot; {<br>  type = list(string)<br>}<br> <br>variable &quot;google_oauth_client_id&quot; {<br>  type    = string<br>  default = &quot;&quot;<br>}<br>variable &quot;enabled_tools&quot; {<br>  description = &quot;Ferramentas de leitura do conector Garmin Coach&quot;<br>  type        = list(string)<br>  default = [<br>    &quot;get_activities&quot;,<br>    &quot;get_activity&quot;,<br>    &quot;get_sleep_summary&quot;,<br>    &quot;get_sleep_data&quot;,<br>    &quot;get_stress_summary&quot;,<br>    &quot;get_body_battery&quot;,<br>    &quot;get_stats&quot;,<br>    &quot;get_rhr_day&quot;,<br>    &quot;get_heart_rates_summary&quot;,<br>    &quot;get_training_readiness&quot;,<br>    &quot;get_morning_training_readiness&quot;,<br>    &quot;get_hrv_data&quot;,<br>    &quot;get_hrv_trend&quot;,<br>    &quot;get_training_status&quot;,<br>    &quot;get_training_load_trend&quot;,<br>    &quot;get_vo2max_trend&quot;,<br>  ]<br> <br>  validation {<br>    condition     = alltrue([for t in var.enabled_tools : !can(regex(&quot;^(create|schedule|upload|delete|unschedule|set)_&quot;, t))])<br>    error_message = &quot;Write-shaped tool names are forbidden on the public connector.&quot;<br>  }<br>}<br> <br>variable &quot;deploy_service&quot; {<br>  type    = bool<br>  default = false<br>}<br>variable &quot;image&quot; {<br>  type    = string<br>  default = &quot;&quot;<br>}<br>variable &quot;revision_marker&quot; {<br>  type    = string<br>  default = &quot;1&quot;<br>}<br> <br>provider &quot;google&quot; {<br>  project = var.project_id<br>  region  = var.region<br>}<br> <br>locals {<br>  service_name = &quot;garmin-mcp&quot;<br>  base_url     = &quot;https://${local.service_name}-${var.project_number}.${var.region}.run.app&quot;<br>  secrets      = toset([&quot;garmin-tokens&quot;, &quot;garmin-mcp-oauth-client-secret&quot;])<br>  ttl_collections = toset([<br>    &quot;mcp-authorization-codes&quot;, &quot;mcp-oauth-transactions&quot;, &quot;mcp-jti-mappings&quot;,<br>    &quot;mcp-upstream-tokens&quot;, &quot;mcp-refresh-tokens&quot;<br>  ])<br>}<br> <br>resource &quot;google_project_service&quot; &quot;api&quot; {<br>  for_each = toset([<br>    &quot;run.googleapis.com&quot;, &quot;artifactregistry.googleapis.com&quot;,<br>    &quot;secretmanager.googleapis.com&quot;, &quot;firestore.googleapis.com&quot;,<br>    &quot;iam.googleapis.com&quot;, &quot;cloudresourcemanager.googleapis.com&quot;<br>  ])<br>  service            = each.value<br>  disable_on_destroy = false<br>}<br> <br>resource &quot;google_artifact_registry_repository&quot; &quot;images&quot; {<br>  location      = var.region<br>  repository_id = &quot;garmin-mcp&quot;<br>  format        = &quot;DOCKER&quot;<br>  depends_on    = [google_project_service.api]<br>}<br> <br>resource &quot;google_service_account&quot; &quot;runtime&quot; {<br>  account_id   = &quot;garmin-mcp-run&quot;<br>  display_name = &quot;Garmin MCP runtime&quot;<br>  depends_on   = [google_project_service.api]<br>}<br> <br>resource &quot;google_secret_manager_secret&quot; &quot;secret&quot; {<br>  for_each  = local.secrets<br>  secret_id = each.value<br>  replication {<br>    auto {}<br>  }<br>  depends_on = [google_project_service.api]<br>}<br> <br>resource &quot;google_secret_manager_secret_iam_member&quot; &quot;read&quot; {<br>  for_each  = local.secrets<br>  secret_id = google_secret_manager_secret.secret[each.key].id<br>  role      = &quot;roles/secretmanager.secretAccessor&quot;<br>  member    = &quot;serviceAccount:${google_service_account.runtime.email}&quot;<br>}<br> <br>resource &quot;google_firestore_database&quot; &quot;oauth&quot; {<br>  name                    = &quot;garmin-mcp-oauth&quot;<br>  location_id             = var.region<br>  type                    = &quot;FIRESTORE_NATIVE&quot;<br>  delete_protection_state = &quot;DELETE_PROTECTION_ENABLED&quot;<br>  deletion_policy         = &quot;ABANDON&quot;<br>  depends_on              = [google_project_service.api]<br>}<br> <br>resource &quot;google_project_iam_member&quot; &quot;oauth_storage&quot; {<br>  project = var.project_id<br>  role    = &quot;roles/datastore.user&quot;<br>  member  = &quot;serviceAccount:${google_service_account.runtime.email}&quot;<br>  condition {<br>    title      = &quot;garmin-mcp-oauth-only&quot;<br>    expression = &quot;resource.name == \&quot;projects/${var.project_id}/databases/${google_firestore_database.oauth.name}\&quot;&quot;<br>  }<br>}<br> <br>resource &quot;google_firestore_field&quot; &quot;ttl&quot; {<br>  for_each   = local.ttl_collections<br>  database   = google_firestore_database.oauth.name<br>  collection = each.value<br>  field      = &quot;expires_at&quot;<br>  ttl_config {}<br>}<br> <br>resource &quot;google_cloud_run_v2_service&quot; &quot;mcp&quot; {<br>  count               = var.deploy_service ? 1 : 0<br>  name                = local.service_name<br>  location            = var.region<br>  deletion_protection = true<br> <br>  template {<br>    service_account = google_service_account.runtime.email<br>    timeout         = &quot;300s&quot;<br>    scaling {<br>      min_instance_count = 0<br>      max_instance_count = 1<br>    }<br>    containers {<br>      image = var.image<br>      ports {<br>        container_port = 8080<br>      }<br>      resources {<br>        limits            = { cpu = &quot;1&quot;, memory = &quot;512Mi&quot; }<br>        startup_cpu_boost = true<br>      }<br>      dynamic &quot;env&quot; {<br>        for_each = {<br>          BASE_URL               = local.base_url<br>          GCP_PROJECT            = var.project_id<br>          FIRESTORE_DATABASE     = google_firestore_database.oauth.name<br>          GOOGLE_OAUTH_CLIENT_ID = var.google_oauth_client_id<br>          ALLOWED_EMAILS         = join(&quot;,&quot;, var.allowed_emails)<br>          GARMIN_ENABLED_TOOLS   = join(&quot;,&quot;, var.enabled_tools)<br>          GARMINTOKENS           = &quot;/secrets/garminconnect&quot;<br>          REVISION_MARKER        = var.revision_marker<br>        }<br>        content {<br>          name  = env.key<br>          value = env.value<br>        }<br>      }<br>      env {<br>        name = &quot;GOOGLE_OAUTH_CLIENT_SECRET&quot;<br>        value_source {<br>          secret_key_ref {<br>            secret  = google_secret_manager_secret.secret[&quot;garmin-mcp-oauth-client-secret&quot;].secret_id<br>            version = &quot;latest&quot;<br>          }<br>        }<br>      }<br>      volume_mounts {<br>        name       = &quot;garmin-tokens&quot;<br>        mount_path = &quot;/secrets/garminconnect&quot;<br>      }<br>    }<br>    volumes {<br>      name = &quot;garmin-tokens&quot;<br>      secret {<br>        secret = google_secret_manager_secret.secret[&quot;garmin-tokens&quot;].secret_id<br>        items {<br>          version = &quot;latest&quot;<br>          path    = &quot;garmin_tokens.json&quot;<br>        }<br>      }<br>    }<br>  }<br> <br>  lifecycle {<br>    precondition {<br>      condition     = var.image != &quot;&quot; &amp;&amp; var.google_oauth_client_id != &quot;&quot;<br>      error_message = &quot;Publique a imagem e configure o client ID antes do serviço.&quot;<br>    }<br>  }<br>  depends_on = [<br>    google_secret_manager_secret_iam_member.read,<br>    google_project_iam_member.oauth_storage,<br>  ]<br>}<br> <br>resource &quot;google_cloud_run_v2_service_iam_member&quot; &quot;public&quot; {<br>  count    = var.deploy_service ? 1 : 0<br>  name     = google_cloud_run_v2_service.mcp[0].name<br>  location = var.region<br>  role     = &quot;roles/run.invoker&quot;<br>  member   = &quot;allUsers&quot;<br>}<br> <br>output &quot;base_url&quot; {<br>  value = local.base_url<br>}<br>output &quot;connector_url&quot; {<br>  value = &quot;${local.base_url}/mcp&quot;<br>}<br>output &quot;google_redirect_uri&quot; {<br>  value = &quot;${local.base_url}/auth/callback&quot;<br>}<br>output &quot;image_repository&quot; {<br>  value = &quot;${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.images.repository_id}/garmin-mcp&quot;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2272c9eecea0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Prompting Claude Code. Write the Loop That Prompts It.]]></title>
            <link>https://matheusjerico.medium.com/stop-prompting-claude-code-write-the-loop-that-prompts-it-39a5adb4eeff?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a5adb4eeff</guid>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 15:28:16 GMT</pubDate>
            <atom:updated>2026-07-13T15:28:16.119Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Part three of the agent loop series: how to build a loop that finds its own work and drives Claude Code through it, instead of you typing prompts all day.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89CvcCGiPubHEvBj-0NENw.png" /></figure><p>Boris Cherny, who leads Claude Code at Anthropic, said something that reframes the whole job: <em>“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”</em></p><p>The first two parts of this series were about the loop inside a single run. <a href="https://matheusjerico.medium.com/who-owns-the-agent-loop-node-js-harnesses-vs-python-frameworks-langgraph-crewai-langchain-8c6a153178b4">Part one</a> drew the line between a harness that owns the agent loop and a framework where you author it. <a href="https://matheusjerico.medium.com/you-dont-own-the-agent-loop-here-s-how-to-control-it-anyway-f5be40ee7313">Part two</a> showed how to control a runtime-owned loop from the outside. This part climbs one floor higher. We are not steering a single run anymore. We are writing the standing loop that decides what to run, dispatches it to Claude Code, checks the result, and decides what to do next, on repeat, without you in the chair.</p><p>If you have used /goal in Claude Code, you have met the simple version: you state a condition and Claude works turn after turn until a judge says it is met. That is a loop you did not have to write, aimed at one goal you stated. The full version is different in one decisive way: the loop also <em>finds the work</em>. It reads your issue tracker, your CI, your review queue, decides what is worth doing now, and drives an agent through it. You stop being the thing that picks the next task. That is what Boris means by writing loops.</p><p>This article shows you how to build that, concretely, with Claude Code as the worker.</p><h3>The simple version and the full version</h3><p>The simple version answers “is this one task done yet?” The full version answers a harder question every tick: “what should be worked on right now, and is the last thing actually finished?”</p><p>That second question is why the full version is more than a while loop around a prompt. A real loop has six moving parts:</p><ol><li><strong>Find work.</strong> Look at the world (issues, failing CI, PR comments, a queue) and pick the next unit of work. If there is nothing, idle.</li><li><strong>Build context.</strong> Turn that unit into a task the agent can act on, and give it a clean place to work.</li><li><strong>Dispatch.</strong> Run the agent against the task with a fixed goal, bounded tools, and caps.</li><li><strong>Verify.</strong> Run the gate. The gate is the truth, never the agent’s self-report.</li><li><strong>Record and hand off.</strong> Commit, update a progress file, comment on the issue, so the next tick (or a fresh context) continues cleanly.</li><li><strong>Decide.</strong> Done, keep working, or escalate to a human. Then loop.</li></ol><pre>            +-----------------------------------------------+<br>   timer -&gt; | 1 find work  -&gt; 2 build context -&gt; 3 dispatch  |<br>            |      ^                                  |      |<br>            |      |                                  v      |<br>            | 6 decide &lt;- 5 record/hand off &lt;- 4 verify(gate)|<br>            +-----------------------------------------------+</pre><p>Notice the model only appears in step 3. The other five steps are ordinary engineering: read a queue, write a file, run a command, parse a result, make a decision. That is the point of loop engineering. Most of the system is deterministic code you fully control, and the model is one step inside it.</p><h3>The agent’s contract: the goal, written once</h3><p>Before any code, the worker needs a contract: a goal that survives a context reset, because a fresh worker starts each tick knowing nothing about the last one. A goal that holds up has five sections.</p><ul><li><strong>Objective.</strong> Name the outcome, self-contained. Per work item this is the issue; the standing parts below apply to every run.</li><li><strong>Why.</strong> Include it only when it changes the plan. If deleting it would not change what the agent does, delete it.</li><li><strong>Done when.</strong> Observations, not activities. One fact per line, never “or,” ordered weakest to strongest. Make the worker run the check and show the output, because the evaluator and you can only judge what is in the transcript, and a claim is not proof.</li><li><strong>Out of scope.</strong> The fence, and the highest-leverage section, because it shapes decisions before they are made. Forbid the worst reasonable misinterpretation, plus one line that matters more than it looks: never edit, weaken, or delete a test or the gate to make it pass. The cheapest way to turn a red gate green is to break the gate.</li><li><strong>If it fails.</strong> A recovery line, and a stop condition (turns, time, or cost) on every goal, so a bad run escalates cleanly instead of improvising or looping forever.</li></ul><p>The standing parts (Out of scope, If it fails, how to verify) live in a .claude/loop-goal.md that becomes the worker&#39;s system prompt every tick. The dynamic parts (this issue&#39;s Objective and Done when) arrive as the per-tick task. And notice the split the rest of this article leans on: the goal is prose, the deterministic gate that proves done is code, and the gate lives in the driver, not in the goal.</p><p>If you want the blank template, filled examples, and the run setups to copy from, I put them in a small companion repo: <a href="https://github.com/matheusjerico/claude-code-goals">github.com/matheusjerico/claude-code-goals</a>.</p><h3>Build it, step one: the worker</h3><p>The worker is a single Claude Code run in headless mode. Claude Code is the harness here, so you do not write the agent loop inside a run. You hand it the task and the contract and let it work. The whole worker is one command:</p><pre>claude -p &quot;Work the task in TASK.md. Read claude-progress.md first, make the smallest next increment, end with the STATUS line.&quot; \<br>  --append-system-prompt-file .claude/loop-goal.md \<br>  --allowedTools &quot;Read&quot; &quot;Edit&quot; &quot;Write&quot; &quot;Bash(npm *)&quot; &quot;Bash(git *)&quot; &quot;Bash(gh *)&quot; \<br>  --permission-mode acceptEdits \<br>  --max-turns 40 \<br>  --output-format json</pre><p>Every flag maps to a part of the contract. --append-system-prompt-file loads the goal while keeping Claude Code&#39;s coding behavior. --allowedTools is the capability boundary. --permission-mode acceptEdits lets it work unattended. --max-turns is the inner budget so a single run cannot spin forever. --output-format json gives you a parseable result with the final text, the cost, and the session id.</p><p>Run with -p, the worker is fully non-interactive: no human approves anything mid-run. That is exactly what you want for a loop, and exactly why the gate and the tool boundary have to be right.</p><h3>Build it, step two: the driver that finds work and decides</h3><p>The driver is the loop. It is plain TypeScript, and it is where you spend your engineering. Here is one tick: find an issue, run the worker against it, verify, and decide. I am using the GitHub CLI (gh) as the work source because it is the most common one, but the shape is identical for a Linear queue, a database table, or failing CI.</p><pre>import { execFileSync, writeFileSync } from &quot;node:child_process&quot;;<br><br>const sh = (cmd: string, args: string[]) =&gt;<br>  execFileSync(cmd, args, { encoding: &quot;utf8&quot; });<br><br>const tryGate = () =&gt; { try { sh(&quot;npm&quot;, [&quot;run&quot;, &quot;ci&quot;]); return true; } catch { return false; } };<br><br><br>function tick(): &quot;idle&quot; | &quot;done&quot; | &quot;working&quot; | &quot;escalated&quot; {<br><br>  // 1. find work: the next ready issue, or nothing<br>  const issues = JSON.parse(sh(&quot;gh&quot;, [&quot;issue&quot;, &quot;list&quot;, &quot;--label&quot;, &quot;ready-for-agent&quot;,<br>    &quot;--json&quot;, &quot;number,title,body&quot;, &quot;--limit&quot;, &quot;1&quot;]));<br>  if (issues.length === 0) return &quot;idle&quot;;<br>  const issue = issues[0];<br><br>  // 2. build context: turn the issue into the task the worker reads<br>  writeFileSync(&quot;TASK.md&quot;, `# ${issue.title}\n\n${issue.body}`);<br><br>  // 3. dispatch: one headless worker run, goal as system prompt<br>  const raw = sh(&quot;claude&quot;, [&quot;-p&quot;, &quot;Work the task in TASK.md. Read claude-progress.md first. End with STATUS.&quot;,<br>    &quot;--append-system-prompt-file&quot;, &quot;.claude/loop-goal.md&quot;,<br>    &quot;--allowedTools&quot;, &quot;Read&quot;, &quot;Edit&quot;, &quot;Write&quot;, &quot;Bash(npm *)&quot;, &quot;Bash(git *)&quot;, &quot;Bash(gh *)&quot;,<br>    &quot;--permission-mode&quot;, &quot;acceptEdits&quot;, &quot;--max-turns&quot;, &quot;40&quot;, &quot;--output-format&quot;, &quot;json&quot;]);<br>  const status = (JSON.parse(raw).result.match(/STATUS:\s*(\w+)/i)?.[1] ?? &quot;working&quot;).toLowerCase();<br><br>  // 4. verify: the gate is the truth, not the agent&#39;s claim<br>  const green = tryGate();<br><br>  // 5 and 6. record and decide<br>  if (green &amp;&amp; status === &quot;done&quot;) {<br>    sh(&quot;gh&quot;, [&quot;pr&quot;, &quot;create&quot;, &quot;--fill&quot;]);<br>    sh(&quot;gh&quot;, [&quot;issue&quot;, &quot;edit&quot;, `${issue.number}`, &quot;--add-label&quot;, &quot;in-review&quot;, &quot;--remove-label&quot;, &quot;ready-for-agent&quot;]);<br>    return &quot;done&quot;;<br>  }<br>  if (status === &quot;blocked&quot; || !green) {<br>    sh(&quot;gh&quot;, [&quot;issue&quot;, &quot;comment&quot;, `${issue.number}`, &quot;--body&quot;, &quot;Agent could not finish. A human should look.&quot;]);<br>    sh(&quot;gh&quot;, [&quot;issue&quot;, &quot;edit&quot;, `${issue.number}`, &quot;--add-label&quot;, &quot;needs-human&quot;, &quot;--remove-label&quot;, &quot;ready-for-agent&quot;]);<br>    return &quot;escalated&quot;;<br>  }<br><br>  return &quot;working&quot;; // next tick continues this issue from claude-progress.md and git<br>}</pre><p>Read step 5 and 6 closely, because that is the part /goal does not give you. The loop advances the issue only when the gate is green <em>and</em> the agent reported done. It escalates on a blocked status or a red gate. Otherwise it leaves the issue open and the next tick continues it. The model proposed; the loop disposed.</p><p>Wrap the tick in a bounded outer loop with a cap and a back-off so it halts and does not hammer an empty queue:</p><pre>const MAX_TICKS = 50;<br>for (let i = 0; i &lt; MAX_TICKS; i++) {<br>  const result = tick();<br>  if (result === &quot;idle&quot;) await sleep(&quot;10m&quot;); // nothing to do, back off<br>}</pre><p>That is the full version in one screen. A program that finds work, prompts Claude Code, checks it, and decides, with you nowhere in the tick.</p><h3>The guardrails that make it safe to leave alone</h3><p>An autonomous loop is only as good as the things that stop it from doing damage. Four are non-negotiable.</p><p><strong>The gate is separate from the claim.</strong> This is the single most important rule and it is worth repeating. The agent ends with STATUS: done; the loop runs npm run ci and believes the exit code, not the sentence. A green gate plus a done status is the only thing that ships. With /goal the judge is a model reading the transcript, which is softer. In a real loop you want a deterministic gate.</p><p><strong>Budgets at two levels.</strong> Each worker run is capped with --max-turns (and you can track cost from the json result). The driver is capped with MAX_TICKS and a spend ceiling. This is the same budgets-as-a-fold idea from part two, now applied at the outer layer. A loop without a cap is a way to wake up to a very large bill.</p><p><strong>Escalation is a first-class outcome.</strong> “Blocked” is not failure, it is the loop knowing its limits. The most common production loop is not fully autonomous, it is autonomous until it is not, and then it hands a labeled issue and a clear reason to a human. Build the escalation path before you build the happy path.</p><p><strong>Isolation for parallelism.</strong> The moment you want more than one issue in flight, give each worker its own git worktree so they do not fight over the working tree. git worktree add ../work-123 per issue, run the worker there, and the loop can fan out across several issues at once. This is how a single loop spawns several agents on a busy morning.</p><p>One more, quieter than the others: <strong>observability</strong>. Log every tick (issue, status, gate result, cost) to a file or a dashboard. When a loop runs unattended for hours, the log is how you trust it, and how you debug it when a tick goes sideways.</p><h3>Running it for real</h3><p>The driver is just a program, so “running the loop” is whatever scheduling you already know.</p><ul><li><strong>A cron job or a long-running process</strong> is the simplest. Run the driver every ten minutes, or run it once as a daemon with its own sleep. This is the baseline Boris-style setup.</li><li><strong>GitHub Actions</strong> works well when the work source is GitHub. A scheduled workflow checks out the repo, runs the driver for a bounded number of ticks, and pushes branches or opens PRs. You get isolation and logs for free.</li><li><strong>Claude Code’s own scheduling.</strong> For lighter cases you do not need a custom driver at all. The /loop skill re-runs a prompt on an interval, and scheduled tasks run a prompt on a cron schedule. These are the built-in cousins of the driver: good when the &quot;find work and decide&quot; logic is simple enough to live in a prompt rather than in code.</li></ul><p>The progression is worth seeing clearly. A plain session has a human in every turn. /goal removes the human from the turns but keeps one stated goal. The driver removes the human from the task selection too. Each step up trades immediacy for leverage, and each step up demands stronger gates because there is less human judgment left in the loop to catch a mistake.</p><h3>What stays human</h3><p>The full version is powerful, and the temptation is to point it at everything. Resist it. Some things belong outside the loop, on purpose.</p><p>Deciding <em>what</em> to build, and whether it is worth building, is a product judgment. The loop is great at working a well-specified ready-for-agent issue. It is not the right place to decide which issues earn that label. Keep the labeling, the prioritization, and the &quot;should we even do this&quot; upstream of the loop, with a person.</p><p>Anything irreversible or high-blast-radius stays gated by a human too. The loop can open a PR; a person merges it. The loop can draft a migration; a person runs it against production. The pattern is the same one good engineers already use: automate the work, keep a human at the consequential edges.</p><p>And when a loop is new, run it in the open before you run it asleep. Watch a dozen ticks. Read what it did. The progress files and the gate logs are right there. Trust is earned by observation, not by configuration.</p><h3>The whole series in one line</h3><p>Three parts, one idea, climbing a ladder of abstraction. Part one: a harness owns the agent loop, and that is a feature, not a loss. Part two: you control that runtime-owned loop from the outside, through a handful of seams. Part three: you stop being inside any loop at all, and write the standing loop that finds work and drives the agent through it.</p><p>The skill that carries across all three is the same. It is not memorizing an SDK. It is knowing which layer a problem lives on, and reaching for the smallest mechanism that solves it. Prompt when you are exploring. Use /goal when you have one clear target. Write the loop when the work is repetitive, well-specified, and gated. Boris&#39;s line is not that the work got easier. It is that the leverage moved. Your job is to write the loop, and to stay the engineer who owns it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a5adb4eeff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Don’t Own the Agent Loop. Here’s How to Control It Anyway.]]></title>
            <link>https://matheusjerico.medium.com/you-dont-own-the-agent-loop-here-s-how-to-control-it-anyway-f5be40ee7313?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5be40ee7313</guid>
            <category><![CDATA[harness-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Tue, 16 Jun 2026 15:31:21 GMT</pubDate>
            <atom:updated>2026-06-16T15:31:21.261Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Part two of “Who Owns the Agent Loop?”: five concrete seams for steering a runtime-owned loop in TypeScript, without rewriting it or forking the runtime.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*727z_SzKV79jurDsh_uRLQ.png" /></figure><blockquote>Part 2 of 2. Part 1: <a href="https://matheusjerico.medium.com/who-owns-the-agent-loop-node-js-harnesses-vs-python-frameworks-langgraph-crewai-langchain-8c6a153178b4">Who Owns the Agent Loop?</a></blockquote><h3>You don’t own the loop. So what?</h3><p>In <a href="https://matheusjerico.medium.com/who-owns-the-agent-loop-node-js-harnesses-vs-python-frameworks-langgraph-crewai-langchain-8c6a153178b4">Who Owns the Agent Loop?</a> I drew a line between two ways to build agents. A harness, like Pi in Node and TypeScript, owns the send-prompt, run-tools, feed-results, repeat cycle for you, and you steer it with config and hooks. A framework, like LangGraph, CrewAI, or LangChain in Python, hands you the building blocks and asks you to author the control flow yourself. I ended on a promise: with a harness you reshape behavior rather than rewrite the loop. This article is that reshaping, with code.</p><p>There is an objection hiding in that promise, and it is worth saying out loud. “You don’t own the loop” sounds a lot like “you don’t have control.” If the runtime runs the cycle, what is left for you to decide? The fear is that a harness is a black box that does what it wants and hands you back a result.</p><p>It is not, and the proof is mechanical. A harness is a sealed machine, but the manufacturer left ports on the outside. Through a small set of seams you can change almost everything that matters in production: what the agent is, what it can see, when it must stop, what it is forbidden to do, and even which engine runs underneath. Five seams, and you reach all of them from outside the loop.</p><p>There is a short list of things you genuinely cannot do without forking the runtime, and I will name it at the end rather than pretend the seams are infinite. But that list is shorter than the fear suggests. Here is the how, using the same research agent from part one.</p><h3>Five seams in a loop you didn’t write</h3><p>Picture the loop as a sealed box. You cannot open the case and edit the cycle inside. But there are five ports on the outside, and each one is a place where your code touches the running loop.</p><ol><li><strong>Configuration and state</strong>: change what the agent is. Its system prompt, model, tools, and thinking level, even partway through a run.</li><li><strong>Events</strong>: subscribe to a typed stream of everything the loop does. This is the observability seam.</li><li><strong>Budgets</strong>: fold those events into accumulators and abort the run when a limit trips. This is a control seam built on top of events.</li><li><strong>Hooks</strong>: intercept a tool call before it executes and block or rewrite it. This is the interception seam.</li><li><strong>The engine port</strong>: wrap the whole harness behind your own interface so you can replace the runtime later. This is the swappability seam.</li></ol><p>The organizing idea: seams 1 through 4 reshape behavior inside one runtime, and seam 5 protects you across runtimes. Inside-the-box control, plus across-the-box insurance.</p><pre>            CONFIG/STATE          HOOKS (beforeToolCall)<br>                |                        |<br>                v                        v<br>        +=================================================+<br>        |   runtime-owned loop (you cannot edit this)     |<br>        |   prompt -&gt; model -&gt; tool calls -&gt; results -&gt; ^ |<br>        +=================================================+<br>                |                        |<br>                v                        v<br>            EVENTS (subscribe)     ENGINE PORT (your wrapper)<br>                |<br>                v<br>            BUDGETS (fold events, abort)</pre><p>Here is the whole article on one page. Skim it now, and the rest fills in each row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U9rPUxEbgDFegClRo1aalg.png" /></figure><h3>Seam 1: reconfigure what the agent is</h3><p>The cheapest seam is the one most people walk past. The loop reads its behavior from a state object, and you hold a reference to that object. Change the object, change the loop. No re-instantiation required.</p><p>Start the same research agent from part one:</p><pre>import { getModel } from &quot;@mariozechner/pi-ai&quot;;<br>import { createAgentSession, SessionManager } from &quot;@mariozechner/pi-coding-agent&quot;;<br><br>const { session } = await createAgentSession({<br>  model: getModel(&quot;anthropic&quot;, &quot;claude-sonnet-4-6&quot;),<br>  cwd: process.cwd(),<br>  sessionManager: SessionManager.open(&quot;./session.jsonl&quot;),<br>  tools: [&quot;read&quot;, &quot;grep&quot;, &quot;bash&quot;, &quot;search_docs&quot;], // built-ins plus your custom tools, by name<br>  customTools: [searchDocsTool],<br>});<br><br>// Seam 1: reconfigure the running loop by mutating its state.<br>session.agent.state.systemPrompt = &quot;You are a research assistant. Always cite sources.&quot;;</pre><p>That last line is the seam. It is not just startup configuration. The system prompt, model, tools, and thinking level all live on state, and you can override them mid-session. After the first turn you might tighten the system prompt, swap to a cheaper model for routine follow-ups, or drop a tool the agent no longer needs.</p><p>This is control, not just setup. The loop’s personality and its capability set are data, not code. You are editing the loop’s inputs while it runs, which is exactly the kind of steering you would have to hand-write in a framework.</p><p>One honest edge: a state change takes effect on the next turn, not retroactively on a model call already in flight. Set it, and the loop picks it up on the next cycle, so do not expect mid-stream magic.</p><p>Configuration tells the loop what to be. To know what it is actually doing, you subscribe to events.</p><h3>Seams 2 and 3: watch the loop, then put it on a budget</h3><p>This is the heart of it, and it carries the most code.</p><h3>The event stream</h3><p>The runtime emits a typed event for every meaningful step: agent_start, turn_start, message_start, message_update, message_end, tool_execution_start, tool_execution_end, turn_end, and agent_end. You subscribe with one call, and you get back a function that unsubscribes.</p><pre>// Seam 2: subscribe is a complete, read-only view of the loop. It returns an unsubscribe.<br>const unsubscribe = session.agent.subscribe((event) =&gt; {<br>  if (event.type === &quot;tool_execution_start&quot;) {<br>    console.log(&quot;tool:&quot;, event); // log, drive a progress UI, or feed a tracer<br>  }<br>});<br>// later: unsubscribe();</pre><p>Subscription is pure observation. It does not change the loop, it watches it. The important property is that it is complete. Nothing the loop does is invisible to a subscriber, and that completeness is what makes the next seam possible.</p><h3>Budgets as pure event folds</h3><p>Here is the reframing that carries the section. A budget is not a special harness feature you switch on. It is a pure fold over the event stream. Give it a name, a way to update an accumulator on each event, and a way to say whether the limit has been crossed.</p><p>A turn limit increments on every turn_end and aborts when the count reaches the cap:</p><pre>// Seam 3: a budget is a pure fold over events. maxTurns counts turn_end events.<br>function maxTurns(limit: number) {<br>  let turns = 0;<br>  return {<br>    name: &quot;maxTurns&quot;,<br>    onEvent(event) { if (event.type === &quot;turn_end&quot;) turns += 1; },<br>    shouldAbort() { return turns &gt;= limit; },<br>  };<br>}</pre><p>A cost ceiling has the same shape with a different accumulator. It sums the cost reported on each assistant message and aborts when the total crosses your threshold:</p><pre>// Seam 3: same shape, different accumulator. Sum assistant message cost.<br>function maxCostUsd(limit: number) {<br>  let total = 0;<br>  return {<br>    name: &quot;maxCostUsd&quot;,<br>    onEvent(event) {<br>      if (event.type === &quot;message_end&quot; &amp;&amp; event.message?.role === &quot;assistant&quot;) {<br>        total += event.message.usage?.cost?.total ?? 0;<br>      }<br>    },<br>    shouldAbort() { return total &gt;= limit; },<br>  };<br>}</pre><p>A deadline is the third fold, and it proves the pattern generalizes: capture the start time when the budget is created and compare against it in shouldAbort. Same interface, wall-clock instead of turns or dollars.</p><p>Now wire any number of budgets through one subscription. Run each event through every budget, and abort the moment one of them says so. The budget’s name becomes the run’s stop reason, which is a clean, debuggable signal:</p><pre>// Fold N budgets through one subscription; the first to trip names the stop reason.<br>function runBudgets(session, budgets) {<br>  return session.agent.subscribe((event) =&gt; {<br>    for (const b of budgets) {<br>      b.onEvent(event);<br>      if (b.shouldAbort()) { session.agent.abort(); return; } // b.name is the stop reason<br>    }<br>  });<br>}<br><br>runBudgets(session, [maxTurns(20), maxCostUsd(2)]);</pre><p>Notice what isn’t there. You did not write the loop, you did not count tokens by hand, and you did not patch the runtime. You stood at the event stream and decided, in a few lines of accumulator code, exactly when to pull the cord. The runtime runs the loop, and you own when it stops.</p><h3>Seam 4: intercept the tool call before it runs</h3><p>Events let you watch and stop. Hooks let you change a decision before it happens. The beforeToolCall hook fires after the model has asked for a tool and after its arguments are validated, but before the tool actually runs. That timing is the whole point: you get a veto.</p><pre>// Seam 4: CHAIN, do not clobber. The runtime may have installed its own guard.<br>const prior = session.agent.beforeToolCall;<br>session.agent.beforeToolCall = async (ctx, signal) =&gt; {<br>  const blocked = await prior?.(ctx, signal); // honor the runtime&#39;s guard first<br>  if (blocked?.block) return blocked;<br>  if (ctx.toolCall.name === &quot;bash&quot; &amp;&amp; isDangerous(ctx.args.command)) {<br>    return { block: true, reason: &quot;blocked by policy&quot; }; // the model sees only this<br>  }<br>  return undefined; // allow<br>};</pre><p>Three things make this seam powerful, and the third is where people get hurt.</p><p>First, it intercepts before execution. A blocked tool never touches the system, so this is a real safety boundary and not after-the-fact logging.</p><p>Second, the model is told the reason, but it never knows you intervened. The reason string comes back as the tool result, so from the model&#39;s point of view the tool simply returned that message. You shaped the agent&#39;s behavior invisibly, and the conversation continues without a special case.</p><p>Third, chain, do not clobber. The runtime may have installed its own beforeToolCall hook, and many harnesses do. If you write session.agent.beforeToolCall = yours, you can silently drop the runtime&#39;s guard and never notice. So capture the previous hook, await it first, honor its block, and only then run your own logic, exactly as the snippet does. A clobbered guard is a security regression you will not see until it bites.</p><p>There is a protective variant of the same idea. Oversized tool output, a giant file read or a noisy command, can blow your context window and your token budget before any limit trips. Truncate it before it re-enters the conversation:</p><pre>// Seam 4: protect the context window. Truncate oversized tool output before it returns.<br>function cap(text: string, max = 8_000) {<br>  return text.length &lt;= max ? text : text.slice(0, max) + &quot;\n[truncated]&quot;;<br>}</pre><p>Wrap a tool’s output in cap and a runaway read becomes a bounded one. It is a small thing that protects the budget seam from the inside.</p><p>Configuration, events, budgets, and hooks all reshape one runtime. The last seam is about not being trapped in it.</p><h3>Seam 5: wrap the runtime so you can replace it</h3><p>The first four seams are real control, but they share a weakness: every one of them is a call into a specific runtime’s API. If session.agent.* is scattered across twenty files, you have coupled your whole application to that SDK. The day you want to swap it, every one of those files is a change.</p><p>The fix is an old idea in a new place: hexagonal architecture, ports and adapters, applied to agents. Define a narrow interface that you own, and let a thin adapter implement it on top of the harness.</p><pre>// Seam 5: ports and adapters for agents. This interface is yours, not the harness&#39;s.<br>interface EngineSession {<br>  prompt(input: string): Promise&lt;void&gt;;<br>  continue(): Promise&lt;void&gt;;<br>  dispose(): Promise&lt;void&gt;;<br>}<br>interface Engine {<br>  open(spec: { model: string; tools: string[]; systemPrompt: string }): Promise&lt;EngineSession&gt;;<br>}</pre><p>This interface is yours, not Pi’s. The harness does not ship a pluggable engine port, and that is exactly the point: the seam is a discipline you impose, not a feature you import. Your application talks to Engine and EngineSession, and a small adapter translates those calls into createAgentSession, prompt, and dispose underneath.</p><p>Two payoffs, both concrete. The first is swap insurance. Moving from one harness to another, to the Claude Agent SDK, or to a homegrown loop on the core package becomes one new adapter instead of a refactor, because your application only ever saw the port. The second is testability. The same boundary lets you implement a fake Engine for tests, so you can exercise your budgets, hooks, and orchestration without booting a real model or a real runtime. The loop becomes testable precisely because you stopped calling it directly.</p><p>The event stream from seam 2 has one more use worth a sentence. Because it is a complete record of the run, a single subscriber can map each event to an observability span: a message_end becomes a generation, and a tool_execution_start with its matching end becomes a tool span. That is all a tracing tool like Langfuse or an OpenTelemetry exporter needs. The seams compose: the same events that power your budgets also power your traces.</p><h3>Which seam, and where the seams run out</h3><p>A quick map of which seam answers which need:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20Cy1v79hXgMTulosFo_vA.png" /></figure><p>Now the honest part, because “control it anyway” is not the same as “control it completely.” A few things genuinely need a fork or a custom loop on the core package:</p><ul><li><strong>Restructuring the loop itself.</strong> You cannot insert a mandatory planning step between every tool call, or run a tool before the first model call, without owning the cycle. The seams sit at the edges, they do not rewrite the middle.</li><li><strong>A fundamentally different context strategy.</strong> If the runtime compacts history its own way and you need a different approach, configuration and hooks may not reach far enough. Some runtimes expose a context transform, many do not.</li><li><strong>An interception point the runtime does not emit.</strong> You get the seams the runtime gives you. If you need a hook it does not provide, you are out of seams.</li></ul><p>This is exactly where seam 5 earns its keep. Because your application talks to your own Engine port, &quot;fork the runtime&quot; or &quot;drop to the core package and write a custom loop&quot; becomes a new adapter, not a rewrite. The seams and the port are complementary: seams 1 through 4 bend the common case, and the port lets you escape the rare one.</p><h3>Owning the control, not the loop</h3><p>Part one asked who owns the agent loop. The answer this article adds is that owning the loop and controlling the agent are two different things. The runtime can own the loop while you own the control, and you reach that control through five seams.</p><p>In one breath: configure what the agent is, watch what it does, budget when it stops, intercept what it is allowed to do, and wrap it so you can replace it. Four seams to bend behavior inside a runtime, and one to protect yourself across runtimes.</p><p>The lasting skill is not memorizing one harness API. It is recognizing which seam a requirement maps to, and reaching for the smallest one that does the job. Config before events, events before budgets, budgets before hooks, hooks before forking. You don’t have to write the loop to be the one in charge of it.</p><p><em>This is part two. If you missed it, part one is </em><a href="https://matheusjerico.medium.com/who-owns-the-agent-loop-node-js-harnesses-vs-python-frameworks-langgraph-crewai-langchain-8c6a153178b4"><em>Who Owns the Agent Loop?</em></a><em>. I am curious which seam you reach for most, and where you have had to fork anyway.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5be40ee7313" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who Owns the Agent Loop? Node.js Harnesses vs. Python Frameworks (LangGraph, CrewAI, LangChain)]]></title>
            <link>https://matheusjerico.medium.com/who-owns-the-agent-loop-node-js-harnesses-vs-python-frameworks-langgraph-crewai-langchain-8c6a153178b4?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c6a153178b4</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Sun, 31 May 2026 20:35:44 GMT</pubDate>
            <atom:updated>2026-06-03T21:16:07.076Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>One paradigm hands you a runtime and gets out of the way. The other hands you abstractions and asks you to compose. Here’s how to choose.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7jL4_jyuW2F4-Y28fabNlg.png" /><figcaption><em>A harness owns the loop and you steer it with config and hooks (left); a framework makes you author every transition yourself (right).</em></figcaption></figure><p>The space of tools for building AI agents has split into two camps that look almost identical from a distance. Both give you an LLM, tool calling, and some way to keep the model running until a task is done. Pick one and you can ship a working agent in an afternoon either way.</p><p>But the camps disagree on something fundamental, and the disagreement only surfaces once you’re past the demo and into production: <strong>who owns the agent loop?</strong></p><p>On one side are <em>harnesses</em>: runtimes like <strong>Pi</strong> (a Node.js/TypeScript agent harness) that own the entire send-prompt → run-tools → feed-results → repeat cycle. You don’t write that loop. You configure it and extend it. On the other side are <em>frameworks</em>: Python libraries like <strong>LangGraph</strong>, <strong>CrewAI</strong>, and <strong>LangChain</strong> that hand you explicit abstractions (graphs, crews, chains) and ask you to assemble the control flow yourself.</p><p>This isn’t really a Node versus Python argument, even though that’s how it tends to get framed. It’s a question of how much of the machinery you want to own. By the end of this article you’ll have a decision framework (not a winner) for picking the right side for a given problem.</p><h3>Harness vs. framework: a definition that actually holds up</h3><p>Let’s pin down the two words before we lean on them.</p><p>A <strong>harness</strong> is a runtime that owns the agent loop. It runs the cycle for you: send the prompt, collect the model’s tool calls, execute them, feed the results back, and loop until the model stops asking. It also tends to own the surrounding concerns: context and memory management, history compaction near the token limit, retries, and session persistence. You steer it through <strong>configuration</strong> (system prompt, model, allowed tools, turn limits) and through <strong>hooks and extensions</strong>, not by writing the loop yourself. Pi’s own framing captures the philosophy, <em>primitives, not features</em>: adapt the harness to your workflow, not the other way around.</p><p>A <strong>framework</strong> (or SDK) is a set of explicit abstractions you compose in code. LangGraph gives you a graph of nodes and edges. CrewAI gives you crews of role-playing agents. LangChain gives you chains. In every case <em>you</em> author the control flow; the framework supplies the building blocks and you own the wiring between them.</p><p>The distinction that matters: in a harness, the loop is <strong>invisible and owned by the runtime</strong>. In a framework, the loop (or the graph topology, or the crew structure) is <strong>explicit and authored by you</strong>.</p><p>It’s a spectrum, not a binary. Here’s the axis:</p><pre>HARNESS    you → [config + hooks] → ╔═══ runtime owns the loop ═══╗ → result<br>                                    ║ prompt → tools → results → ↺║<br>FRAMEWORK  you author every node / edge / crew / chain → the loop IS your code</pre><p>Keep that picture in mind. Everything below is a consequence of where a tool sits on it.</p><h3>The harness paradigm: hand the loop to the runtime</h3><p>Pi is a minimal, Node-native agent harness. Its design is a layered stack, and you consume only the layer you need: a provider abstraction over 15+ model providers at the bottom, the agent loop and tool dispatch in the middle, a full runtime with session persistence on top, and a terminal UI at the surface. The interesting layer for most engineers is the middle one, the loop.</p><p>Starting an agent means describing <em>what it should be</em>, then handing control to the runtime. Here’s a minimal research agent built on Pi’s session API:</p><pre>import { getModel } from &quot;@mariozechner/pi-ai&quot;;<br>import { createAgentSession, SessionManager } from &quot;@mariozechner/pi-coding-agent&quot;;<br><br>const { session } = await createAgentSession({<br>  model: getModel(&quot;anthropic&quot;, &quot;claude-sonnet-4-6&quot;),<br>  cwd: process.cwd(),<br>  sessionManager: SessionManager.open(&quot;./session.jsonl&quot;),<br>  tools: [&quot;read&quot;, &quot;grep&quot;, &quot;bash&quot;, &quot;search_docs&quot;], // built-ins + your custom tools, by name<br>  customTools: [searchDocsTool],                   // your typed tools<br>});<br><br>// You steer the runtime with config, not loop code:<br>session.agent.state.systemPrompt = &quot;You are a research assistant. Always cite sources.&quot;;<br><br>// A single call runs the entire prompt → tools → results → repeat cycle:<br>await session.prompt(&quot;Summarize the latest on agent frameworks and cite sources.&quot;);</pre><p>Notice what isn’t there. There’s no while loop. No &quot;if the model returned tool calls, execute them and call the model again.&quot; No token accounting to decide when to compact history. The runtime owns all of it. You declared a model, a set of tools, and a system prompt, and called prompt() once.</p><p>A custom tool is a small, typed object: a name, a description for the model, a parameter schema, and an execute function.</p><pre>const searchDocsTool = {<br>  name: &quot;search_docs&quot;,<br>  description: &quot;Search the internal docs and return matching passages.&quot;,<br>  parameters: { type: &quot;object&quot;, properties: { query: { type: &quot;string&quot; } }, required: [&quot;query&quot;] },<br>  async execute(_id: string, params: { query: string }, signal?: AbortSignal) {<br>    const passages = await search(params.query, signal);<br>    return { content: [{ type: &quot;text&quot;, text: passages }], details: {} };<br>  },<br>};</pre><p>So far this looks like convenience and not much else. The depth shows up when you need to <em>change the loop’s behavior</em>, because you can’t rewrite a loop you don’t own. Instead, you reach into it through events and hooks.</p><p>The runtime emits a typed event for every meaningful step: agent_start, message_update, tool_execution_start, turn_end, message_end, agent_end. You subscribe to observe, and you can act on what you see. Enforcing a turn limit, for example, is a pure fold over the event stream:</p><pre>let turns = 0;<br>const MAX_TURNS = 20;<br><br>const unsubscribe = session.agent.subscribe((event) =&gt; {<br>  if (event.type === &quot;turn_end&quot; &amp;&amp; ++turns &gt;= MAX_TURNS) {<br>    session.agent.abort(); // pull the cord on a loop you didn&#39;t write<br>  }<br>});</pre><p>The same shape works for a cost ceiling: accumulate usage.cost.total from each assistant message and abort when it crosses a threshold. The runtime runs the loop; you watch it go by and decide when to stop it.</p><p>Hooks go one step further. They let you intercept a decision <em>before</em> it happens, without the model ever knowing. A beforeToolCall hook can inspect and block a tool call:</p><pre>session.agent.beforeToolCall = async (ctx) =&gt; {<br>  if (ctx.toolCall.name === &quot;bash&quot; &amp;&amp; isDangerous(ctx.args.command)) {<br>    return { block: true, reason: &quot;blocked by policy&quot; };<br>  }<br>  return undefined; // allow<br>};</pre><p>The model “asked” to run a shell command; your hook quietly refused and handed back a reason, and the loop continued as if nothing happened. You reshaped the agent’s behavior around a loop you never authored.</p><p>That’s the trade you’re making with a harness. You give up the ability to rewrite the control flow, and in exchange you get a working agent fast, strong defaults for the hard parts (context management, compaction, persistence), and a clean set of seams (config, events, hooks) to bend behavior. It’s a great deal precisely when the runtime’s opinions match yours.</p><h3>The framework paradigm: compose the control flow yourself</h3><p>The Python ecosystem took the opposite bet. Instead of a runtime that owns the loop, it offers abstractions you compose. Three matter most.</p><h3>LangChain: the chain</h3><p>LangChain is the oldest and broadest of the three. It began as a way to chain LLM calls together and grew into a sprawling platform: over a thousand integrations, a model-agnostic interface, and a companion observability product (LangSmith). Its agent abstraction is a ReAct-style tool-calling loop, and its composition model pipes components together.</p><p>Its strength is reach: if a model, vector store, or external tool exists, there’s probably a LangChain integration for it. Its weakness is the cost of that generality. Deep abstraction layers can make debugging feel like spelunking, and it’s stateless by default, so durability is something you add rather than something you get. LangChain fits standard, mostly linear single-agent workflows where integration breadth is the thing you care about most.</p><h3>LangGraph: the state machine</h3><p>LangGraph makes a sharp, opinionated bet: agent systems are <strong>state machines</strong>, and you should control every transition. You model the agent as a graph (nodes are steps, edges are transitions) operating over a typed state object.</p><pre>from langgraph.graph import StateGraph, MessagesState, START, END<br><br>def call_model(state: MessagesState):<br>    return {&quot;messages&quot;: [llm.invoke(state[&quot;messages&quot;])]}<br>graph = StateGraph(MessagesState)<br>graph.add_node(call_model)<br>graph.add_edge(START, &quot;call_model&quot;)<br>graph.add_edge(&quot;call_model&quot;, END)<br>app = graph.compile()<br>app.invoke({&quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;hi&quot;}]})</pre><p>Where a harness hides the loop, LangGraph makes it the artifact you author. Conditional branching is explicit:</p><pre>graph.add_conditional_edges(&quot;call_model&quot;, should_continue, {&quot;tools&quot;: &quot;tools&quot;, &quot;end&quot;: END})</pre><p>That should_continue function is the branch the harness ran for you behind prompt(). Here it&#39;s yours: visible, testable, versionable.</p><p>The payoff for that verbosity is LangGraph’s killer feature: <strong>native checkpointing at every transition</strong>. State is persisted as the graph runs, which gives you durable execution, crash recovery, time-travel debugging, and the ability to pause for human input mid-run. Its costs are a steep learning curve and a design that is deliberately <em>not</em> serverless-friendly. LangGraph fits long-running, cyclic, crash-proof, stateful workflows.</p><h3>CrewAI: the team</h3><p>CrewAI bets that most multi-agent problems map cleanly onto <strong>teams of specialists</strong>. You give each agent a role, a goal, and a backstory, then assemble them into a crew with tasks.</p><pre>from crewai import Agent, Task, Crew<br><br>researcher = Agent(role=&quot;Researcher&quot;, goal=&quot;Find key insights on {topic}&quot;,<br>                   backstory=&quot;A meticulous analyst.&quot;)<br>writer = Agent(role=&quot;Writer&quot;, goal=&quot;Summarize the research clearly&quot;,<br>               backstory=&quot;A concise technical writer.&quot;)<br><br>research = Task(description=&quot;Research {topic}.&quot;, expected_output=&quot;3 insights.&quot;, agent=researcher)<br>summary = Task(description=&quot;Write a summary.&quot;, expected_output=&quot;A short summary.&quot;, agent=writer)<br><br>Crew(agents=[researcher, writer], tasks=[research, summary]).kickoff(inputs={&quot;topic&quot;: &quot;agent frameworks&quot;})</pre><p>The mental model is intuitive and the path to a working multi-agent pipeline is short, which is why it sees heavy enterprise adoption. The flip side: the role-and-goal abstraction is convenient until something goes wrong, and then it’s a black box; delegation between agents can loop; and you have to add your own guardrails around runaway cost. CrewAI fits role-based multi-agent systems and fast prototyping.</p><p>The thread connecting all three: <strong>you are the author of the control flow.</strong> That’s the cost (more code, more decisions, more to maintain) and the benefit (you can see, test, and reshape every transition). Python’s gravitational pull in data and ML is a real tailwind, since your agent lives next to the libraries and notebooks you already use.</p><h3>The trade-offs that actually decide it</h3><p>Lined up against each other, the paradigms diverge along a handful of dimensions that tend to decide real projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0YboWwTN9WDKNARwoSq19w.png" /></figure><p>A few of these deserve more than a cell.</p><p><strong>Agent-loop ownership</strong> is the axis everything else hangs from. With a harness, the loop is the runtime’s; with a framework, it’s yours. LangGraph sits at the far “you own it” end, Pi at the far “runtime owns it” end, and the others in between.</p><p><strong>Durability</strong> is where LangGraph wins decisively. Its checkpoint-every-transition model gives you crash recovery and replay essentially for free. Harness session persistence and CrewAI’s memory types are real, but they are not the same thing as durable, replayable state. If the process dies mid-run, a checkpointed graph can resume where it stopped, while a session log mostly lets you reconstruct what happened.</p><p><strong>Multi-agent orchestration</strong> is native to CrewAI (roles) and LangGraph (subgraphs). A harness does it through sub-agents, extensions, or coordination you write yourself.</p><p><strong>Language ecosystem</strong> is the quiet decider. Python brings data and ML gravity: your agent sits beside pandas, your model code, and your notebooks. TypeScript brings web and edge gravity: your agent sits beside your API, your frontend types, and your serverless deploy target. That pull is often stronger than any single feature comparison, and it’s worth being honest about which one your team already lives in.</p><p>One caveat keeps the table honest: the Node versus Python split is correlation, not law. Harness-style runtimes exist in Python, and graph-style workflow libraries exist in TypeScript. Choose on the paradigm and your constraints, not the language label on the box.</p><h3>One task, two philosophies</h3><p>Take a single, concrete task: <em>an agent that looks something up with a tool, then writes a summary.</em> Watch what each paradigm forces you to think about.</p><p>The <strong>harness</strong> version is mostly <em>configuration</em>. You declared a search_docs tool and a system prompt, then called prompt(). The loop ran itself: call the model, see it ask for search_docs, run it, feed the passages back, let the model write the summary, stop. The question the harness asked you was: <em>what should this agent be able to do?</em></p><p>The <strong>framework</strong> version (LangGraph) is mostly <em>topology</em>. You define the state, a node that calls the model, a node that runs the tool, and a conditional edge that decides whether to loop back to the model or finish. Nothing runs until you’ve drawn the graph. The question LangGraph asked you was: <em>what is the exact control flow?</em></p><p>Here’s the thing worth sitting with: the task is trivial in both. The difference isn’t difficulty. It’s <em>what you’re forced to reason about up front</em>, and <em>what you can inspect or recover later</em>. The harness optimizes the first question away and gives you a clean agent quickly. The framework makes you answer it, and pays you back with a control flow you can step through transition by transition when something goes sideways at 2 a.m.</p><p>That tension scales with the system. For a coding assistant or an agent embedded in an app, “what should it be able to do?” is the right question and the harness is a gift. For a long-running workflow that must survive crashes and pass audits, “what is the exact control flow?” is the right question and the explicit graph earns its verbosity.</p><h3>How to choose</h3><p>Match the tool to the loop you actually need to own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2nVV48Uvif6N8oP5h-Ezw.png" /></figure><p>In prose, the rules of thumb:</p><ul><li><strong>Reach for a harness (Pi)</strong> when you want a working agent fast, the runtime’s defaults match your needs, you’re building a coding or dev-tool agent, or you’re embedding an agent inside a TypeScript app and value type safety and co-location with your frontend.</li><li><strong>Reach for LangGraph</strong> when durability is non-negotiable: long-running or cyclic workflows that must survive crashes, demand precise control over state and transitions, or need a human in the loop.</li><li><strong>Reach for CrewAI</strong> when your problem genuinely maps to a team of specialist roles and you want a multi-agent pipeline quickly.</li><li><strong>Reach for LangChain</strong> when integration breadth for a standard single-agent, tool-calling workflow matters more than anything else, especially inside an existing Python stack.</li></ul><p>And the honest caveat once more: these lines blur. Choose on the paradigm and the constraints, not the language label.</p><h3>The loop is the decision</h3><p>The real question was never Node versus Python, or library X versus library Y. It was whether you want a runtime to own the agent loop or you want to own it yourself.</p><p>A harness like Pi optimizes for speed-to-agent and for reshaping behavior through config, events, and hooks, bending a loop you didn’t write. A framework like LangGraph, CrewAI, or LangChain optimizes for explicit control, durability, and orchestration, at the price of authoring and maintaining the control flow yourself.</p><p>A reasonable default for most teams: start with a harness for prototypes, coding agents, and anything embedded in an app, where moving fast is the win. Graduate to LangGraph when state and durability become hard requirements. Pick CrewAI when the problem is genuinely a team of specialists. The tools are converging on each other’s strengths, so the lasting skill isn’t memorizing one API. It’s recognizing, for the agent in front of you, who should own the loop.</p><p><em>If this was useful, a few claps and a follow help more readers find it. I’d also like to hear which side of the loop your team landed on, and why.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c6a153178b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Needs Specs, Not Just Prompts]]></title>
            <link>https://matheusjerico.medium.com/ai-needs-specs-not-just-prompts-846b4f033191?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/846b4f033191</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[spec-driven-development]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 21:31:00 GMT</pubDate>
            <atom:updated>2026-03-23T21:31:00.678Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Why the bottleneck in AI-assisted development is not code generation: it is specification.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QmyZxSFjfwBRBua8oiiGg.png" /></figure><h3>The Paradox</h3><p>AI can write code faster than any human alive. It is real, it is true.</p><p>In the time it takes you to read this sentence, a large language model could generate an entire module, complete with error handling, type hints, and docstrings. Code generation, the thing that consumed most of a developer’s day for decades, is effectively a solved problem. You, Senior Software Engineer, need to leave your ego at the door, because AI writes code better than you do.</p><p>So why are most AI-assisted projects still a mess?</p><p>I do not mean the code itself. Line for line, AI-generated code is often fine. Sometimes it is better than what I’d write manually. The problem is what happens when you zoom out. The files do not connect cleanly. The architecture feels accidental. The same decision gets made three different ways across three different conversations. A month later, you can barely explain what the system does, even though you “built” it.</p><p>We talk about AI-assisted development as if the hard part is behind us. It is not. We automated the easy part (turning clear intent into code) and left the hard part untouched.</p><p>Maybe the problem was never about writing code.</p><h3>Vibe Coding — And Why It Works</h3><p>There is a workflow that most of us have settled into with AI tools. You open a chat, describe what you want in natural language, get code back, paste it in, run it, and iterate. If it does not work, you feed the error back to the AI and try again. If the context gets too long, you start a new conversation.</p><p>Andrej Karpathy called this <strong>vibe coding </strong>and the name stuck because it is exactly what it feels like. You are not engineering. You are vibing. You are steering by feel, adjusting on the fly, trusting the AI to fill in the gaps.</p><p>And for a lot of tasks, it is genuinely great.</p><p>Vibe coding is perfect for prototyping an idea in twenty minutes instead of two hours. It is how you explore an unfamiliar API without reading documentation. It is how you write a one-off script to clean up a dataset, learn a new language by building something small, or spike a proof-of-concept to show your team. For contained, low-stakes tasks, vibe coding is not just good enough; it is the right approach.</p><p>The revolution is real. People who never wrote code before are shipping products. Developers are moving faster on routine work. Entire categories of “glue code” that used to take hours now take minutes. The barrier to entry has never been lower, and that is a genuinely good thing.</p><p>I say this without irony: vibe coding has made me a better developer for certain tasks. It freed up mental energy I used to spend on boilerplate so I could think about design instead. That matters.</p><h3>The Ceiling</h3><p>Now picture something different. You are not writing a script. You are building a system.</p><p>You start the way you always do: open a chat, describe the first component, get code back. It works. Second component, same thing. It works too. Third component… and now things get interesting. The data model from component one does not match what component three expects. You refactor. You re-prompt. You paste context from the first conversation into the third. The AI gives you something that looks right, until you realize it made different assumptions about error handling than the code it generated yesterday.</p><p>You start a new conversation because the old one is too long. But now the AI does not know about the decisions you made two hours ago. So you explain everything again. And it makes slightly different choices this time.</p><p>Each file works in isolation. The system does not cohere. And the frustrating part is you can not point to a single bad file. They are all fine. The problem is not any one piece: it is that nobody defined how the pieces fit together.</p><p>This happens because the AI is optimizing each response independently. It does not know your architectural constraints. It does not know what you decided last Tuesday. It does not know what is out of scope. It does not have a shared data model to reference. Every conversation starts from zero, and the AI fills in the blanks with its best guess, which is a different guess every time.</p><p>That distance between what you prompted and what a coherent system actually requires? I call it the <strong>specification gap</strong>.</p><p>It is the space where architectural intent lives. Where shared data models are defined. Where scope boundaries are drawn. Where decisions are recorded so they do not have to be re-made.</p><p>Vibe coding has nothing to offer here. Not because it is bad, but because it was never designed for this. It is like navigating a city by feel: fine for a few blocks, dangerous for a cross-country trip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*psxoNtN6Mw2N09amuo51DQ.png" /><figcaption>The Specification Gap: the distance between what you prompted and what a coherent system requires</figcaption></figure><blockquote><em>The specification gap grows with complexity. One file, no gap. Ten files, and you’re building a house room by room without blueprints.</em></blockquote><h3>The Insight</h3><p>For decades, the bottleneck in software development was writing code. Developers spent most of their time translating requirements into syntax: typing, debugging, refactoring. AI removed that bottleneck almost overnight.</p><p>But removing one bottleneck revealed the real one behind it: <strong>knowing what to build</strong>.</p><p>A prompt tells the AI what you want <em>right now</em>, in <em>this conversation</em>. A specification tells the AI what the system needs to be. Those are fundamentally different things.</p><p>A spec answers the questions a prompt never asks. Where does this code live in the project? What data model does it share with other components? What are the acceptance criteria? What did we explicitly decide <em>not</em> to build? How will we know when it is done?</p><p>You do not need a formal document with Roman numerals and signature blocks. A spec can be three paragraphs: what you are building, what you are not, and how the pieces connect. The point is not ceremony. It is intent.</p><blockquote><em>A prompt is a request. A spec is a contract.</em></blockquote><p>When you give AI a spec instead of a prompt, you are not adding bureaucracy. You are giving it the one thing it ca not infer from a chat window: the full picture. And when the AI has the full picture, it stops guessing.</p><p>The bottleneck was never code generation. It was specification. And specs are the one thing you can not vibe your way through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ibk0nCj83xTZdqKlwbgbZw.png" /><figcaption>The bottleneck shifted: AI solved code generation, but revealed that specification was the real constraint all along</figcaption></figure><h3>Evidence</h3><p>I tested this on a real project. Not a weekend experiment, a production system that my team depends on daily.</p><p>The project is a multiagent LLM system that processes real-time operational data and returns structured diagnostics. Multiple services, external integrations, structured LLM output, and a comprehensive test suite. The kind of system that would absolutely collapse under vibe coding.</p><p>I used a spec-driven approach. Before writing any code, I followed a structured process for each feature: brainstorm the idea, define the requirements, design the architecture, build it, ship it. Each phase produced a document. Each document fed the next phase. The AI always knew what it was building, what the boundaries were, and how the current piece connected to everything else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*lbFMNmPpE2HTPFgVxjbiyA.png" /><figcaption>Vibe Coding vs Spec-Driven: the chaotic prompt-iterate loop versus the linear spec-driven flow</figcaption></figure><p>The results surprised me.</p><p>The core system (44 files, roughly 5000 lines of code, 182 tests) was built in a single day. Not a cowboy coding day. A spec-driven day. By the time the AI started writing code, every decision had already been made: what to build, what to skip, where each file lives, how the components connect. The AI was not guessing. It was executing.</p><p>Over seven weeks, I shipped 11 features using this approach. The test suite grew to 363 tests. Every feature has a complete trail: what was brainstormed, what was decided, what was deliberately cut, and what was learned.</p><p>That last part matters more than the speed. Each shipped feature captured lessons: what went well, what went wrong, what to do differently next time. Those lessons fed into the next feature’s spec. The system did not just grow. It got smarter about how it grows.</p><p>Adding structure did not slow me down. It made the AI dramatically more effective, because it always knew exactly what to build, where to put it, and when to stop.</p><p>Here is what I did not expect: the spec-driven approach also changed how <em>I</em> thought about features. Writing a spec forced me to confront ambiguity before it became code. Questions that would have surfaced as bugs (“wait, what happens when this input is empty?”) surfaced as line items in the requirements instead. The spec was a conversation with myself before it became a conversation with the AI.</p><h3>What This Means for You</h3><p>Vibe coding is not wrong. It is a starting point.</p><p>For a script, a prototype, an exploration, keep vibing. That is where it shines. But the next time you open your AI tool to start building something real, something with multiple components that need to work together, pause before you type that first prompt.</p><p>Ask yourself: do I know what “done” looks like? Can I describe what I am building, what I am <em>not</em> building, and how the pieces connect?</p><p>If not, try writing it down first. Even a few paragraphs. Define the problem. List what is in scope and what is out. Sketch the data model. Name the components. Decide what you are deliberately not building (because scope is a feature, not a constraint).</p><p>You are not adding overhead. You are closing the specification gap. You are giving the AI the one thing it cannot generate for itself: intent.</p><p>The tools will keep getting faster. The models will keep getting smarter. But the gap between a prompt and a coherent system won’t close on its own. That is your job. And a spec (even a simple one) is how you do it.</p><blockquote><em>AI can write the code. But only you can write the spec.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=846b4f033191" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Delta Lake to Apache Iceberg: A Complete Migration Blueprint for Modern Data Platforms]]></title>
            <link>https://blog.dataengineerthings.org/from-delta-lake-to-apache-iceberg-a-complete-migration-blueprint-for-modern-data-platforms-400ce1e996f5?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/400ce1e996f5</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 19:40:49 GMT</pubDate>
            <atom:updated>2025-12-12T18:16:20.720Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>If you’re using Delta Lake and need ad-hoc interactive SQL with engines like Trino or Presto, this guide is essential reading.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CFlwuXX5tai8KM1FMpF5ZQ.png" /></figure><h3>1. Why Delta Lake to Apache Iceberg Migration Has Become a Strategic Decision</h3><p>Imagine this: your data platform started on Delta Lake. It served you well: ACID transactions, time travel, streaming + batch unification. Over time, your data grows. New analytics engines are introduced — maybe Trino, Presto, or Flink. Business requirements shift, data access expands to more teams, perhaps you’re moving away from a vendor-locked environment (e.g. no longer tied to a single cloud or to a proprietary runtime).</p><p>You begin to notice: with Delta Lake, you’re locked to Spark (or the Delta protocol) for most features; cross-engine interoperability is limited. Also, metadata scaling becomes a concern — huge tables with many partitions/files start to slow down planning or maintenance.</p><p>At that moment, you realize: migrating to Apache Iceberg isn’t just a “format change” — it’s an architectural shift. It’s a move toward an <strong>open, multi-engine, future-proof Lakehouse</strong>.</p><p>In this article, I walk you through a <strong>complete migration blueprint</strong>: when to migrate, how, best practices, pitfalls, and how to operate Iceberg tables at scale post-migration.</p><h3>2. Delta Lake vs Apache Iceberg — A Deep Architectural Comparison</h3><p>Migration only makes sense if you understand the mechanics behind both formats. Let’s break them down properly.</p><h4><strong>2.1 Delta Lake — Strengths, Architecture, and Limitations</strong></h4><p>Delta Lake is built around a <strong>transaction log</strong> (_delta_log/) that tracks every table modification. It uses:</p><ul><li><strong>JSON log entries</strong> for each commit</li><li><strong>Checkpoint parquet files</strong> as compacted snapshots</li><li><strong>Underlying Parquet files</strong> as storage</li></ul><p>Delta is tightly integrated with Spark. That’s a blessing for adoption but a limitation for multi-engine environments.</p><p>✔ <strong>Strengths</strong></p><ul><li>Simple architecture (log + checkpoint)</li><li>Fast adoption curve</li><li>Excellent integration with Spark APIs</li><li>Strong support for upserts, deletes, merges</li><li>Time travel via log history</li></ul><p>✖ <strong>Limitations</strong></p><ul><li>Metadata scaling challenges for very large tables</li><li>Planning latency can increase with file count</li><li>Multi-engine interoperability is <strong>limited</strong></li><li>Vendor ecosystem is <strong>narrow</strong></li></ul><h4><strong>2.2 Apache Iceberg — A Table Format Designed for Scale &amp; Multi-Engine Workloads</strong></h4><p>Iceberg takes a <strong>very different approach</strong>. Instead of one transaction log, it uses a <strong>metadata tree</strong>.</p><p><strong>Iceberg’s Metadata Hierarchy:</strong></p><ol><li><strong>Table metadata file</strong> (metadata.json)</li><li><strong>Snapshots</strong></li><li><strong>Manifest lists</strong></li><li><strong>Manifest files</strong></li><li><strong>Data files &amp; Delete files</strong></li></ol><p>This structure provides:</p><ul><li>Efficient metadata pruning</li><li>Predictable planning time</li><li>Clear isolation of metadata vs data</li><li>Multi-engine compatibility</li><li>Support for delete files (position/equality deletes)</li></ul><h4>2.3 Diagram — Delta vs Iceberg Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QFTIYuUMzM_ye6evSmPHcw.png" /></figure><h4>2.4 Key Differences That Matter for Migration</h4><p><strong>Metadata Scaling</strong></p><ul><li><strong>Delta</strong>: metadata grows linearly with commits</li><li><strong>Iceberg</strong>: metadata is hierarchical =&gt; more efficient for large tables</li></ul><p><strong>Multi-Engine Compatibility</strong></p><p>Iceberg supports:</p><ul><li>Spark</li><li>Trino</li><li>Presto</li><li>Flink</li><li>Snowflake</li><li>BigQuery</li><li>Dremio</li><li>StarRocks</li></ul><p>Delta supports:</p><ul><li>Spark (full)</li><li>Some engines with limited functionality</li></ul><p><strong>Partitioning</strong></p><ul><li>Delta: directory-based</li><li>Iceberg: <strong>hidden partitioning</strong>, <strong>partition transforms</strong>, <strong>partition evolution</strong></li></ul><p><strong>Deletes</strong></p><ul><li>Delta: copy-on-write or vectorized deletes</li><li>Iceberg: equality &amp; position delete files (MERGE-on-read natural design)</li></ul><p><strong>Governance</strong></p><p>Iceberg is <strong>Apache-governed</strong> and vendor-neutral.</p><h3>3. Should You Migrate? A Decision Framework for Real Data Platforms</h3><p>Migration is a cost. It takes engineering time, adds operational risk, and requires coordination.</p><p>So the first strategic question is:<br><strong>Do you even need to migrate?</strong></p><h4>3.1 When Migration Does Make Sense</h4><p>✔ You are adopting <strong>Trino, Snowflake, BigQuery, or Flink</strong> and want a unified table layer. <em>It’s my case.</em><br>✔ You expect data sizes to grow significantly<br>✔ Your Delta tables have:</p><ul><li>high file counts</li><li>planning latency issues</li><li>metadata bloat</li></ul><p>✔ You want <strong>open governance</strong><br>✔ Your platform is evolving toward a <strong>multi-engine Lakehouse</strong></p><h4>3.2 When You Should Stay on Delta</h4><p>✔ You cannot afford the migration overhead today<br>✔ Your workloads are Spark-centric and stable<br>✔ You rely on:</p><ul><li>Delta Live Tables</li><li>Photon engine</li><li>Unity Catalog Delta-specific governance</li></ul><h4>3.3 Migration Decision Diagram</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*zGPfKreSm1JlHmEtP3Bewg.png" /></figure><h3>4. Migration Building Blocks — What You Need Before You Start</h3><p>Before touching any data, you must setup the foundation. This is where many teams fail.</p><p>Migration is not “convert format =&gt; done”.</p><p>It requires:</p><ul><li>Environment prep</li><li>Catalog configuration</li><li>Compute readiness</li><li>Write-path analysis</li><li>Table classification</li><li>Governance planning</li><li>And communication with all downstream consumers</li></ul><p>Let’s break down the essentials.</p><h4>4.1 Spark Configuration — Enabling Delta and Iceberg Side-by-Side</h4><p>You need Spark to support <strong>both</strong> formats simultaneously during migration.</p><p><strong>4.1.1 Delta Configuration</strong></p><p>This is typically your existing config:</p><pre>from pyspark.sql import SparkSession<br><br>spark = (<br>    SparkSession.builder<br>    .appName(&quot;DeltaProduction&quot;)<br>    .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;)<br>    .config(<br>        &quot;spark.sql.catalog.spark_catalog&quot;,<br>        &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;,<br>    )<br>    .getOrCreate()<br>)</pre><p><strong>4.1.2 Add Iceberg Support in the Same Spark Session</strong></p><pre>spark = (<br>    SparkSession.builder<br>    .appName(&quot;DeltaToIceberg&quot;)<br>    .config(<br>        &quot;spark.sql.extensions&quot;,<br>        &quot;io.delta.sql.DeltaSparkSessionExtension,&quot;<br>        &quot;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot;<br>    )<br>    .config(&quot;spark.sql.catalog.spark_catalog&quot;, &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;)<br><br>    # Iceberg catalog<br>    .config(&quot;spark.sql.catalog.lh&quot;, &quot;org.apache.iceberg.spark.SparkCatalog&quot;)<br>    .config(&quot;spark.sql.catalog.lh.type&quot;, &quot;hadoop&quot;)<br>    .config(&quot;spark.sql.catalog.lh.warehouse&quot;, &quot;s3://my-warehouse/&quot;)<br><br>    .getOrCreate()<br>)</pre><p>You can now:</p><ul><li>Read Delta tables normally</li><li>Create or write Iceberg tables using lh.db.table notation</li></ul><h4>4.2 Catalogs — The Brain of Apache Iceberg</h4><p>Iceberg supports multiple types of catalogs:</p><p><strong>✔ Hadoop Catalog</strong></p><ul><li>Simple, filesystem-based</li><li>Good for POCs or single-cluster setups</li></ul><p><strong>✔ Hive Catalog</strong></p><ul><li>Stores metadata in Hive Metastore</li><li>Useful if you already use HMS</li></ul><p><strong>✔ REST Catalog (Recommended for Production)</strong></p><ul><li>Decoupled metadata</li><li>Multi-engine, multi-cluster</li><li>Better concurrency &amp; scalability</li></ul><p>During migration, most teams use:</p><ul><li>Hadoop/Hive for dev/test</li><li>REST catalog for production</li></ul><h4>4.3 Classifying Tables Before Migration (Critical Step)</h4><p>Every table should be labeled before migration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*zZwaEudip0gFRa5g-XP55Q.png" /></figure><p>This classification determines your migration strategy.</p><h4>4.4 Pre-Migration Diagram — Table Readiness Flow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*ZH10fxJO7mZP2zQBbxr8Gg.png" /></figure><h4>4.5 Validating Your Environment</h4><p>Before migrating even one row, validate:</p><p><strong>Storage</strong></p><ul><li>Ensure S3/GCS/ADLS permissions for reading Delta and writing Iceberg</li><li>Validate warehouse paths</li><li>Confirm no cross-account restrictions</li></ul><p><strong>Compute</strong></p><ul><li>Enough Spark cluster resources to handle backfills</li><li>Executor memory adjustments for large manifest generation</li></ul><p><strong>Monitoring</strong></p><p>Enable metrics for:</p><ul><li>file counts</li><li>snapshot counts</li><li>manifest counts</li><li>query latency</li></ul><p><strong>Governance &amp; Permissions</strong></p><ul><li>Catalog-level access</li><li>Table-level permissions</li><li>Downstream consumer mapping</li></ul><h3>5. Migration Strategies — The Four Practical Patterns</h3><p>In real-world platforms, migrations happen under constraints: uptime requirements, table size, compute budgets, consumer dependencies, schema stability, governance rules.</p><p>Because of that, there is no universal method.</p><p>Across companies, the same <strong>four patterns</strong> emerge over and over again.</p><p>Let’s study each one.</p><h4>5.1 Strategy 1 — Metadata Snapshot (Iceberg Delta Migration Module)</h4><p><strong><em>Minimal rewrite, preserves history</em></strong></p><p>This is the <strong>most elegant</strong> approach when it works.</p><p>Iceberg’s Delta migration module can:</p><ul><li>Read the _delta_log</li><li>Replay commits</li><li>Build Iceberg metadata (snapshots, manifests)</li><li>Reuse existing Parquet data files</li></ul><p><strong>✔ Strengths</strong></p><ul><li>Extremely fast</li><li>No data rewrite =&gt; minimal cost</li><li>Preserves historical time travel</li><li>Low operational risk</li></ul><p><strong>✖ Limitations</strong></p><ul><li>Only works for supported Delta versions &amp; protocol features</li><li>Does not optimize file layout</li><li>Does not fix existing small-file problems</li><li>Requires careful compatibility validation</li></ul><p><strong>Diagram — Metadata Snapshot Migration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y4EVG5tijAV_i5GwrmZ4Zg.png" /></figure><p>This works best for stable, append-only, or lightly mutated tables where file layout has not degraded significantly.</p><h4>5.2 Strategy 2 — Full Rewrite (Read Delta =&gt; Write Iceberg)</h4><p><strong><em>Rebuild the table with a clean schema &amp; partition layout</em></strong></p><p>This is the most <strong>predictable</strong> and <strong>commonly used</strong> strategy.</p><p>You simply:</p><ol><li>Read from the Delta table</li><li>Write into a new Iceberg table with your desired:</li></ol><ul><li>schema</li><li>partition spec</li><li>table properties (format version, default file sizes)</li></ul><p><strong>✔ Strengths</strong></p><ul><li>Completely clean slate</li><li>Fix poor historical file layout</li><li>Ideal for medium-size tables</li><li>Full control over partitioning</li></ul><p><strong>✖ Limitations</strong></p><ul><li>Costly for very large tables (I/O + storage)</li><li>Does not preserve full Delta log history</li></ul><p><strong>Diagram — Full Rewrite Migration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6XTTjzstltMVKDXGgbwDQ.png" /></figure><p>This is a great option for 1–20 TB tables.</p><h4>5.3 Strategy 3 — Dual Write (Forward Only)</h4><p><strong><em>For tables where downtime is impossible</em></strong></p><p>You configure pipelines so <strong>new data</strong> is written simultaneously to:</p><ul><li>Delta (legacy)</li><li>Iceberg (new target)</li></ul><p>During a transition period:</p><ul><li>Consumers continue using Delta</li><li>Platform teams validate Iceberg</li><li>Once confidence is high → consumer cutover</li></ul><p><strong>✔ Strengths</strong></p><ul><li>Lowest risk</li><li>No downtime</li><li>Allows long validation windows</li></ul><p><strong>✖ Limitations</strong></p><ul><li>Complex write pipelines</li><li>Must ensure consistency between both tables</li><li>Requires additional compute/storage during transition</li></ul><p><strong>Diagram — Dual Write Pattern</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tafbB0MGAXD2w2PT4zVU2A.png" /></figure><p>This method shines for critical, frequently updated tables.</p><h4>5.4 Strategy 4 — Backfill + CDC Replay (Enterprise-Grade)</h4><p><strong><em>The gold standard for large mutable tables</em></strong></p><p>This is the recommended pattern for:</p><ul><li>Highly mutable tables</li><li>Fact tables with late-arriving data</li><li>Slowly changing dimension tables</li><li>Very large datasets (&gt;20 TB)</li><li>Systems requiring high correctness guarantees</li></ul><p><strong>Workflow Overview</strong></p><ol><li><strong>Backfill</strong> historical Delta =&gt; Iceberg in partitions/batches</li><li><strong>Capture CDC</strong> from upstream systems or Delta change feed</li><li><strong>Replay changes</strong> into Iceberg via MERGE</li><li><strong>Validate</strong></li><li><strong>Cutover</strong></li><li><strong>Decommission Delta</strong></li></ol><p><strong>Diagram — Backfill + CDC Migration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*gKYdmePEzJIVDwkuP_02Gg.png" /></figure><p>This method gives you a <strong>zero-data-loss, highly controlled, fully validated</strong> migration.</p><h3>6. Hero Workflow — End-to-End Backfill + CDC Migration (Deep Dive)</h3><p>This is the flagship section: the migration strategy used by high-scale data platforms.</p><h4>6.1 Step 1 — Inventory &amp; Planning</h4><p>Collect per-table metadata:</p><ul><li>Size (GB / TB)</li><li>Number of files</li><li>Partition columns</li><li>Update frequency</li><li>Downstream consumers</li><li>SLA sensitivity</li><li>Schema evolution history</li></ul><p>Tables should be assigned migration strategies individually.</p><h4>6.2 Step 2 — Create the New Iceberg Table</h4><p>Example using Spark SQL:</p><pre>CREATE TABLE lh.analytics.events (<br>  event_id STRING,<br>  user_id STRING,<br>  event_time TIMESTAMP,<br>  event_type STRING,<br>  payload STRING<br>)<br>USING iceberg<br>PARTITIONED BY (days(event_time), bucket(16, user_id))<br>TBLPROPERTIES (<br>  &#39;format-version&#39;=&#39;2&#39;,<br>  &#39;write.target-file-size-bytes&#39;=&#39;536870912&#39;<br>);</pre><h4>6.3 Step 3 — Historical Backfill (Partitioned Approach)</h4><p><strong>Python Template — Backfill per Day Partition</strong></p><pre>from pyspark.sql import SparkSession, functions as F<br><br>spark = SparkSession.builder.getOrCreate()<br><br>source = &quot;s3://delta/events&quot;<br>target = &quot;lh.analytics.events&quot;<br><br># Extract partitions<br>dates = (<br>    spark.read.format(&quot;delta&quot;).load(source)<br>    .select(F.to_date(&quot;event_time&quot;).alias(&quot;dt&quot;))<br>    .distinct()<br>    .collect()<br>)<br><br>for row in dates:<br>    dt = row[&quot;dt&quot;]<br>    print(f&quot;Backfilling date={dt}&quot;)<br><br>    (<br>        spark.read.format(&quot;delta&quot;).load(source)<br>        .filter(F.to_date(&quot;event_time&quot;) == dt)<br>        .writeTo(target)<br>        .append()<br>    )</pre><p><strong>Notes</strong></p><ul><li>This avoids large shuffles</li><li>Allows parallelization</li><li>Makes retries easier</li><li>Supports validation per partition</li></ul><h4>6.4 Step 4 — CDC Ingestion Pipeline</h4><p>You apply inserts/updates/deletes via Iceberg’s MERGE.</p><p><strong>Example Spark SQL Job</strong></p><pre>MERGE INTO lh.analytics.events t<br>USING lh.staging.events_cdc s<br>ON t.event_id = s.event_id<br>WHEN MATCHED AND s.op = &#39;DELETE&#39; THEN DELETE<br>WHEN MATCHED AND s.op = &#39;UPDATE&#39; THEN<br>  UPDATE SET *<br>WHEN NOT MATCHED AND s.op IN (&#39;INSERT&#39;,&#39;UPSERT&#39;) THEN<br>  INSERT *</pre><p><strong>Spark Structured Streaming Version (Template)</strong></p><pre>cdc = (<br>    spark.readStream.format(&quot;delta&quot;)<br>    .load(&quot;s3://cdc/events&quot;)<br>)<br><br>def upsert_to_iceberg(batch, batch_id):<br>    batch.createOrReplaceTempView(&quot;cdc_batch&quot;)<br>    spark.sql(&quot;&quot;&quot;<br>        MERGE INTO lh.analytics.events t<br>        USING cdc_batch s<br>        ON t.event_id = s.event_id<br>        WHEN MATCHED AND s.op = &#39;DELETE&#39; THEN DELETE<br>        WHEN MATCHED AND s.op = &#39;UPDATE&#39; THEN UPDATE SET *<br>        WHEN NOT MATCHED THEN INSERT *<br>    &quot;&quot;&quot;)<br><br>cdc.writeStream.foreachBatch(upsert_to_iceberg).start()</pre><p>This ensures Iceberg never drifts from Delta during backfill.</p><h4>6.5 Step 5 — Validation Framework</h4><p>Validation must be strict, automated, and multi-dimensional.</p><p><strong>Validation Template — Partition-Level Checksums</strong></p><pre>from pyspark.sql import functions as F<br><br>delta_df = spark.read.format(&quot;delta&quot;).load(source)<br>ice_df = spark.read.format(&quot;iceberg&quot;).load(target)<br><br>delta_hash = (<br>    delta_df.groupBy(F.to_date(&quot;event_time&quot;).alias(&quot;dt&quot;))<br>    .agg(F.sha2(F.concat_ws(&quot;||&quot;, *delta_df.columns), 256).alias(&quot;h&quot;))<br>)<br><br>ice_hash = (<br>    ice_df.groupBy(F.to_date(&quot;event_time&quot;).alias(&quot;dt&quot;))<br>    .agg(F.sha2(F.concat_ws(&quot;||&quot;, *ice_df.columns), 256).alias(&quot;h&quot;))<br>)<br><br>delta_hash.join(ice_hash, &quot;dt&quot;, &quot;fullouter&quot;).show()</pre><p>You should compare:</p><ul><li>Row counts</li><li>Distinct keys</li><li>Checksums</li><li>Sampled queries</li><li>Distribution statistics</li></ul><h4><strong>6.6 Step 6 — Consumer Cutover</strong></h4><p>Plan the cutover carefully:</p><ul><li>Switch lowest-risk consumers first</li><li>Validate dashboards / ML pipelines</li><li>Update Trino/Presto catalogs</li><li>Update Snowflake external volumes if applicable</li></ul><p>You can cut over gradually in a “shadow read” mode.</p><h4>6.7 Step 7 — Delta Decommissioning</h4><p>Do <strong>NOT</strong> delete Delta immediately.</p><p>Best practice is:</p><ul><li>Keep Delta tables read-only for 30–90 days</li><li>Disable writes</li><li>Archive _delta_log for compliance</li><li>After retention window =&gt; clean storage</li></ul><h3>7. Post-Migration Operations — Running Iceberg in Production</h3><p>Just like Delta, Iceberg requires ongoing care.</p><h4>7.1 Compaction — rewrite_data_files</h4><p><strong>Iceberg Built-in Procedure</strong></p><pre>CALL lh.system.rewrite_data_files(<br>  table =&gt; &#39;analytics.events&#39;,<br>  strategy =&gt; &#39;binpack&#39;,<br>  options =&gt; map(<br>    &#39;target-file-size-bytes&#39;, &#39;536870912&#39;,<br>    &#39;min-input-files&#39;, &#39;5&#39;<br>  )<br>);</pre><h4><strong>7.2 Metadata Management — </strong><strong>rewrite_manifests</strong></h4><p>Keeps planning fast:</p><pre>CALL lh.system.rewrite_manifests(<br>  table =&gt; &#39;analytics.events&#39;,<br>  use_caching =&gt; true<br>);</pre><h4>7.3 Snapshot Cleanup — expire_snapshots</h4><pre>CALL lh.system.expire_snapshots(<br>  table =&gt; &#39;analytics.events&#39;,<br>  retain_last =&gt; 20<br>);</pre><h4>7.4 Orphan File Removal</h4><pre>CALL lh.system.remove_orphan_files(<br>  table =&gt; &#39;analytics.events&#39;,<br>  older_than =&gt; TIMESTAMP &#39;2025-01-01 00:00:00&#39;<br>);</pre><h4>7.5 Automated Maintenance Framework (Template)</h4><p><strong>Sample Python Driver for Airflow / Cron</strong></p><pre>def maintain(table):<br>    spark.sql(f&quot;&quot;&quot;<br>        CALL lh.system.rewrite_data_files(<br>            table =&gt; &#39;{table}&#39;,<br>            strategy =&gt; &#39;binpack&#39;<br>        )<br>    &quot;&quot;&quot;)<br><br>    spark.sql(f&quot;&quot;&quot;<br>        CALL lh.system.rewrite_manifests(<br>            table =&gt; &#39;{table}&#39;<br>        )<br>    &quot;&quot;&quot;)<br><br>    spark.sql(f&quot;&quot;&quot;<br>        CALL lh.system.expire_snapshots(<br>            table =&gt; &#39;{table}&#39;,<br>            retain_last =&gt; 15<br>        )<br>    &quot;&quot;&quot;)</pre><h3>8. Multi-Engine SQL Examples (Spark, Trino, Snowflake)</h3><h4>Spark</h4><pre>SELECT event_type, COUNT(*)<br>FROM lh.analytics.events<br>WHERE event_time &gt;= current_date() - INTERVAL 7 DAYS<br>GROUP BY event_type;</pre><h4>Trino</h4><pre>SELECT user_id, count(*)<br>FROM iceberg.lh.analytics.events<br>WHERE event_type = &#39;purchase&#39;<br>GROUP BY user_id;</pre><p>This is one of the biggest reasons to migrate =&gt; <strong>unified storage, many engines</strong>.</p><h3>9. Pitfalls, Lessons, and Hard-Earned Advice</h3><p>From numerous migrations, here are the patterns:</p><p><strong>❌ Partition Specs Gone Wrong</strong></p><p>Iceberg’s partition transforms are powerful. Misuse leads to bad pruning.</p><p><strong>❌ Not Validating Delete Semantics</strong></p><p>Delta deletion vectors ≠ Iceberg delete files.<br>Test deletes carefully.</p><p><strong>❌ Running Compaction Concurrently with Writes</strong></p><p>This can cause conflicts if CDC pipelines are also rewriting files.</p><p><strong>❌ Forgetting Long-Running Queries</strong></p><p>Snapshot expiry must consider max query duration.</p><p><strong>❌ Underestimating Metadata Size</strong></p><p>Iceberg metadata still needs monitoring.<br> Manifest explosion can happen.</p><p><strong>❌ No Consumer Communication</strong></p><p>Downstream teams must update:</p><ul><li>table paths</li><li>partition columns</li><li>governance rules</li><li>catalogs</li></ul><h3>10. Conclusion — Moving Toward an Open, Scalable Lakehouse</h3><p>Migrating from Delta Lake to Apache Iceberg is more than a technical upgrade — it’s a shift toward an open, multi-engine future for the Lakehouse. Delta remains a strong choice for Spark-centric stacks, but Iceberg unlocks broader interoperability, predictable metadata scaling, and cleaner long-term governance.</p><p>The migration itself is not about choosing one “magic” strategy. It’s about combining the right tools — metadata snapshot, rewrite, dual write, or backfill + CDC — with disciplined validation and clear cutover planning. When done well, the transition is predictable, incremental, and fully reversible.</p><p>What Iceberg ultimately delivers is optionality: the freedom to pair your storage with the best compute engine for each job, without being constrained by a single ecosystem.</p><p>The takeaway is simple:</p><blockquote><strong><em>You migrate to Iceberg when the future of your platform demands openness, performance at scale, and multi-engine flexibility.</em></strong></blockquote><p>If your data environment is growing or diversifying, Iceberg gives you room to evolve — without rebuilding everything later.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=400ce1e996f5" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/from-delta-lake-to-apache-iceberg-a-complete-migration-blueprint-for-modern-data-platforms-400ce1e996f5">From Delta Lake to Apache Iceberg: A Complete Migration Blueprint for Modern Data Platforms</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compaction, Vacuum, and Chaos: Operating Delta Lake and Iceberg Tables in Production]]></title>
            <link>https://blog.dataengineerthings.org/compaction-vacuum-and-chaos-operating-delta-lake-and-iceberg-tables-in-production-380ab6d33a02?source=rss-7b3af2736fca------2</link>
            <guid isPermaLink="false">https://medium.com/p/380ab6d33a02</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Matheus Palhares]]></dc:creator>
            <pubDate>Thu, 04 Dec 2025 18:29:05 GMT</pubDate>
            <atom:updated>2025-12-12T04:51:17.437Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Hidden Cost of Lakehouse Tables: A Production Guide to Spark Table Maintenance at Scale</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODrqbnaP00l_LOyyMFKXRg.png" /></figure><h3>1. Opening Hook</h3><p>Last month, a query that had reliably completed in 30 seconds started timing out at 15 minutes. The table hadn’t grown significantly. The spark cluster was healthy. The code hadn’t changed in weeks.</p><p>The culprit? 47000 small files silently accumulated over three months of streaming ingestion! No compaction. No vacuum. No one watching the metrics.</p><p>This isn’t a rare edge case. I’ve seen variations of this story at nearly every organization running lakehouse architectures in production. The pattern is predictable: teams adopt Delta Lake or Apache Iceberg, celebrate the improved reliability over raw Parquet, build pipelines that work beautifully for months — and then watch performance quietly collapse under the weight of unmaintained tables.</p><p>Here’s the uncomfortable truth that rarely appears in lakehouse tutorials: <strong>table formats solved the metadata and ACID problem, but they created an operational problem that most teams discover far too late.</strong></p><p>Delta Lake and Iceberg don’t magically optimize themselves. Every streaming micro-batch, every frequent append, every update operation leaves behind file fragments and metadata entries. Without deliberate maintenance, your lakehouse slowly transforms into a sprawling directory of tiny files that punish every query with excessive I/O overhead.</p><p>This article is the maintenance guide I wish existed when I first hit this wall. We’ll cover the mechanics of table degradation, walk through compaction and vacuum strategies for both Delta Lake and Iceberg, overview about production scheduling patterns, and monitoring patterns that catches problems before your users do.</p><p>The code examples are battle-tested. The pitfalls are ones I’ve encountered firsthand. Let’s fix your tables.</p><h3>2. The Anatomy of Table Degradation</h3><h4>Why Tables Degrade: The Mechanics of Lakehouse Entropy</h4><p>Before diving into solutions, we need to understand exactly why lakehouse tables degrade over time. This isn’t a bug — it’s an inherent consequence of how these systems achieve their guarantees.</p><p>Both Delta Lake and Iceberg implement ACID transactions by writing immutable files. When you update a row, the system doesn’t modify the existing file. Instead, it writes a new file containing the change and updates metadata to reflect which files are current. This immutability is what enables time travel, concurrent reads, and transactional guarantees. It’s also what creates the maintenance burden.</p><p>Three distinct degradation vectors compound over time:</p><figure><img alt="Table degradation cycle" src="https://cdn-images-1.medium.com/max/788/1*GGyxI6aXaw5By-YxyGEutg.png" /><figcaption>Table degradation cycle</figcaption></figure><p><strong>File Proliferation</strong> is the most visible problem. Streaming jobs writing micro-batches every few seconds can generate thousands of files per day, each potentially just a few megabytes or even kilobytes. Frequent UPDATE and MERGE operations create similar fragmentation — every operation that touches existing data produces new files. A table that started with 500 well-sized files can balloon to 50000 fragments within months.</p><p>The impact is severe. Object storage systems like <strong>S3</strong> and <strong>GCS</strong> charge per API request. Listing 50000 files costs 100x more than listing 500. More critically, Spark must open each file individually during query execution. File open operations carry fixed overhead regardless of file size, so reading 1GB from 10000 files is dramatically slower than reading the same 1GB from 10 files.</p><p><strong>Metadata Overhead</strong> grows in parallel with file count. Delta Lake maintains a transaction log (_delta_log/) where every commit appends a new <strong>JSON</strong> file. Iceberg tracks files through manifest files that must be read during query planning. As commits accumulate, the metadata itself becomes a performance bottleneck.</p><p>Consider a table with 100000 commits. Delta Lake must replay the transaction log to reconstruct current table state. While checkpointing mitigates this (writing periodic Parquet snapshots of the log), the checkpoint files themselves grow large. Iceberg’s manifest lists face similar scaling challenges — query planning time increases linearly with manifest count until you run manifest rewriting.</p><p><strong>Orphan Files</strong> are the silent storage leak. When a Spark job fails mid-write, it may leave partial files that never get registered in metadata. Schema evolution operations can orphan files when migrations fail. Even successful operations can leave temporary files behind. These orphans consume storage indefinitely because neither Delta nor Iceberg knows they exist — they’re invisible to standard table operations.</p><p>I’ve read some medium article discussing about tables where orphan files accounted for 25% of total storage. At petabyte scale, that’s real money disappearing into cloud storage bills every month.</p><p>The compounding effect is what catches teams off guard. Each vector reinforces the others. More files mean more metadata entries. More metadata means slower planning. Slower planning means longer job runtimes. Longer runtimes mean more opportunities for failures that create orphans. Without intervention, the system trends toward <strong>chaos</strong>.</p><p>Understanding these mechanics shapes our maintenance strategy. Compaction addresses file proliferation. Vacuum and expire operations clean orphans and obsolete files. Manifest rewriting tackles metadata bloat. Each operation targets a specific degradation vector, and production systems need all three running on appropriate schedules.</p><h3>3. Maintenance Operations Deep-Dive</h3><h4>Maintenance Operations: Compaction, Vacuum, and Metadata Management</h4><p>With the degradation mechanics clear, let’s examine each maintenance operation in detail. I’ll provide working code for both Delta Lake and Iceberg, along with the configuration decisions that matter in production.</p><h4>3.1. Compaction Strategies</h4><p>Compaction consolidates small files into larger, optimally-sized files. The target is typically 128MB to 256MB per file — large enough to amortize file-open overhead, small enough to enable parallelism and predicate pushdown efficiency.</p><p><strong>Delta Lake Compaction</strong></p><p>Delta Lake uses the OPTIMIZE command for compaction. The basic form consolidates all files in a table:</p><pre>from delta.tables import DeltaTable<br><br>spark.sql(&quot;&quot;&quot;<br>    OPTIMIZE delta.`s3://lakehouse/events`<br>&quot;&quot;&quot;)</pre><p>In production, you rarely want to compact an entire table at once. Partition filtering limits the scope to recently modified data, reducing both runtime and resource consumption:</p><pre>spark.sql(&quot;&quot;&quot;<br>    OPTIMIZE delta.`s3://lakehouse/events`<br>    WHERE date &gt;= current_date() - INTERVAL 7 DAYS<br>&quot;&quot;&quot;)</pre><p>The real power comes from clustering. Z-ORDER reorganizes data within files to put rows that share column values, dramatically improving query performance when those columns appear in filters:</p><pre>spark.sql(&quot;&quot;&quot;<br>    OPTIMIZE delta.`s3://lakehouse/events`<br>    WHERE date &gt;= current_date() - INTERVAL 7 DAYS<br>    ZORDER BY (customer_id, event_type)<br>&quot;&quot;&quot;)</pre><p>Z-ORDER creates a space-filling curve that maps multiple dimensions into a single sortable value. Queries filtering on customer_id, event_type, or both will skip far more files after Z-ORDER optimization. Choose columns that frequently appear together in WHERE clauses—typically high-cardinality identifiers and common filter dimensions.</p><p>For programmatic control with metrics, use the Python API:</p><pre>delta_table = DeltaTable.forPath(spark, &quot;s3://lakehouse/events&quot;)<br><br>metrics = (delta_table<br>    .optimize()<br>    .where(&quot;date &gt;= current_date() - INTERVAL 7 DAYS&quot;)<br>    .executeCompaction())<br><br>print(f&quot;Files removed: {metrics[&#39;numFilesRemoved&#39;]}&quot;)<br>print(f&quot;Files added: {metrics[&#39;numFilesAdded&#39;]}&quot;)</pre><p>These metrics feed directly into monitoring systems — a sudden drop in numFilesRemoved might indicate your compaction filter no longer matches your write patterns.</p><p><strong>Iceberg Compaction</strong></p><p>Iceberg provides compaction through the rewrite_data_files procedure. The bin-pack strategy is equivalent to Delta&#39;s basic OPTIMIZE:</p><pre>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.rewrite_data_files(<br>        table =&gt; &#39;db.events&#39;,<br>        strategy =&gt; &#39;binpack&#39;,<br>        options =&gt; map(<br>            &#39;target-file-size-bytes&#39;, &#39;536870912&#39;,<br>            &#39;min-input-files&#39;, &#39;5&#39;,<br>            &#39;max-concurrent-file-group-rewrites&#39;, &#39;10&#39;<br>        ),<br>        where =&gt; &#39;date &gt;= current_date() - INTERVAL 7 DAYS&#39;<br>    )<br>&quot;&quot;&quot;)</pre><p>The min-input-files option prevents unnecessary rewrites—if a file group contains fewer than 5 files, it&#39;s skipped. This avoids churning through already-optimized partitions.</p><p>For query-optimized layouts, use sort-based compaction:</p><pre>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.rewrite_data_files(<br>        table =&gt; &#39;db.events&#39;,<br>        strategy =&gt; &#39;sort&#39;,<br>        sort_order =&gt; &#39;customer_id ASC NULLS LAST, event_time DESC&#39;,<br>        where =&gt; &#39;date &gt;= current_date() - INTERVAL 7 DAYS&#39;<br>    )<br>&quot;&quot;&quot;)</pre><p>Iceberg also supports Z-ORDER through the zorder strategy in recent versions:</p><pre>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.rewrite_data_files(<br>        table =&gt; &#39;db.events&#39;,<br>        strategy =&gt; &#39;zorder&#39;,<br>        options =&gt; map(<br>            &#39;zorder-columns&#39;, &#39;customer_id,event_type&#39;<br>        )<br>    )<br>&quot;&quot;&quot;)</pre><h4>3.2. Vacuum and Orphan File Cleanup</h4><p>Compaction creates new files but doesn’t delete the old ones. Vacuum operations remove files that are no longer referenced by current table metadata. This is where retention policies become critical.</p><blockquote><em>⚠️ </em><strong><em>Production Warning:</em></strong><em> Aggressive vacuum settings will break concurrent readers and time-travel queries. Your retention period must exceed your longest-running query plus a safety buffer. A query that started before vacuum ran might still need files that vacuum wants to delete.</em></blockquote><p><strong>Delta Lake Vacuum</strong></p><p>Always check table history before vacuuming to understand what you’re about to remove:</p><pre>history_df = spark.sql(&quot;&quot;&quot;<br>    DESCRIBE HISTORY delta.`s3://lakehouse/events`<br>    LIMIT 20<br>&quot;&quot;&quot;)<br>history_df.show(truncate=False)</pre><p>Run a dry run first to see what would be deleted:</p><pre>from delta.tables import DeltaTable<br><br>delta_table = DeltaTable.forPath(spark, &quot;s3://lakehouse/events&quot;)<br><br># Dry run - returns files that would be deleted<br>files_to_delete = delta_table.vacuum(retentionHours=168, dry_run=True)<br>print(f&quot;Files that would be deleted: {files_to_delete.count()}&quot;)</pre><p>The default retention is 7 days (168 hours). Delta Lake prevents vacuum with retention under 7 days unless you explicitly disable the safety check:</p><pre># Only disable this if you truly understand the implications<br>spark.conf.set(<br>    &quot;spark.databricks.delta.retentionDurationCheck.enabled&quot;, <br>    &quot;false&quot;<br>)<br><br># Production vacuum with 7-day retention<br>delta_table.vacuum(retentionHours=168)</pre><p>For tables with time-travel requirements or long-running downstream jobs, extend retention accordingly:</p><pre># 30-day retention for tables with compliance requirements<br>delta_table.vacuum(retentionHours=720)</pre><p><strong>Iceberg Snapshot Expiration and Orphan Removal</strong></p><p>Iceberg separates cleanup into distinct operations: snapshot expiration (metadata cleanup) and orphan file removal (storage cleanup).</p><p>Expire old snapshots first:</p><pre>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.expire_snapshots(<br>        table =&gt; &#39;db.events&#39;,<br>        older_than =&gt; TIMESTAMP &#39;2025-11-01 00:00:00&#39;,<br>        retain_last =&gt; 10,<br>        max_concurrent_deletes =&gt; 20<br>    )<br>&quot;&quot;&quot;)</pre><p>The retain_last parameter ensures you always keep recent snapshots regardless of age—critical for tables with infrequent writes where all snapshots might be &quot;old.&quot;</p><p>Then remove orphan files:</p><pre># Always dry run first<br>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.remove_orphan_files(<br>        table =&gt; &#39;db.events&#39;,<br>        older_than =&gt; TIMESTAMP &#39;2025-11-01 00:00:00&#39;,<br>        dry_run =&gt; true<br>    )<br>&quot;&quot;&quot;)<br><br># If dry run looks safe, execute<br>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.remove_orphan_files(<br>        table =&gt; &#39;db.events&#39;,<br>        older_than =&gt; TIMESTAMP &#39;2025-11-01 00:00:00&#39;<br>    )<br>&quot;&quot;&quot;)</pre><p>The older_than timestamp for orphan removal should be conservative. Files must be older than this timestamp AND not referenced by any snapshot. Setting this too aggressively can delete files from in-progress writes.</p><h4>3.3. Manifest and Metadata Compaction</h4><p>At scale, metadata itself becomes a bottleneck. Both systems provide mechanisms to consolidate metadata files.</p><p><strong>Iceberg Manifest Rewriting</strong></p><p>Manifest files track which data files belong to each snapshot. As files are added and removed, manifests accumulate entries for deleted files. Rewriting consolidates active entries:</p><pre>spark.sql(&quot;&quot;&quot;<br>    CALL catalog.system.rewrite_manifests(<br>        table =&gt; &#39;db.events&#39;,<br>        use_caching =&gt; true<br>    )<br>&quot;&quot;&quot;)</pre><p>This operation is particularly impactful after large delete operations or schema evolution, where manifests may contain many tombstoned entries.</p><p><strong>Delta Lake Log Compaction</strong></p><p>Delta Lake automatically creates checkpoint files every 10 commits by default. These Parquet files snapshot the transaction log state, allowing readers to skip JSON log replay. You can tune checkpoint frequency:</p><pre>spark.conf.set(&quot;spark.databricks.delta.checkpointInterval&quot;, &quot;10&quot;)</pre><p>For tables with extremely high commit rates, consider more frequent checkpointing:</p><pre># Checkpoint every 5 commits for high-frequency streaming tables<br>spark.conf.set(&quot;spark.databricks.delta.checkpointInterval&quot;, &quot;5&quot;)</pre><p>To force a checkpoint outside the normal interval:</p><pre>from delta.tables import DeltaTable<br><br>delta_table = DeltaTable.forPath(spark, &quot;s3://lakehouse/events&quot;)<br>delta_table.generate(&quot;symlink_format_manifest&quot;)  # Triggers checkpoint</pre><h4>3.4. Putting It Together: A Complete Maintenance Job</h4><p>Here’s a production-ready maintenance script that handles all three degradation vectors:</p><pre>from delta.tables import DeltaTable<br>from pyspark.sql import SparkSession<br>from datetime import datetime, timedelta<br>import logging<br><br>logging.basicConfig(level=logging.INFO)<br>logger = logging.getLogger(&quot;table_maintenance&quot;)<br><br>def run_delta_maintenance(<br>    spark: SparkSession,<br>    table_path: str,<br>    compaction_days: int = 7,<br>    vacuum_hours: int = 168,<br>    zorder_columns: list[str] | None = None,<br>    dry_run: bool = False<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Complete maintenance cycle for a Delta Lake table.<br>    <br>    Args:<br>        spark: Active SparkSession<br>        table_path: Full path to Delta table<br>        compaction_days: Compact partitions modified within this window<br>        vacuum_hours: Retention period for vacuum<br>        zorder_columns: Columns for Z-ORDER clustering<br>        dry_run: If True, only report what would happen<br>    <br>    Returns:<br>        Dictionary of maintenance metrics<br>    &quot;&quot;&quot;<br>    <br>    metrics = {<br>        &quot;table&quot;: table_path,<br>        &quot;timestamp&quot;: datetime.utcnow().isoformat(),<br>        &quot;compaction&quot;: {},<br>        &quot;vacuum&quot;: {},<br>    }<br>    <br>    delta_table = DeltaTable.forPath(spark, table_path)<br>    <br>    # Step 1: Compaction<br>    logger.info(f&quot;Starting compaction for {table_path}&quot;)<br>    <br>    where_clause = f&quot;date &gt;= current_date() - INTERVAL {compaction_days} DAYS&quot;<br>    <br>    if dry_run:<br>        # Estimate files that would be affected<br>        file_count = len(spark.read.format(&quot;delta&quot;).load(table_path).inputFiles())<br>        metrics[&quot;compaction&quot;] = {&quot;estimated_files&quot;: file_count, &quot;dry_run&quot;: True}<br>    else:<br>        optimize_builder = delta_table.optimize().where(where_clause)<br>        <br>        if zorder_columns:<br>            compact_metrics = optimize_builder.executeZOrderBy(zorder_columns)<br>        else:<br>            compact_metrics = optimize_builder.executeCompaction()<br>        <br>        metrics[&quot;compaction&quot;] = {<br>            &quot;files_removed&quot;: compact_metrics[&quot;numFilesRemoved&quot;],<br>            &quot;files_added&quot;: compact_metrics[&quot;numFilesAdded&quot;],<br>            &quot;bytes_removed&quot;: compact_metrics[&quot;numBytesRemoved&quot;],<br>            &quot;bytes_added&quot;: compact_metrics[&quot;numBytesAdded&quot;],<br>        }<br>    <br>    logger.info(f&quot;Compaction complete: {metrics[&#39;compaction&#39;]}&quot;)<br>    <br>    # Step 2: Vacuum<br>    logger.info(f&quot;Starting vacuum for {table_path}&quot;)<br>    <br>    if dry_run:<br>        files_to_delete = delta_table.vacuum(retentionHours=vacuum_hours, dry_run=True)<br>        metrics[&quot;vacuum&quot;] = {<br>            &quot;files_to_delete&quot;: files_to_delete.count(),<br>            &quot;dry_run&quot;: True<br>        }<br>    else:<br>        delta_table.vacuum(retentionHours=vacuum_hours)<br>        metrics[&quot;vacuum&quot;] = {&quot;retention_hours&quot;: vacuum_hours, &quot;status&quot;: &quot;complete&quot;}<br>    <br>    logger.info(f&quot;Vacuum complete: {metrics[&#39;vacuum&#39;]}&quot;)<br>    <br>    return metrics<br><br><br>def run_iceberg_maintenance(<br>    spark: SparkSession,<br>    table_name: str,<br>    compaction_days: int = 7,<br>    snapshot_retention_days: int = 7,<br>    retain_last_snapshots: int = 10,<br>    sort_columns: str | None = None,<br>    dry_run: bool = False<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Complete maintenance cycle for an Iceberg table.<br>    &quot;&quot;&quot;<br>    <br>    metrics = {<br>        &quot;table&quot;: table_name,<br>        &quot;timestamp&quot;: datetime.utcnow().isoformat(),<br>        &quot;compaction&quot;: {},<br>        &quot;expire_snapshots&quot;: {},<br>        &quot;orphan_removal&quot;: {},<br>        &quot;manifest_rewrite&quot;: {},<br>    }<br>    <br>    retention_timestamp = (<br>        datetime.utcnow() - timedelta(days=snapshot_retention_days)<br>    ).strftime(&quot;%Y-%m-%d %H:%M:%S&quot;)<br>    <br>    where_clause = f&quot;date &gt;= current_date() - INTERVAL {compaction_days} DAYS&quot;<br>    <br>    # Step 1: Compaction<br>    logger.info(f&quot;Starting compaction for {table_name}&quot;)<br>    <br>    strategy = &quot;sort&quot; if sort_columns else &quot;binpack&quot;<br>    <br>    if not dry_run:<br>        if sort_columns:<br>            spark.sql(f&quot;&quot;&quot;<br>                CALL catalog.system.rewrite_data_files(<br>                    table =&gt; &#39;{table_name}&#39;,<br>                    strategy =&gt; &#39;sort&#39;,<br>                    sort_order =&gt; &#39;{sort_columns}&#39;,<br>                    options =&gt; map(<br>                        &#39;target-file-size-bytes&#39;, &#39;536870912&#39;,<br>                        &#39;min-input-files&#39;, &#39;5&#39;<br>                    ),<br>                    where =&gt; &quot;{where_clause}&quot;<br>                )<br>            &quot;&quot;&quot;)<br>        else:<br>            spark.sql(f&quot;&quot;&quot;<br>                CALL catalog.system.rewrite_data_files(<br>                    table =&gt; &#39;{table_name}&#39;,<br>                    strategy =&gt; &#39;binpack&#39;,<br>                    options =&gt; map(<br>                        &#39;target-file-size-bytes&#39;, &#39;536870912&#39;,<br>                        &#39;min-input-files&#39;, &#39;5&#39;<br>                    ),<br>                    where =&gt; &quot;{where_clause}&quot;<br>                )<br>            &quot;&quot;&quot;)<br>        metrics[&quot;compaction&quot;] = {&quot;strategy&quot;: strategy, &quot;status&quot;: &quot;complete&quot;}<br>    else:<br>        metrics[&quot;compaction&quot;] = {&quot;strategy&quot;: strategy, &quot;dry_run&quot;: True}<br>    <br>    # Step 2: Expire snapshots<br>    logger.info(f&quot;Expiring snapshots for {table_name}&quot;)<br>    <br>    if not dry_run:<br>        spark.sql(f&quot;&quot;&quot;<br>            CALL catalog.system.expire_snapshots(<br>                table =&gt; &#39;{table_name}&#39;,<br>                older_than =&gt; TIMESTAMP &#39;{retention_timestamp}&#39;,<br>                retain_last =&gt; {retain_last_snapshots}<br>            )<br>        &quot;&quot;&quot;)<br>        metrics[&quot;expire_snapshots&quot;] = {&quot;status&quot;: &quot;complete&quot;}<br>    else:<br>        metrics[&quot;expire_snapshots&quot;] = {&quot;dry_run&quot;: True}<br>    <br>    # Step 3: Remove orphan files<br>    logger.info(f&quot;Removing orphan files for {table_name}&quot;)<br>    <br>    orphan_timestamp = (<br>        datetime.utcnow() - timedelta(days=snapshot_retention_days + 1)<br>    ).strftime(&quot;%Y-%m-%d %H:%M:%S&quot;)<br>    <br>    if not dry_run:<br>        spark.sql(f&quot;&quot;&quot;<br>            CALL catalog.system.remove_orphan_files(<br>                table =&gt; &#39;{table_name}&#39;,<br>                older_than =&gt; TIMESTAMP &#39;{orphan_timestamp}&#39;<br>            )<br>        &quot;&quot;&quot;)<br>        metrics[&quot;orphan_removal&quot;] = {&quot;status&quot;: &quot;complete&quot;}<br>    else:<br>        spark.sql(f&quot;&quot;&quot;<br>            CALL catalog.system.remove_orphan_files(<br>                table =&gt; &#39;{table_name}&#39;,<br>                older_than =&gt; TIMESTAMP &#39;{orphan_timestamp}&#39;,<br>                dry_run =&gt; true<br>            )<br>        &quot;&quot;&quot;)<br>        metrics[&quot;orphan_removal&quot;] = {&quot;dry_run&quot;: True}<br>    <br>    # Step 4: Rewrite manifests<br>    logger.info(f&quot;Rewriting manifests for {table_name}&quot;)<br>    <br>    if not dry_run:<br>        spark.sql(f&quot;&quot;&quot;<br>            CALL catalog.system.rewrite_manifests(<br>                table =&gt; &#39;{table_name}&#39;<br>            )<br>        &quot;&quot;&quot;)<br>        metrics[&quot;manifest_rewrite&quot;] = {&quot;status&quot;: &quot;complete&quot;}<br>    else:<br>        metrics[&quot;manifest_rewrite&quot;] = {&quot;dry_run&quot;: True}<br>    <br>    return metrics</pre><p>Usage is straightforward:</p><pre># Delta Lake maintenance<br>delta_metrics = run_delta_maintenance(<br>    spark=spark,<br>    table_path=&quot;s3://lakehouse/events&quot;,<br>    compaction_days=7,<br>    vacuum_hours=168,<br>    zorder_columns=[&quot;customer_id&quot;, &quot;event_type&quot;],<br>    dry_run=False<br>)<br><br># Iceberg maintenance<br>iceberg_metrics = run_iceberg_maintenance(<br>    spark=spark,<br>    table_name=&quot;catalog.db.events&quot;,<br>    compaction_days=7,<br>    snapshot_retention_days=7,<br>    sort_columns=&quot;customer_id ASC, event_time DESC&quot;,<br>    dry_run=False<br>)</pre><h3>4. Production Scheduling Patterns</h3><h4>Scheduling Maintenance in Production</h4><p>Running maintenance commands manually doesn’t scale. Production lakehouses need automated, observable, and failure-resilient scheduling. The key insight is that maintenance should be <strong>condition-driven</strong>, not purely time-driven.</p><h4>3.1. Orchestration Architecture</h4><p>A robust maintenance system coordinates across tables, isolates resources from production workloads, and responds to actual table conditions rather than fixed schedules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*wWeoOtDZSRJamTC-8NMVuw.png" /></figure><h4>4.2. Condition-Based Scheduling</h4><p>Before running maintenance, evaluate whether it’s actually needed:</p><pre>from dataclasses import dataclass<br><br>@dataclass<br>class TableHealthMetrics:<br>    total_files: int<br>    small_files_count: int<br>    avg_file_size_mb: float<br>    hours_since_last_compaction: float<br><br>def needs_compaction(<br>    metrics: TableHealthMetrics,<br>    small_file_threshold: int = 1000,<br>    avg_size_threshold_mb: int = 64,<br>    max_hours_between_compaction: int = 24,<br>) -&gt; bool:<br>    &quot;&quot;&quot;Decide if compaction is needed based on table metrics.&quot;&quot;&quot;<br>    <br>    return (<br>        metrics.small_files_count &gt; small_file_threshold<br>        or metrics.avg_file_size_mb &lt; avg_size_threshold_mb<br>        or metrics.hours_since_last_compaction &gt; max_hours_between_compaction<br>    )</pre><p>A streaming table receiving micro-batches every few seconds needs hourly compaction. A slowly-changing dimension table might need maintenance once a month. Query table metrics before deciding.</p><h4>4.3. Scheduling Best Practices</h4><p><strong>Run maintenance during off-peak hours.</strong> Compaction and vacuum operations compete for I/O bandwidth with production queries. Schedule them during low-activity windows — typically overnight or early morning.</p><p><strong>Stagger execution across tables.</strong> Running maintenance on all tables simultaneously creates I/O storms. Space out jobs by 15–30 minutes to avoid saturating storage throughput.</p><p><strong>Isolate maintenance resources.</strong> On Kubernetes or YARN, dedicate resource pools for maintenance jobs so they don’t steal capacity from production workloads:</p><pre># ...<br>maintenance_spark_conf = {<br>    &quot;spark.kubernetes.driver.label.queue&quot;: &quot;maintenance&quot;,<br>    &quot;spark.yarn.queue&quot;: &quot;maintenance&quot;,<br>    &quot;spark.dynamicAllocation.maxExecutors&quot;: &quot;20&quot;,<br>}<br># ...</pre><p><strong>Avoid compacting partitions with active writers.</strong> Both Delta Lake and Iceberg handle conflicts through optimistic concurrency, but conflicts mean retries and potential failures. Target partitions that haven’t received writes in the past 1–2 hours.</p><p><strong>Build for failure recovery.</strong> Maintenance jobs fail — networks timeout, clusters get preempted. Design idempotent operations that can safely retry. If today’s compaction fails, tomorrow’s should succeed without manual intervention.</p><p><strong>Prioritize by table criticality.</strong> Not all tables need the same maintenance frequency. High-traffic, query-critical tables get aggressive schedules. Archival tables can run weekly or monthly.</p><h3>5. Monitoring Table Health</h3><p>You can’t maintain what you can’t measure. Table degradation happens gradually — queries slow down by a few seconds each week until someone notices a critical dashboard timing out. Proactive monitoring catches problems before users do.</p><h4>5.1. Key Metrics to Track</h4><p>Four metrics provide early warning of table degradation:</p><p><strong>File count and small file ratio.</strong> Track total files and the percentage under 128MB. A sudden spike indicates write pattern changes or failed compaction jobs.</p><p><strong>Average file size.</strong> Healthy tables maintain 128MB-1GB files. Averages dropping below 64MB signal urgent compaction needs.</p><p><strong>Hours since last maintenance.</strong> Simple but effective. If compaction hasn’t run in 48 hours on a streaming table, something is wrong.</p><p><strong>Query planning time.</strong> Metadata bloat shows up here first. If planning time grows while data volume stays flat, manifest rewriting is overdue.</p><h4>5.2. Health Check Implementation</h4><p>A lightweight function to gather these metrics for Delta tables:</p><pre>def get_delta_table_health(spark, table_path: str) -&gt; dict:<br>    &quot;&quot;&quot;Extract health metrics from a Delta table.&quot;&quot;&quot;<br>    <br>    from delta.tables import DeltaTable<br>    <br>    # File statistics<br>    files = spark.read.format(&quot;delta&quot;).load(table_path).inputFiles()<br>    <br>    fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(<br>        spark._jsc.hadoopConfiguration()<br>    )<br>    <br>    file_sizes = []<br>    for f in files:<br>        path = spark._jvm.org.apache.hadoop.fs.Path(f)<br>        file_sizes.append(fs.getFileStatus(path).getLen())<br>    <br>    small_file_threshold = 128 * 1024 * 1024  # 128MB<br>    <br>    return {<br>        &quot;total_files&quot;: len(file_sizes),<br>        &quot;total_size_gb&quot;: sum(file_sizes) / (1024 ** 3),<br>        &quot;avg_file_size_mb&quot;: (sum(file_sizes) / len(file_sizes)) / (1024 ** 2),<br>        &quot;small_files_count&quot;: sum(1 for s in file_sizes if s &lt; small_file_threshold),<br>        &quot;small_files_pct&quot;: sum(1 for s in file_sizes if s &lt; small_file_threshold) / len(file_sizes) * 100,<br>    }</pre><p>For Iceberg, use the metadata tables directly:</p><pre>def get_iceberg_table_health(spark, table_name: str) -&gt; dict:<br>    &quot;&quot;&quot;Extract health metrics from an Iceberg table.&quot;&quot;&quot;<br>    <br>    files_df = spark.sql(f&quot;SELECT * FROM {table_name}.files&quot;)<br>    <br>    stats = files_df.selectExpr(<br>        &quot;count(*) as total_files&quot;,<br>        &quot;sum(file_size_in_bytes) / pow(1024, 3) as total_size_gb&quot;,<br>        &quot;avg(file_size_in_bytes) / pow(1024, 2) as avg_file_size_mb&quot;,<br>        &quot;sum(case when file_size_in_bytes &lt; 134217728 then 1 else 0 end) as small_files_count&quot;<br>    ).collect()[0]<br>    <br>    return stats.asDict()</pre><h4>5.3. Monitoring Best Practices</h4><p><strong>Run health checks independently from maintenance jobs.</strong> A failed compaction shouldn’t block metric collection. Schedule lightweight health checks every 15–30 minutes.</p><p><strong>Track trends, not just thresholds.</strong> A table with 2000 small files might be fine if it’s been stable for months. A table that jumped from 500 to 2000 in a week needs attention.</p><p><strong>Correlate with query performance.</strong> Connect table health metrics to query latency dashboards. When users report slowness, you’ll immediately see if file fragmentation is the cause.</p><p><strong>Monitor maintenance job success rates.</strong> Track compaction and vacuum completion rates. A job that succeeds 60% of the time indicates resource contention or configuration issues.</p><p><strong>Set up weekly health reports.</strong> Automated summaries showing file counts, sizes, and maintenance history across all tables help spot gradual degradation that doesn’t trigger alerts.</p><p>The goal is visibility. When a query slows down, you should know within seconds whether table health is the culprit — and have the metrics to prove it.</p><h3>6. Cost Analysis</h3><h4>The ROI of Maintenance</h4><p>Table maintenance isn’t just about query performance — it directly impacts cloud costs. Understanding the financial impact helps justify dedicated maintenance infrastructure and engineering time.</p><h4>6.1. Where Unmaintained Tables Cost Money</h4><p><strong>Storage costs from orphan files.</strong> Failed writes, abandoned transactions, and schema migrations leave orphan files that consume storage indefinitely. I’ve audited tables where orphans accounted for 20–30% of total storage. That’s thousands of dollars monthly in wasted storage.</p><p><strong>API request costs from file proliferation.</strong> Object storage charges per operation. S3 LIST requests cost $0.005 per 1000 requests. GET requests cost $0.0004 per 1000. A query scanning a table with 50000 files generates 50x more API calls than the same query on a well-compacted table with 1000 files. Multiply by thousands of queries daily across dozens of tables.</p><p><strong>Compute costs from inefficient scans.</strong> Small files mean excessive file-open overhead. Queries that should complete in 30 seconds take 5 minutes, consuming 10x more executor-hours. Dynamic allocation helps, but you’re still paying for wasted compute.</p><h4>6.2. The Hidden Cost: Engineering Time</h4><p>The most expensive consequence of unmaintained tables isn’t cloud bills — it’s the engineering hours spent debugging slow queries, firefighting production incidents, and manually cleaning up neglected tables. A single outage investigation can consume more engineering time than building proper maintenance automation.</p><p>Invest in maintenance infrastructure early. The ROI compounds over time as your lakehouse grows.</p><h3>7. Common Pitfalls and War Stories</h3><p>Theory only takes you so far. These are real failure scenarios I’ve encountered — each one a lesson learned the hard way.</p><h4>7.1. The Vacuum That Broke Time Travel</h4><p>A team needed to free up storage quickly before month-end billing. Someone ran VACUUM RETAIN 0 HOURS on a critical table. The command succeeded. Storage dropped by 40%.</p><p>The next morning, every downstream job failed. Those jobs used VERSION AS OF to read consistent snapshots for reconciliation. With zero retention, every historical version was gone. Recovery required restoring from backups and rebuilding three days of derived tables.</p><p><strong>Lesson:</strong> Retention policies must account for all consumers, not just active queries. Document time-travel dependencies explicitly and enforce minimum retention at the platform level.</p><h4>7.2. The Infinite Compaction Loop</h4><p>A streaming table received micro-batches every 30 seconds. The team scheduled hourly compaction to control file growth. Sensible in theory.</p><p>In practice, compaction took 45 minutes. While compaction ran, streaming writes continued creating new small files. By the time compaction finished, the next run was due — and the file count was higher than before. The cluster ran compaction continuously, starving production workloads of resources.</p><p><strong>Lesson:</strong> Compaction frequency must account for compaction duration plus write rate. For high-velocity streaming tables, consider write-side coalescing (larger trigger intervals, more partitions per batch) rather than aggressive read-side compaction.</p><h4>7.3. Compaction During Peak Hours</h4><p>An OPTIMIZE job was scheduled for 2 PM — seemingly safe, after the morning batch jobs and before evening reporting. No one realized that 2 PM coincided with peak ad-hoc query activity from analysts.</p><p>The compaction job grabbed 50 executors through dynamic allocation. Analyst queries queued behind it, some timing out entirely. The incident ticket blamed “cluster instability” until someone correlated timestamps with the maintenance schedule.</p><p><strong>Lesson:</strong> Off-peak means off-peak for all workloads, not just scheduled jobs. Monitor cluster usage patterns before setting maintenance windows, and enforce resource pool isolation.</p><h4>7.4. The Orphan Explosion After Schema Evolution</h4><p>A schema migration added a nested struct column. The migration job failed mid-write due to a serialization bug, was fixed, and rerun successfully. Everyone moved on.</p><p>Six months later, a storage audit revealed 2TB of orphan files — all from that failed migration. The standard remove_orphan_files procedure couldn&#39;t identify them because their timestamps fell within the normal retention window of valid files.</p><p><strong>Lesson:</strong> Failed schema migrations require immediate orphan cleanup, not deferred maintenance. After any failed write operation, verify orphan state before proceeding.</p><h4>7.5. Silent Metadata Bloat</h4><p>Query planning time crept up slowly — 30 seconds became 45, then 60, then 90. No one noticed because the degradation was gradual and dashboards only showed query execution time, not planning time.</p><p>The root cause was 18 months of manifest files that had never been rewritten. The table had millions of historical file entries that Iceberg dutifully tracked across thousands of manifests.</p><p><strong>Lesson:</strong> Monitor planning time separately from execution time. Include manifest rewriting in your maintenance runbooks, not just compaction and vacuum.</p><h3>8: Key Takeaways</h3><p>Table formats like Delta Lake and Iceberg solved the ACID and metadata problems for data lakes. They didn’t solve the maintenance problem — they shifted it to you.</p><p><strong>Understand the degradation mechanics.</strong> Small files accumulate from streaming writes and frequent updates. Metadata bloats from transaction logs and manifests. Orphan files leak storage from failed operations. Each vector requires a specific maintenance operation.</p><p><strong>Compaction is not optional.</strong> Without regular OPTIMIZE or rewrite_data_files, query performance degrades predictably. Target 256MB-1GB files. Use Z-ORDER or sort-based compaction for columns that appear frequently in filters.</p><p><strong>Vacuum carefully.</strong> Aggressive retention breaks time travel and concurrent readers. Set retention based on your longest-running query plus a safety margin. Always dry-run before executing.</p><p><strong>Don’t forget metadata.</strong> Manifest rewriting in Iceberg and checkpoint management in Delta Lake prevent planning time from growing unbounded. Schedule these operations monthly at minimum.</p><p><strong>Automate with conditions, not just schedules.</strong> Query table health metrics before running maintenance. A streaming table needs hourly attention; a slowly-changing dimension needs monthly. One schedule doesn’t fit all.</p><p><strong>Monitor proactively.</strong> Track file counts, average file sizes, and hours since last maintenance. Alert before users notice slowdowns. Correlate table health with query latency.</p><p><strong>Isolate maintenance resources.</strong> Run compaction and vacuum in dedicated resource pools during off-peak hours. Stagger execution across tables to avoid I/O storms.</p><p><strong>Budget for maintenance.</strong> The cost of unmaintained tables compounds — storage waste, API overhead, compute inefficiency, and engineering time debugging slow queries. Invest in automation early.</p><p>The lakehouse promise is real, but it requires operational discipline. Build maintenance into your platform from day one, and your tables will perform reliably at any scale.</p><h3>9. Further Reading</h3><p><strong>Delta Lake Documentation</strong></p><ul><li><a href="https://docs.delta.io/latest/optimizations-oss.html">OPTIMIZE Command</a></li><li><a href="https://docs.delta.io/latest/delta-utility.html#vacuum">VACUUM Reference</a></li><li><a href="https://docs.delta.io/latest/best-practices.html">Best Practices for Production</a></li></ul><p><strong>Apache Iceberg Documentation</strong></p><ul><li><a href="https://iceberg.apache.org/docs/latest/spark-procedures/">Maintenance Procedures</a></li><li><a href="https://iceberg.apache.org/docs/latest/configuration/">Table Configuration</a></li><li><a href="https://iceberg.apache.org/docs/latest/performance/">Performance Tuning</a></li></ul><p><strong>Architecture Deep-Dives</strong></p><ul><li><a href="https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf">Lakehouse: A New Generation of Open Platforms</a> — CIDR 2021</li><li><a href="https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf">Delta Lake: High-Performance ACID Table Storage</a> — VLDB 2020</li></ul><p><strong>Community Resources</strong></p><ul><li><a href="https://netflixtechblog.com/">Netflix Tech Blog: Scaling Iceberg</a></li><li><a href="https://www.databricks.com/blog/category/engineering">Databricks Engineering Blog</a></li><li><a href="https://iceberg.apache.org/community/">Apache Iceberg Community Slack</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=380ab6d33a02" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/compaction-vacuum-and-chaos-operating-delta-lake-and-iceberg-tables-in-production-380ab6d33a02">Compaction, Vacuum, and Chaos: Operating Delta Lake and Iceberg Tables in Production</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>