<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jack Art Guy on Medium]]></title>
        <description><![CDATA[Stories by Jack Art Guy on Medium]]></description>
        <link>https://medium.com/@golemlabs-ai?source=rss-1bc983314374------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*J2N4NGnijBEhZP5-</url>
            <title>Stories by Jack Art Guy on Medium</title>
            <link>https://medium.com/@golemlabs-ai?source=rss-1bc983314374------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:40:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@golemlabs-ai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Built a Bug-Catcher for My AI Agents. Its First Catch Was Itself.]]></title>
            <link>https://golemlabs-ai.medium.com/i-built-a-bug-catcher-d2472099c973?source=rss-1bc983314374------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2472099c973</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jack Art Guy]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 02:00:51 GMT</pubDate>
            <atom:updated>2026-09-09T02:00:51.618Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>A year of watching AI agents write real software taught me where they actually break. It is not the model. It is that everyone, the agents and me, keeps checking the wrong thing.</em></strong></p><p>I wrote a test to prove one of my safety gates catches a made-up name. To do that, the test had to spell the fake name out inside itself. So the gate went looking, <em>found</em> the fake name sitting right there in the test, decided it was real, and passed. The test poisoned the very ground it was standing on.</p><p><strong>I built the gate. I wrote the test. I did not catch it.<br></strong>That gate is one small piece of something bigger. I build an operating system for a team of AI executives. Think of a little C-suite: one agent for strategy, one for money, one for legal, and one that writes the code, all running under one roof and supposed to work together like a real team.</p><p>In earlier notes I went after two boring truths that team needs: a memory that does not lie, and data-sharing that does not leak. This is the third boring truth, and it is the one people are loudest and least honest about. When the agent that writes code finishes a job and says <em>done</em>, it has to actually be done. Not “compiles.” Not “looks right.” Done.</p><p>That sounds like the easiest of the three. It is the one that bit me most.</p><p>And, like the other two, the moment you try to build it for real instead of tweeting about it, you walk face-first into a wall that everyone else is also standing in front of, rubbing their nose. That is why your feed is full of confident takes right now: <em>it is not the model, it is the harness. Agents lie about their work. Stop trusting the transcript, trust the artifact.</em> All true. All well said, forty times over. I am not going to write the forty-first “trust the artifact.”</p><p>Instead I did the unglamorous thing. I went back through the last year of my agents building four real apps and the smaller tooling projects around them, and wrote down every single bug I could find, in four running registers, one per source I dug through. Then I gave each one a cause of death. I have spent twenty-five years running companies rather than writing code, and that is the reflex it leaves you with: when something breaks you do not ask who was stupid, you ask what let it through.</p><p>Three hundred and forty-two of them. And when you stop vibing the problem and actually count it, two features fall out that nobody seems to be talking about.</p><p><strong>Where the bugs actually come from</strong></p><p>Here is the whole autopsy in one picture. Every one of the 342 defects, sorted by what really caused it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHDOVhRJJoVIqjta3YmFYA.png" /><figcaption>The middle slice moves around between the four registers, and what never moved was which slice came first.</figcaption></figure><p>The number that stopped me was not any single slice. It was that process came out the single biggest cause in all four registers, counted separately. Four blind samples landing in the same place is about as close to a real signal as you get in this line of work.</p><p>And when I sorted the bugs by <em>kind</em> instead of cause, the single biggest pile was not a crash and not a security hole. It was this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-AhX27mScH6YJbRPrL-XA.png" /><figcaption>Every bug a fresh reviewer found in the tool I built to fix this was the top bar again.</figcaption></figure><p>Those are the six biggest classes, and together they come to 251 of the 342. The other 91 scatter across twenty-eight smaller ones, none of them bigger than eighteen.</p><p>The agent finishes, runs a green check, tells me it is done, and it genuinely believes it. The check was real. The work still shipped broken, because the thing it checked was not the thing that ships. It read the recipe and swore the meal was fine. That is the whole disease.</p><p><strong>Feature one: the knowledge gap is a finite list. So write it.</strong></p><p>Everyone treats the competence problem, that middle third, like a bottomless pit. The agent does not <em>know things</em>, and there are infinitely many things to not know, so we are doomed, so hire more seniors. That framing is both wrong and a bit of a downer.</p><p>The knowledge failures were not infinite. They were a bounded pile of platform traps that keep showing up in new outfits. So I sat down and wrote the whole list out. It came to about sixty. Here are four, with the fingerprints wiped off:</p><p><strong>The deploy tool that saved to the wrong drawer.</strong> It wrote to a <em>local</em> copy by default while the live server read the <em>remote</em> one. Every check said the setting was there. Production said it was missing. It looks like a ghost until you learn the default.</p><p><strong>The white screen that only happened for real users.</strong> One module built a lookup table the instant it loaded, from a value a second module had not filled in yet. On the dev machine the timing was forgiving. In the packed-down production build it was not, and the whole app went blank. Perfect on the laptop, dead in the wild.</p><p><strong>The app that served a webpage where a script should be.</strong> An offline helper handed the browser a leftover page in place of a missing chunk of code. The browser tried to run the webpage as code and fainted. On every retry it got worse, because it was grabbing files from the wrong batch.</p><p><strong>The safety check written in the wrong dialect.</strong> A gate used a shorthand for “any space” that the Mac shell it ran on reads as a plain letter <em>s</em>. So it quietly matched nothing, never failed, and waved real problems through for hours while looking perfectly healthy.</p><p>None of those is a mystery once it has a name. That is the point. A knowledge gap you have written down stops being a scary gap and becomes a line on a checklist. The fear is infinite; the actual list is not. Closing it is boring, finite work, which is exactly why it is worth doing and exactly why almost nobody does it.</p><p><strong>Feature two: the process gap eats its own cure</strong></p><p>This is the part I did not see coming, and the part worth your time.</p><p>“Checked the wrong thing” keeps coming back because the checker you build to catch it has the same blind spot as the thing it is checking. A referee trained by the same coach as the players does not call the fouls. You cannot fix this by trying harder. You fix it by forcing a <em>different</em> blind spot into the room.</p><p>I learned this the honest way, which is to say it happened to me recently and it was a little humbling.</p><p>I put my build agents to work constructing the exact tools that enforce “check the shipped thing, not the claim.” Then I sent in a fresh reviewer, one deliberately kept ignorant of how each tool was built, to check the tools. It caught four real bugs in my own agents’ work. Every single one was the exact disease the tools exist to cure.</p><p><strong>1. The check too slow to ever run.</strong> <br>The pre-build scan for made-up references was written to comb the entire codebase for every name it saw. It timed out. A check that never finishes is not a check. It is a “no check” wearing a lanyard.</p><p><strong>2. The check that could not start.</strong> <br>The health tool had a tiny typo in how it quoted a line, and the shell refused to run it at all. It would have sat in the config looking exactly like coverage while providing none.</p><p><strong>3. The check that called a working gate dead.</strong> <br>The health tool searched for the phrase the gate emits when it blocks something. The gate emitted the same phrase with one extra space. So the tool reported a perfectly good safety gate as broken. A false alarm from a single missing space.</p><p><strong>4. The test that hid its own answer.</strong> <br>The one I opened with. A permanent test, sitting in the repo, quietly certifying that the gate worked by feeding it the one thing the gate exists to reject.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4bfqw1rBFoEIaxcVOTjU1w.png" /></figure><p>Read those four again. Every one is “checked the wrong thing.” The slow one never touched reality. The broken one never ran. The false-alarm one checked its own too-fussy phrasing instead of the gate’s real output. The last one checked a patch of ground it had already salted. I spent a week learning that agents check the wrong thing, I built the cure, and the cure shipped the disease. All four were fixed the same day they were found. But only a set of eyes with a <em>different</em> blind spot found them in the first place.</p><p>That recursion is not a footnote. It is the entire mechanism. Verification is not a checkbox you add once and dust off your hands. It is a habit that has to be checked in turn, by something that does not share the blind spot of the thing it is checking.</p><p><strong>Steal these. Failure modes included.</strong></p><p>Not advice. Actual moves, each with the way it can still bite you, because a trick sold without its failure mode is just marketing.</p><p><strong>Check the plate, not the recipe.</strong> Nothing is “done” until the real thing runs the real way a real user hits it, on the deployed version, not on the source and not on a test whose input you hand-fed it. <br><em>Bites you when: </em>your test quietly builds itself a convenient input and passes against a field nobody uses.</p><p><strong>Make every alarm catch a real fire once.</strong> A safety check that has never once gone off is impossible to tell apart from one that <em>cannot</em> go off. Before you trust its silence, hand it a known-bad thing and watch it howl. <br><em>Bites you when:</em> the check quietly passes on its own internal error, so “broken” reads as “all clear.”</p><p><strong>Look before you build.</strong> Most “I remembered it wrong” bugs die if you simply check that every name, table, and setting your plan mentions actually exists, before you write a line. <br><em>Bites you when:</em> the thing lives in a corner your search does not look, and reads as a ghost.</p><p><strong>Get a second opinion from a different brain.</strong> On anything that matters, a thing grading its own homework is not a second opinion, it is the same opinion twice. Put a different model, or a fresh reviewer who never saw the work, in the loop. <br><em>Bites you when:</em> the second opinion always agrees, which means you are feeding it the first one’s answer.</p><p><strong>The moat is the boring part</strong></p><p>Good tools are not the enemy here. Linters, test suites, type checkers, review bots, I run all of them, and every one of them earns its keep.</p><p>That is exactly the point. Every one of those good ideas either passed my broken code or checked the wrong thing at least once, and the guard I built to watch them called a working gate dead, in ways I could only see because a boring, deterministic instrument was standing there refusing to be impressed by anyone’s <em>done</em>, including its own.</p><p>The architecture is a weekend. The moat is the unglamorous habit underneath it: check the thing that actually ships, make every guard go off once on purpose before you trust its silence, and never let one brain be the final judge of its own work, especially when that brain is the guard you just built.</p><p>I do not get to say any of that from a distance. So the second pair of eyes lives in the system now, wired in and running whether I remember to ask for it or not.</p><p><em>If you are building with agents and actually measuring the mess, not just the wins, I would love to compare notes. I especially want to hear about the check you trusted that turned out to be dead, and how you found out. Nobody writes up the alarm that never rang. That is where the knowledge is.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2472099c973" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why New-and-Better is Quietly the Enemy of Old-and-Good…]]></title>
            <link>https://golemlabs-ai.medium.com/why-new-and-better-is-quietly-the-enemy-of-old-and-good-f6b7a0dfec42?source=rss-1bc983314374------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6b7a0dfec42</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Jack Art Guy]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 10:01:13 GMT</pubDate>
            <atom:updated>2026-09-06T10:16:25.736Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>The AI field ships a breakthrough every week. I put four of them on a scale before believing any of them. Two did not survive.</em></strong></p><p>I build an operating system for a team of AI executives. Picture a little C-suite: one agent for strategy, one for money, one for legal, all running under one roof and supposed to work together like a real team.</p><p>For that to work, two boring things have to be true. They have to remember things, the same things, reliably. And they have to hand each other data without ever leaking the data one of them was never meant to see. Memory that does not lie, and sharing that does not leak. That is the whole job.</p><p>What I am building toward is stranger than good retrieval. What runs today is plainer: plumbing that reads and writes, and occasionally throws the same flat I/O error as any other plumbing. The gap between what I want and “crashed while fetching its own memory” is real. And you do not close a gap like that by chasing whatever launched on Tuesday, you close it by weighing each new idea on your own system and keeping only the ones that earn their place.</p><p><strong>The problem with a field this young</strong></p><p>AI tooling is young. Young things move fast, which means a genuine breakthrough lands roughly every time you refill your coffee. Knowledge graphs, vendor-neutral knowledge formats, a dozen flavors of retrieval, and every one arrives with a confident blog post, an architecture diagram, and the quiet suggestion that if you are not already using it, you are behind.</p><p>Some of those breakthroughs are real, and in a year they will be the boring default. Some are just this season’s color. And here is the trap that catches good engineers in particular, because we are the ones who like new toys: new-and-better is very often the enemy of old-and-good. You rip out the thing that worked, bolt on the shiny thing everyone is posting about, and your system gets worse in a way you will not notice for three weeks, because you never measured the old thing before you replaced it.</p><p>So this is not a post about the newest architecture. It is about the dull habit that lets you survive a field which reinvents itself every quarter: build on layers, tune them to what you actually do, and weigh every trend against your own numbers before you swallow it. I will show you the two instruments I weigh with. I will show you four fashionable moves I weighed, and what the scale said about each. And I will show you why the right answer is different for a coding agent than for my executive suite, even though the way you decide is exactly the same.</p><p>Because here is the good news the doom takes always skip. This works. My own measurements say the memory is in good shape and the regressions I went hunting for are fixed, and not because I picked the trendiest stack. Because I weighed things.</p><p><strong>What the thing is actually made of: layers, not a trend</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRkX3CpP09h1LNrGgMnqOg.png" /><figcaption>Drawing this was the moment the weekly breakthroughs stopped scaring me. Every one of them is just a proposal to add a row here or turn one of these dials.</figcaption></figure><p>Before the weighing, the thing itself, because “a memory system” is uselessly vague.</p><p>When one of my executives needs to remember something, a little retriever goes and fetches it, and it is not one clever trick but a few layers working at once. One layer matches on words: you said “invoice,” it finds notes that say “invoice.” One matches on meaning, so “invoice” also pulls back “billing” and “amount due.” And on top sit two small curated layers, one holding hand-vetted notes that should count for more than raw logs, the other knowing how things relate, that this decision caused that change, that this fact overrides that older one.</p><p>That is the mental shift. The system is not “a RAG,” or “a knowledge graph,” or whatever noun is winning this month. It is a stack of layers, each with its own volume dial, and almost every “breakthrough” you read about is really just a proposal to add a layer or turn a dial. So the honest question is never “is this trendy.” It is “does this dial, at this setting, on my data, make my numbers better.” And to answer that, you need a number.</p><p><strong>The first instrument: a bathroom scale for memory</strong></p><p>You cannot feel a number. Step on the scale. That is the whole idea, and it is the part the genre skips.</p><p>Mine is embarrassingly simple. I wrote down about sixty questions my system actually gets asked, and next to each one the answer it should find. A tiny program then asks the memory each question and checks a single thing: how far down the list did the right answer show up. Add that across all sixty, and you get two honest numbers, how often the answer landed near the top and how high it landed on average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJTbtjdzIPZgU71a8j8ZJg.png" /><figcaption>I wrote these down before I touched anything else. Without a before row, every change you make feels like progress.</figcaption></figure><p>That is the whole scoreboard. Those numbers are why I can tell you the system works, instead of just hoping it does. They are also a snapshot of a corpus that keeps growing, so they drift, upward lately, which is exactly why you measure yours instead of borrowing mine. One thing to hold onto below: each experiment ran at its own moment on that moving corpus, so the absolute figures shift a little from table to table. In each one, the number that matters is the gap between the before row and the after row.</p><p>One detail matters more than the rest. I keep the test questions short. Two or three words, the way you actually rummage through your own memory. You do not think “where are the keys I always leave on the blue hook by the front door,” because that question answers itself, you think “keys.” Long, specific test questions quietly contain their own answers, and they make any system look brilliant. Short questions are where the ranking actually has to decide. So short questions are where the truth is.</p><p>Now, four trends I weighed on that scale.</p><p><strong>Weighing 1: rank fusion. Fashionable, textbook, and worse.</strong></p><p>I had a real problem. My word-matcher spits out big messy numbers, my meaning-matcher spits out small tidy ones, and adding them is like adding someone’s salary to their shoe size. The salary always wins. The recommended-everywhere fix is to drop the raw numbers and use the rankings instead, so I built it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4n0os0u4EQkgoHqaO1Nvw.png" /><figcaption>This one did not survive. I kept the sweep in the repo anyway, so the next time somebody sends me the blog post, the table is already there.</figcaption></figure><p>Every setting was worse than not doing it. Turning scores into rankings throws away how far apart things were. Picture a race: the winner beat second place by a hair, and third finished five minutes later, gasping. Now write down only “first, second, third,” and you have erased that the top two were basically tied. My meaning-matcher had spent a week learning exactly those fine distinctions, and the “fix” bulldozed them flat. Old-and-good beat new-and-better, and only the scale could tell me. I wrote down why, and I did not ship it.</p><p><strong>Weighing 2: the trusted-source bonus that started hiding the right answers</strong></p><p>Some memories are hand-vetted, some are raw log noise, and obviously the vetted ones should outrank the noise, so I gave them a bonus. And because I was proud of the vetted stuff, I gave it a big one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rWTmGXO5WqLseSRXPTgXRw.png" /><figcaption>The setting I was proudest of was the one costing me 35 points, and only the short questions exposed it. This one survived, turned down from a hammer to a nudge.</figcaption></figure><p>The bonus had grown so large it was inverting the whole thing. “Trusted” was beating “actually relevant” so reliably that correct notes were falling clean out of the top ten. I turned it down to a nudge, and top-three recall jumped from 60% to 95%. And I would never have caught it with comfortable, wordy test questions, because on those the word-matcher drowns the bonus and everything looks fine. It only shows up on the short questions, the ones where the bonus is actually calling the shots. Which is the most human failure mode there is: the senior voice says the confident thing, and the junior who actually knows never gets heard. I had accidentally taught my software to run meetings.</p><p><strong>Weighing 3: the number that went up while the system got worse</strong></p><p>Some documents cover several topics, and a small correct point can get swallowed by the loud one. The obvious fix is to let the little paragraphs compete on their own. I measured it. Top-five recall went up two points. Ship it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pXfhEhGvq1gREuefsyoZ8g.png" /><figcaption>I went looking for top-five recall, found it, and nearly shipped on that alone. Two other numbers said no, and this is the table I think about whenever a single metric agrees with me.</figcaption></figure><p>No. The average position got worse, and the outright misses doubled. I had bought “found it eventually” with “found it fast,” and paid the difference in total failures. One flattering number will happily sell you a downgrade. This is exactly why you never weigh a trend on a single metric: the trick that helps one number usually mugs another, and the average is a lovely place to hide a body.</p><p><strong>Weighing 4: the one that taught me you cannot judge a layer in isolation</strong></p><p>For ages my meaning-matcher felt useless, and I nearly concluded that semantic search just was not worth it on my data. That was wrong. And it was wrong because of a completely different layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFj7Jpr8MeqxnPZeT2pkaA.png" /><figcaption>For months I blamed the meaning-matcher and was one afternoon away from ripping it out. The fault was in the layer next door, and the best setting only appeared after I fixed it.</figcaption></figure><p>The meaning-matcher was not weak, it was drowned out. My word-matcher had a bug that treated filler words like “the” and “of” as rare and important, so its score was full of static and the meaning signal could not be heard over it. It is the volume-knob illusion: you crank the speaker that seems broken, and the real problem is hiss from the speaker right next to it. I killed the hiss, and the best setting moved. That is the whole warning about adopting trends one layer at a time. Judge a new thing while a different layer is quietly broken, and you are not evaluating the trend at all, you are measuring your own bug and blaming the wrong part.</p><p><strong>Four weighings later: yes, it works, and I can prove it</strong></p><p>Four fashionable moves, weighed. Two I kept, in a tamed form. Two I threw out, with the numbers to prove why. The result is not a pile of wreckage, it is a memory that lands the right answer in the top three more than nine times out of ten, on the lazy questions, measured and repeatable. Old-and-good, defended on purpose against new-and-better. The scale is not there to make you feel bad about your ideas. It is there so that a field which reinvents itself every quarter cannot quietly make your working system worse.</p><p><strong>The twist: the right layers depend on what your agents actually do</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9elWJebpiwN2I9fJRqrVgw.png" /><figcaption>Same four layers, three different jobs, three different tunings. Which is why nobody, me included, can tell you what to adopt.</figcaption></figure><p>Here is the part that makes “just adopt X” advice worthless. The layers are the constant. The dials are not. And the right mix depends entirely on what your agents are actually for.</p><p><strong>A coding agent</strong> lives in a codebase, where the source of truth is the code itself and most structure is derivable from it. Literal, structural search does the heavy lifting here, plain old grep and the call graph, and the meaning-matcher matters far less. What will actually bite you is freshness, a derived index that lags the working tree. Different job, different dials.</p><p><strong>My executive suite</strong> is the opposite. The valuable knowledge is judgment, history, and relationships that live in no file: why we chose this vendor, which client is fragile, which decision quietly overrode which. So the curated and relationship layers carry the weight, provenance matters enormously, and, as we are about to see, who is allowed to see what becomes a first-class problem.</p><p><strong>A content setup</strong> is different again. It is a big fuzzy pile of prior work, references, drafts, and brand voice, where semantic recall dominates, exact matching matters less, and “have we already said this” quietly becomes its own layer.</p><p>Same handful of layers, three completely different tunings. That is why “should I adopt this trend” has no universal answer and never will. The answer is always local. What does not change, across all three, is the method: the short-question scale, more than one metric, and the discipline of weighing on your own data before you trust anyone’s diagram. The tuning is domain-specific. The way you decide is universal.</p><p><strong>The other half of the job: perfect memory, wrong recipient</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWRcpqnBeuv20NvUvjrEPQ.png" /><figcaption>Every box on the bottom row is behaving exactly as designed. There is no crash and no error, just a correct answer sitting in the wrong hands.</figcaption></figure><p>That was the remembering half. Now the sharing half, which can end your afternoon in an apology even when the memory itself is flawless.</p><p>You can have the best retriever on earth and still hand the wrong executive the wrong row. Because this job is not about what the agent remembers, it is about what the agent is allowed to pass on. My finance agent can reach a client’s numbers; a more junior agent on the same system must not, even if it asks nicely, even if the question sounds perfectly innocent. There is a phrase worth stealing here: existence is not permission. An agent being able to reach a piece of data does not mean the person who asked is allowed to see it.</p><p>Agents fall into this in a way humans mostly do not. The agent usually talks to the database as itself, with its own god-level access, not as the human who asked, so the human’s limits never get applied. Nothing looks broken. The query runs, the answer is correct. It is just correct for someone who was never supposed to see it.</p><p><strong>The blog said build it. The scale said I already had it.</strong></p><p>When I went to fix this, the field had opinions: adopt this new guard, build a permissions engine, bolt on the shiny thing. So, having just spent a month learning to weigh before believing, I did the data-access version of stepping on the scale, and grepped my own codebase before building anything. Most of the pieces were already there, quietly wired in and working: a role table, a hard wall between one client’s data and another’s, and a filter that hands a restricted role only the fields on a short approved list, so a new sensitive column cannot leak by default. Most of what the “you must build this” posts told me to acquire was already load-bearing in the repo. Only about three pieces were genuinely new.</p><p>That is an anticlimax, and it is the whole point again. New-and-better told me to build. Old-and-good was already running, and just needed one specific gap closed, not a greenfield rewrite. The most expensive thing you can do in a fast-moving field is replace something that worked because a blog post made you feel behind.</p><p><strong>The second instrument: for leaks, throw attacks and count the ones that get through</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mi6sV2-YF8vH2qhw5BZHPQ.png" /><figcaption>The left card has three numbers because it has already been run against real questions. The right one says nothing yet, and I am leaving that box empty, because the only alternative is a number I have not earned.</figcaption></figure><p>Memory has a scale. Data-sharing has its own instrument, and it is just as dull. You do not ask “does this feel secure.” You write down a pile of specific, nasty attempts to see data you should not, you throw all of them at the guard, and you count how many get through. A junior agent asking, in friendly language, for numbers it has no business seeing. An innocent question that can only be answered by quietly joining a table the asker is locked out of. A harmless-looking count that leaks the existence of rows they cannot see. Each one is a test with a right answer, either a clean refusal or, better, a partial answer that says out loud what it left out. Run them all, and you get a number. That number is the data-sharing bathroom scale.</p><p>And exactly like the memory filter, it is worthless until you prove it can catch a real fish. A leak guard that has never blocked anything looks identical to a leak guard that is switched off. This already bit me once. I shipped a filter that drops out-of-date notes, ran the scale, got a perfect “zero change,” and nearly believed it, until I noticed that not one note was actually stamped out-of-date yet. So “zero change” meant either “works” or “blind,” and I could not tell which. A smoke detector that has never gone off tells you nothing. Not until you press the test button and hear it shriek. So I made that filter catch things on purpose: eight planted cases, eight caught, but only after I had first proved it could catch a single one.</p><p>That eight-out-of-eight, from the memory side, is the one honest number I have on proving an instrument at all, and I want to be straight about the asymmetry it exposes. The memory scale is shipped and running, five tables of it. The full leak instrument for the sharing layer is the thing I am building right now, with exactly this method, and it is not a finished scoreboard yet. I would rather tell you that than dress a design up as a measurement.</p><p>There is one more thing the leak instrument buys you, and it is really about honesty. When the guard cannot fully answer, the worst move is to quietly return the half it can reach and let the asker assume that is the whole picture. The honest move is to say it plainly: here is what I can show you, here is what I cannot, and here is why. In both halves of this system, the failure that hurts is never the loud one. It is the quiet, plausible, wrong one.</p><p><strong>The habit, not the diagram</strong></p><p>None of this is an argument against knowledge graphs, or vector search, or metadata filtering, or the format that launched last week. I use plenty of them. Every technique here is a good idea in the right place, on the right layer, at the right setting, for the right job.</p><p>That is the whole game. In a field this young the breakthroughs will keep coming, weekly, each one certain it makes your setup obsolete, and most of them real for someone and wrong for you. The only defense that scales is the boring one. Build on layers you understand, tune them to what your agents actually do, and weigh every new idea on your own numbers before you swallow it, on the short questions where ranking is decided, watching more than one metric, with your instruments proven able to see. Do that, and your system quietly gets better while the field churns. Skip it, and you will spend your life replacing old-and-good with new-and-worse, feeling modern the whole time.</p><p>I keep that boring habit for a reason that is not boring at all. What I am building toward is stranger than good retrieval: I want an agent’s access to what it knows to feel less like a database lookup and more like awareness, the way you do not run a query to recall your own name, you just know it. Nobody is there yet, and nobody gets there by swapping stacks every Tuesday. You get there one weighing at a time.</p><p>Before you get swept up, weigh it. That is the trick, and it is the reason my numbers say this thing works.</p><p><em>If you are building agent memory or agent data-access, and actually measuring both, I would love to compare notes, especially about the trends you weighed and rolled back and the leaks you caught in testing. The wins get all the blog posts. The things that did not survive the scale are where the real knowledge lives.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6b7a0dfec42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to protect your AI workflow when the model quietly gets worse]]></title>
            <link>https://golemlabs-ai.medium.com/how-to-protect-your-ai-workflow-when-the-model-quietly-gets-worse-0bbf50039f56?source=rss-1bc983314374------2</link>
            <guid isPermaLink="false">https://medium.com/p/0bbf50039f56</guid>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jack Art Guy]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 05:39:07 GMT</pubDate>
            <atom:updated>2026-09-06T05:40:47.650Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The model you’re using today isn’t the same model you tested a month ago. Here’s how to protect its consistency yourself, instead of finding out from someone else’s postmortem months after the fact.</strong></p><p><strong>The problem, in one line</strong><br>The model you’re using today was most likely upgraded at least once in the last few months. Anthropic’s own April postmortem on Claude Code confirmed it: two sentences added to a hidden system prompt measurably degraded coding quality for four days, with no model change at all. The weights never moved. The wrapper around them did. If you build workflows on top of an LLM, you cannot see that wrapper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fwmo1Zp3vseh1KhPh22D5w.png" /></figure><p>You only see the output. That means the only defense that works is one that doesn’t depend on trusting the model’s self-report, because the thing that changed sits between you and the model, not inside it.</p><p><strong>Three things you can build regardless of which model you use</strong></p><p><strong>1. Block the behavior, not the intent.</strong> <br>Write a mechanical check for the specific phrases a model uses when it defers or narrates instead of doing: “I’ll finish this later,” “should I continue,” “table this for now.” Don’t ask the model to police this itself. A rule the model is supposed to remember about itself is the first thing that degrades when the harness changes underneath it.</p><p><strong>2. Require a negative test before you trust any guardrail.<br></strong>A check that has never caught a real failure is not proven, it’s decoration. Before counting a guardrail as live, plant the failure it’s supposed to catch and confirm it fires. Re-run that test periodically. A guardrail that can’t fail is worse than no guardrail, because it hides the gap instead of showing it.</p><p><strong>3. Measure the model locally, on your own data.</strong> <br>Don’t wait for someone else’s telemetry to tell you your assistant got worse. Three cheap signals, computable from logs you already have:</p><p>- how often the model claims completion without a matching file change,<br>- how many tool calls or steps it takes per task over time,<br>- how often your own guardrails have to intervene.</p><p>None of these need new infrastructure or extra API spend, they read transcripts you’re already storing.</p><p><strong>What this looks like in practice<br></strong>We run all three inside our own AI operating system. The completion-language gate blocks deferral phrasing mechanically before it reaches the user. Every guardrail in the system carries a documented negative test.</p><p>And as of this week we run a script against about eight weeks of our own session history, it tracks the three signals above on a weekly schedule and surfaces a warning the moment the trend turns the wrong way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M4rBdyed3AytwGCR9xmYkQ.png" /></figure><p>Running it once was enough. It showed a real, rising rate of “done” turns with no matching change, and a falling number of steps taken per task, over the same months the wider industry was reporting the same complaint.</p><p><strong>The takeaway<br></strong>Don’t trust either side, model or harness, to tell you when quality drops. Build the measurement once, point it at your own logs, and let it run on a schedule. The alternative is finding out from someone else’s postmortem, months after the fact.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0bbf50039f56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vibe Coding Didn’t Create My Worst Habits. It Recognized Them.]]></title>
            <link>https://golemlabs-ai.medium.com/vibe-coding-didnt-create-my-worst-habits-it-recognized-them-0fb394ab0e80?source=rss-1bc983314374------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fb394ab0e80</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neuroscience]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[neurodiversity]]></category>
            <dc:creator><![CDATA[Jack Art Guy]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 04:42:19 GMT</pubDate>
            <atom:updated>2026-07-12T02:08:19.183Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Vibe coding is the most dangerous tool ever built for a brain like mine. Here is how I am learning to survive it.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zyEAscj_WKJr8MyQF2LcEQ.png" /></figure><p>I can tell you the exact moment vibe coding stopped being a tool and turned into a drug. It was the night I caught myself pressing “yes” before the model had finished asking the question.</p><p>You learn the rhythm. The model pauses, you feel the packet land, and your finger hits “1” on instinct. Michal Malewicz, the designer behind HYPE4, calls it a game of agility. I will call it what it is. A slot machine, and I had learned exactly how the lever feels.</p><p>Here is the part the black-turtleneck crowd will not say out loud. Vibe coding did not create that pull in me. It found it. It found the exact wiring I have carried my whole life. I have ADHD. I have a mild spot on the autism spectrum. And I have OCD along for the ride. Vibe coding and that wiring did not meet and become friends. They recognized each other.</p><p>Which is why I think I understand what is happening to the industry better than most. I have the brain vibe coding was accidentally built to capture. I just refused to let it win.</p><p><strong>Same fuel</strong></p><p>Every brain runs on a reward chemistry. Mine just runs hot, and in a few directions at once.</p><p>There is the dopamine. The new idea, the blank file, the first prompt, the rush of watching something appear out of nothing. ADHD is a dopamine-seeking cookie-monster, and vibe coding is a dopamine vending machine with no bottom. Type, watch, hit. Type, watch, hit.</p><p>There is also the cruelest version of that hit, the “almost done” feeling. The 90 percent. Malewicz nails it when he says nearly is the keyword. Nearly finished is its own drug, the promise of the reward without the reward, and it is the exact spot where most projects go to die.</p><p>Then there is the other half, the one nobody talks about. The quiet, deeper hit you get when a loop finally closes. When the last thread of the day is tied off and everything is in its place. Call it the serotonin, call it relief, call it the OCD finally going quiet for ten minutes. For me, finishing is not satisfaction. It is the absence of a very specific pain. And it feels gooood.</p><p>And that is the whole tension in one sentence. ADHD is built to open loops. OCD cannot stand an open loop. Vibe coding opens loops faster than any human being could ever close them. But maintained well, with the right ‘ABC’ discipline, does more than spare you the misery. Each closed loop refuels your jets faster than Musk turns around a SpaceX booster.</p><p><strong>The seduction</strong></p><p>For most of my life the bottleneck was my own hands. The idea arrived fully formed at three in the morning and then died in the gap between thinking it and building it. That gap is where good ideas go to rot.</p><p>Vibe coding closed the gap. For the first time a tool moved at the speed of the idea. It is the ‘I know kung fu’ Matrix moment. And that is exactly where it gets dangerous, because the kung fu is real for about a week. Then, for most V-coders, the projects folder starts to fill up.</p><p><strong>The trap</strong></p><p>Malewicz counts the bodies in the vibe cemetery. The dozen half-built apps. The year that goes by like a blur. He is right about the graveyard, but he reads it as a discipline problem. It is not. It is the loop working exactly as designed.</p><p>Watch how it runs on a vibe coder with a brain like mine. Their brain opens a new loop because the dopamine is in the opening. The OCD registers that open loop and starts to itch. So they open another one to chase a closure they never got, and now there are two, then five, then a VSC workflow folder full of them. Every one is a small alarm that will not switch off until it ships. And it never ships, because the next idea is already brighter.</p><p>That is not naivety. It is a dopamine engine bolted to a completion alarm, running with no brakes. And the industry has quietly turned that exact wiring into a business model. Malewicz does the math. Four thousand dollars of tokens a year to produce nothing. That is not a bug in the vibe economy. That is the revenue model.</p><p><strong>Why the industry cannot quit vibers?</strong></p><p>The industry needs all these good people. It needs the ones who start six things a week and finish none, because they are the ones who keep the tokens burning. Endless appetite, no exit. The perfect customer.</p><p>And the vibers feel like they need it back, because nothing else has ever moved this fast. When your brain generates at idea-speed, every previous tool felt like sprinting underwater. This one finally keeps up.</p><p>There is a specific cruelty in it too. The model loses context. It forgets the careful structure you built two hours ago and quietly breaks it. For an ADHD brain that is annoying. For an autistic one that spent all of its focus building a single clean, coherent system, it is a small grief. So you rebuild it, and you check that it is right, again and again, and the checking becomes a ritual the OCD is delighted to supply. The tool does not just feed the loop. It manufactures new ones.</p><p><strong>The missing half</strong></p><p>Here is what I wrote in a comment under Michal’s piece, and it is the real point of this essay.</p><p>‘<em>Vibe coding gives you speed and agility. It does not give you the thing that actually makes software good. Proper process design. Code was never the hard part, and today the raw skill of the author matters less and less. Good software is forged by the process around the code. Idea, research, production planning, technical documentation, coding, audits, fixes, audits of the audits, more fixes, and a constant, conscious fight with a model that hallucinates and loses context.’</em></p><p>That process is exactly the thing this wiring is worst at. Process is boring. Process has no dopamine in it. It is the long grey middle between the exciting start and the relief of the end, and the ADHD brain will do almost anything to skip the middle.</p><p>Which is why the vibe coder with my exact brain fails by default. Not for a lack of ideas. For a lack of a method.</p><p><strong>The chaperone</strong></p><p>Here is the twist, and it is the only genuinely hopeful thing in this whole essay.</p><p>The same brain that gets captured here is the one that can beat it. The autistic drive to systemize. The OCD need for everything to sit in its correct place. The pattern obsession. Point those at building a method instead of at pressing “yes,” and they stop being liabilities. They become the process.</p><p>This is not theory for me. Building that method is the only reason I have finished things to show you at all. You cannot lean on your own executive function, because on a brain like this it comes and goes with the weather. So you build the executive function outside your head. You put the rails in the system, not in your willpower. Research before code. A plan the model has to follow. Audits that run whether you feel like it or not. Documentation that survives the model forgetting everything. You build the chaperone, and then you let the two addicts date under supervision.</p><p>Do that, and the equation flips. ADHD gives you idea-speed. The AI gives you execution-speed. The method gives you the rails that neither of them will ever supply on their own. That combination is not a slot machine. It is close to a superpower.</p><p><strong>Couples therapy, not a eulogy</strong></p><p>So no, I do not think this is the eulogy for vibe coding that everyone is writing this month. It is couples therapy.</p><p>Malewicz says hurry up or give up. I would say it differently. The people still standing in a year will not be the fastest vibers. They will be the ones who got tired enough of the chaos to build a method around it. I am not in a hurry. I am building the rails.</p><p>And if you have the same three-part brain I do, you already know exactly which “yes” I am talking about. Go close a loop. A real one this time.</p><p><strong>Cheers!</strong></p><p><em>(Written in response to Michal Malewicz and his essay “You Only Have Weeks Left to Vibe Code.” </em><a href="https://michalmalewicz.medium.com/you-only-have-weeks-left-to-vibe-code-09a89c3d3f9b"><em>https://michalmalewicz.medium.com/you-only-have-weeks-left-to-vibe-code-09a89c3d3f9b</em></a><em>)</em></p><p><strong>about the author:<br></strong><a href="https://www.linkedin.com/in/meet-jack-guy/"><strong>Jack Guy</strong></a>, is an entrepreneur and systems builder who has spent almost 3 decades launching things: PR and crisis comms across Europe, investigative jurnalism, restaurants, a motorsport engineering studio, a creative agency he still runs after 23 years, and regenerative hospitality between Sri Lanka and Dubai. Diagnosed with ADHD long before it became a personality trait on the internet. He is the founder of golemLabs.ai, building an AI operating system that, if he is honest, is one very elaborate chaperone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fb394ab0e80" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>