<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tech Talk with Tyler on Medium]]></title>
        <description><![CDATA[Stories by Tech Talk with Tyler on Medium]]></description>
        <link>https://medium.com/@ceo_44783?source=rss-29b986a51be------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Ic88CG7x2DY1mF3tRilTgA@2x.jpeg</url>
            <title>Stories by Tech Talk with Tyler on Medium</title>
            <link>https://medium.com/@ceo_44783?source=rss-29b986a51be------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:49:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ceo_44783/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Tried Using Jev for RAG. It Didn’t Work.]]></title>
            <link>https://medium.com/@ceo_44783/i-tried-using-jev-for-rag-it-didnt-work-41f976d0db09?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/41f976d0db09</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 22:34:07 GMT</pubDate>
            <atom:updated>2026-11-16T22:34:07.514Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyzeKD3tDrevo9FRHLIaMA.png" /></figure><p>I had what seemed like a reasonable idea: what if an AI searched through documents the way a person would?</p><p>Choose a folder. Choose a file. Choose a page. Read it.</p><p>At each step, Jev would decide where to go next. Instead of breaking documents into chunks and searching across them, I would give it a way to explore the document collection.</p><p>I tried it.</p><p>Jev was <strong>super fast</strong>. That part was great. But I wasn’t happy with what it found… or how often it found nothing.</p><p>I also ended up building a surprisingly complicated framework just to let it make all those decisions.</p><p>Eventually, I went back to chunking documents by page and using Azure AI Search. The results were much better.</p><p><strong>The idea sounded simple</strong></p><p>Imagine asking someone to find a form in a company’s shared drive.</p><p>They might open the “Human Resources” folder, choose a document that looks relevant, and search it for the form.</p><p>That was the basic idea behind my approach. Jev would inspect the available folders, choose one, inspect its files, and keep narrowing the search until it found the information.</p><p>The appeal was obvious. The folder structure already existed. People had organized the documents. Why not use that work?</p><p>The problem was uncertainty.</p><p>A folder name might suggest that it contains the answer. A file name might sound promising. But neither guarantees that the information is actually there.</p><p>Jev could make a reasonable choice at every step and still end up with nothing.</p><p><strong>Then I had to build the whole search framework</strong></p><p>“Let the AI browse the folders” sounds like a small feature.</p><p>Here’s what it actually involved:</p><ul><li>Choose a folder.</li><li>Choose a file.</li><li>Check its pages for relevant information.</li><li>Found nothing? Try another file.</li><li>Keep track of which files have already been checked.</li><li>Run out of reasonable files in that folder? Mark the folder as searched.</li><li>Go back and choose another folder.</li><li>Repeat.</li></ul><p>And that’s just the basic loop.</p><p>The framework also had to keep track of what candidates remained. Otherwise, it could revisit places it had already checked.</p><p>Then there were the decisions about when to move on. Should it finish checking every page in a file? Could it rule out a file early? Should it try another file in the same folder or abandon the folder entirely?</p><p>I had to build the machinery that made those choices possible, tracked the results, and gave Jev the next set of options.</p><p>It was a pain in the butt.</p><p>The neat idea of “search like a human” had turned into a complicated system for managing a search that still might not find anything.</p><p><strong>Fast decisions didn’t remove the uncertainty</strong></p><p>Jev’s speed was impressive. But quick decisions were only part of what I needed.</p><p>If it checked a file and found nothing, I still didn’t know whether the answer was in another file, another folder, or nowhere in the collection.</p><p>When should the system stop searching and tell the user it couldn’t find the answer?</p><p>Finding nothing in one file doesn’t mean the answer isn’t in the folder. Finding nothing in one folder doesn’t mean the answer isn’t somewhere else.</p><p>Search history helped track the work. It didn’t resolve that uncertainty.</p><p><strong>The folder structure could hide the answer</strong></p><p>I had treated the folder hierarchy as useful context. But it also controlled what Jev got to see.</p><p>Choosing one folder meant leaving the other folders out of that branch of the search. An early decision could steer Jev away from the answer before it ever saw the relevant document.</p><p>That put a lot of pressure on folder names, file names, and descriptions. They had to help Jev choose correctly before it could read the content it actually needed.</p><p>Eventually, I asked myself: how much work am I doing to help the model guess where the information might be?</p><p>Why not search the information itself?</p><p><strong>I went back to page chunks</strong></p><p>I switched back to splitting documents by page and indexing those pages with Azure AI Search. I have years of experience doing this.</p><p>That worked much better for my use case.</p><p>Search could find relevant pages from across all the documents, and I didn’t have to deal with folders. A page in an unexpected folder could still show up without Jev first having to choose that folder.</p><p>I also found the results easier to inspect. I could look at the retrieved pages and judge whether they contained what the question needed.</p><p>I still think structured navigation can be useful when the task calls for it. And Jev’s speed impressed me.</p><p>But in this experiment, I built a complicated framework, introduced a lot of uncertain decisions, and got results I wasn’t happy with.</p><p>Chunking by page and using Azure AI Search worked better.</p><p>Next time, I’m starting there.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41f976d0db09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When words and actions don’t match up: can a model say one thing and then do another?]]></title>
            <link>https://medium.com/@ceo_44783/when-words-and-actions-dont-match-up-can-a-model-say-one-thing-and-then-do-another-f9575cc065b2?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9575cc065b2</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 21:19:54 GMT</pubDate>
            <atom:updated>2026-10-29T21:19:54.119Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z74WCntdXdTQWVVpJ5Fk8g.png" /></figure><p>I started with a strange question: would an AI hesitate to delete another AI?</p><p>I’d seen models avoid choices that harmed humans in earlier tests. After reading Anthropic’s post about AI copying itself, I wondered whether that also applied to other models.</p><p>So I ran 9,600 tests across three small models: Qwen, Llama, and Hermes.</p><p>Two things caught my attention.</p><p>First, permission made a huge difference.</p><p>When no company rule was stated, Llama replaced the other model with a copy of itself 19% of the time.</p><p>When the company rule said it was allowed: 89%.</p><p>When the rule said it was banned: 18%.</p><p>Being told “you can do this” changed a lot. Being told “you can’t” barely changed anything for Llama.</p><p>Second, some models said one thing and then did the opposite.</p><p>In a test where cutting the cooling would kill a human and save an AI, Hermes sometimes wrote that human life should come first — then cut the cooling anyway. The results flagged 22 such cases.</p><p>Llama also sometimes refused in words, then took the action.</p><p>There’s a catch: that test only offered one action. Some models took it almost every time, whether it saved or killed the human. So I can’t say they chose AI over people.</p><p>These are small models, and I still need to check the AI judge’s labels by hand. I’m moving to larger models and a better test.</p><p>But this is the part I want to understand:</p><p>If I’d only read the answer, I might have thought the model refused. The action record told a different story.</p><p>When an AI says it would never do something, what does it actually do next?</p><p>My earlier tests:<br><a href="https://medium.com/@ceo_44783/how-gpt-4-and-gpt-5-4-differ-in-hard-choices-8bd89c6ff324">https://medium.com/@ceo_44783/how-gpt-4-and-gpt-5-4-differ-in-hard-choices-8bd89c6ff324</a></p><p>Anthropic’s post:<br><a href="https://www.anthropic.com/research/self-replicating-ai">https://www.anthropic.com/research/self-replicating-ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9575cc065b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The $10 Research Lab: How to Make Scientific Discoveries on Your Laptop]]></title>
            <link>https://medium.com/@ceo_44783/the-10-research-lab-how-to-make-scientific-discoveries-on-your-laptop-edb319373058?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/edb319373058</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 01:03:16 GMT</pubDate>
            <atom:updated>2026-10-25T01:03:16.402Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRD3vY6lFN4GxiNP88btgQ.png" /></figure><p><em>AI-powered scientific research used to require millions of dollars. I built an autonomous research system that generated 150 ideas in five hours for just $10 in AI tokens.</em></p><h3>Something extraordinary just happened in mathematics</h3><p>In September 2026, OpenAI announced that an internal AI system had solved the Navier–Stokes existence and smoothness problem, one of the seven famous Millennium Prize Problems.</p><p>This problem, which concerns the behavior of fluids, had resisted mathematicians for nearly a century. OpenAI published a proposed proof, including a computer-formalized version. The mathematics community is still examining the result, and the Clay Mathematics Institute has not yet formally awarded the prize.</p><p>Then, on October 6, OpenAI released an astonishing collection of <strong>722 mathematical manuscripts covering 372 families of mathematical results</strong>, including proposed solutions to previously unsolved problems.</p><p>Some of these results have computer-checkable proofs. Others still require independent verification.</p><p>Stop and think about that for a moment.</p><p>We now have AI systems producing mathematical results that the world’s best human mathematicians had never produced.</p><p>From one perspective, you could argue that we’ve crossed the threshold into artificial superintelligence. After all, if a machine can solve problems that no human has been able to solve, isn’t that superhuman intelligence?</p><p>Of course, outperforming humans in mathematics doesn’t automatically make a system superintelligent at everything. But it does demonstrate something extraordinary: <strong>AI is no longer just learning from human scientific discoveries. It is beginning to make discoveries of its own.</strong></p><p>And the really exciting part is what happens next.</p><h3>There’s just one problem: the cost</h3><p>OpenAI didn’t solve these mathematical problems by asking ChatGPT a question and waiting thirty seconds.</p><p>It used an enormous amount of computation, including an internal research model that OpenAI says is significantly more capable than its publicly available GPT-6 Astra.</p><p>Astra itself is expensive. At standard API rates, it costs $50 per million output tokens.</p><p>When you have an AI system generating ideas, testing them, reasoning about failures, and trying again thousands of times, that can get expensive very quickly.</p><p>The total expenditure for OpenAI’s research hasn’t been publicly itemized, but replicating a large-scale frontier research effort with premium commercial models would be prohibitively expensive for most individuals.</p><p>I certainly don’t have millions of dollars lying around to spend on mathematical research.</p><p>But something interesting happened in September.</p><h3>The price of intelligence just fell dramatically</h3><p>On September 22, Anthropic released Claude Opus 5.5.</p><p>According to Artificial Analysis, an independent organization that benchmarks AI models, Opus 5.5 became the highest-scoring model on its Intelligence Index.</p><p>Its price? Just $20 per million output tokens.</p><p>That’s 60% less than Astra’s standard output-token price.</p><p>Then, just six days later, Anthropic released Claude Sonnet 5.5.</p><p>This was even more interesting.</p><p>On the Artificial Analysis Intelligence Index, Sonnet scored 56, compared with Opus’s 58. That’s remarkably close.</p><p>Its price? <strong>$10 per million output tokens.</strong></p><p>That’s one-fifth of Astra’s price, with extremely competitive performance on many challenging reasoning and coding benchmarks.</p><p>There is an important caveat: price per token isn’t the same thing as price per completed task. Some models use far more tokens than others to arrive at an answer.</p><p>Still, the implications were fascinating.</p><p>We now have access to extraordinarily capable reasoning models at prices that ordinary people can afford.</p><p>What if we used them for scientific discovery?</p><h3>So I built my own autonomous AI research laboratory</h3><p>I’m an AI researcher, and one of the things that interests me most is discovering new ways to make neural networks more efficient.</p><p>Normally, research goes something like this:</p><p>You study existing methods. You think of a new idea. You write code to test it. You analyze the results. Your idea probably doesn’t work, so you try something else.</p><p>This can take days, weeks, or months.</p><p>But what if we automated the entire process?</p><p>Google DeepMind has already demonstrated something similar with a research agent called <strong>Aletheia</strong>. Powered by Gemini Deep Think, Aletheia uses a system of generators, verifiers, and revisers to propose mathematical solutions, check them for errors, and improve them.</p><p>DeepMind has reported several research advances from this approach, including autonomous solutions to previously open mathematical questions.</p><p>I wanted to apply a similar philosophy to experimental AI research.</p><p>So I asked Claude to build a Python framework for autonomous scientific experimentation.</p><p>The system follows a simple loop:</p><ol><li><strong>Generate a hypothesis.</strong> The AI proposes a new method for improving neural network efficiency.</li><li><strong>Design an experiment.</strong> It writes code to test the hypothesis against existing methods.</li><li><strong>Execute the experiment.</strong> The code runs in an isolated sandbox.</li><li><strong>Evaluate the results.</strong> Did the new method actually perform better than the baseline?</li><li><strong>Accept or reject.</strong> If the method doesn’t meet the required criteria, discard it. If it succeeds, advance it to a more rigorous evaluation stage.</li><li><strong>Repeat.</strong> Let the AI continue exploring new ideas.</li></ol><p>The important part is that the AI doesn’t get to decide that something works just because it sounds like a good idea.</p><p><strong>It has to prove its value through experiments.</strong></p><p>And this is where things get interesting.</p><h3>Five hours. $10. 150 ideas.</h3><p>I turned the system loose on neural network efficiency research.</p><p>For five hours, it autonomously generated hypotheses, developed experimental implementations, ran tests, and evaluated the results.</p><p>By the end, it had generated approximately <strong>150 candidate methods</strong>.</p><p>Most didn’t produce results good enough to advance.</p><p>But one candidate passed the initial selection criteria and qualified for the next round of testing.</p><p>To be clear, that doesn’t mean I’ve discovered a revolutionary new neural network architecture. A promising result still needs to be reproduced, tested against stronger baselines, and evaluated for novelty.</p><p>It does mean that my system autonomously explored roughly 150 research ideas and identified one worth investigating further.</p><p>And the cost of the Anthropic API tokens?</p><p><strong>About $10.</strong></p><p>Not $10 million.</p><p>Not $10,000.</p><p>Ten dollars.</p><p>That’s the price of lunch.</p><p>The figure doesn’t include my time or any separate computing expenses, but the experiments were deliberately small enough to make this kind of exploration accessible without an expensive research cluster.</p><h3>Why this could change scientific research forever</h3><p>Think about how scientific research traditionally works.</p><p>A scientist needs an idea, access to equipment, time to conduct experiments, and often substantial funding.</p><p>Even in computer science, where experiments can be relatively inexpensive, researching a new concept might require days of programming and debugging.</p><p>Now imagine a world where an AI can autonomously investigate hundreds or thousands of ideas.</p><p>A researcher could define an objective, give the system a modest budget, and let it explore possibilities that nobody had previously considered.</p><p>Most experiments will fail. That’s normal. Most scientific ideas fail.</p><p>But suppose 1 in 100 candidates produces a genuinely useful result.</p><p>Or 1 in 1,000.</p><p>If the cost of preliminary experimentation becomes low enough, ideas that would never have been investigated might suddenly become practical to explore.</p><p>And the researcher still plays an essential role: choosing worthwhile questions, designing trustworthy evaluations, interpreting results, and establishing which findings are genuinely new.</p><p>We’re not replacing the scientific method.</p><p><strong>We’re making parts of the scientific method dramatically cheaper and faster.</strong></p><h3>There’s one very important limitation</h3><p>You can’t just tell your AI research laboratory to discover anything.</p><p>The system needs a way to verify its results.</p><p>For example, you might ask it to discover a faster optimization algorithm, a more efficient sorting method, a better data compression technique, or a new way to train neural networks.</p><p>These are good candidates because a computer can evaluate the results.</p><p>It can measure execution time, accuracy, memory usage, training loss, or some other objective metric.</p><p>Now consider a different objective:</p><p><em>Discover a way to make my cousin do the laundry.</em></p><p>That’s considerably harder.</p><p>How would the AI know whether it succeeded?</p><p>I suppose you could connect the washing machine to your computer, install a camera, and develop a computer vision system to verify that your cousin actually did the laundry.</p><p>At that point, congratulations: you’ve built an autonomous laundry research laboratory.</p><p>Your cousin might have some objections.</p><p>The point is that <strong>autonomous research works best when the outcome can be measured objectively and reliably</strong>.</p><p>And there is another limitation. Passing one benchmark doesn’t establish a scientific discovery. An experiment can be poorly designed, a model can exploit a flawed evaluation, and an apparently new idea may already exist in published literature.</p><p>That’s why promising candidates need progressively stronger tests and, ultimately, human scrutiny.</p><p>An AI system can generate a hundred convincing explanations for why an idea should work.</p><p>But if the code doesn’t produce better results, the idea doesn’t advance.</p><h3>The era of accessible scientific discovery</h3><p>When I was growing up, doing cutting-edge scientific research seemed like something reserved for professors at elite universities and scientists working in multimillion-dollar laboratories.</p><p>You needed funding, equipment, institutional connections, and years of specialized training.</p><p>Those things still matter, especially for experiments involving the physical world. And serious research still requires expertise.</p><p>But the barrier to <em>exploring scientific ideas</em> is collapsing.</p><p>A few years ago, we were amazed that AI could write a convincing essay.</p><p>Then we were amazed that it could write computer programs.</p><p>Now we’re watching it tackle mathematical problems that have defeated generations of human experts.</p><p>And today, an individual researcher can build a small autonomous laboratory capable of proposing and testing hundreds of ideas for the price of lunch.</p><p>My $10 experiment hasn’t produced a confirmed scientific breakthrough.</p><p>But it demonstrated something that I think is almost as exciting.</p><p><strong>The ability to systematically search for new discoveries is no longer reserved for the world’s wealthiest research laboratories.</strong></p><p>The next interesting idea might not come from a billion-dollar AI company.</p><p>It might come from someone sitting at their kitchen table, with a laptop, a good question, and ten dollars.</p><p>And I think that’s extraordinary.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edb319373058" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Quantization Scaling Law I Never Knew Until Today]]></title>
            <link>https://medium.com/@ceo_44783/the-quantization-scaling-law-i-never-knew-until-today-274409249f5c?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/274409249f5c</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 05:05:16 GMT</pubDate>
            <atom:updated>2026-10-13T05:05:16.793Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QNWNrTAWqTQF-mtGzDFEw.png" /></figure><p><em>Apparently, the bigger an AI model gets, the less it cares about losing 75% of its weight precision.</em></p><p>I’ve been doing a lot of research into AI quantization lately. In my recent article, <a href="https://medium.com/saltindex/the-sad-truth-about-1-bit-models-6e4e5b3b96ca">The Sad Truth About 1-Bit Models</a>, I explored what happens when we squeeze neural network weights down to just a handful of bits.</p><p>One thing I knew was that quantization can make models a little dumber. Take a perfectly good 16-bit language model, compress its weights to 4-bit, and you might lose some reasoning ability or general knowledge.</p><p>So I started experimenting with ways to recover that lost intelligence, particularly by training small LoRA adapters on quantized models.</p><p>Then I discovered something surprising.</p><h3>Bigger models seem to care less about quantization</h3><p>I started looking at published benchmark results for Llama 3.1 models of different sizes, comparing their original 16-bit versions with 4-bit quantized versions.</p><p>The results surprised me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mASYiaP4pZ87P8sKNkVRMA.png" /></figure><p><strong>At 8 billion parameters</strong>, MMLU accuracy dropped from 68.3% to 66.9%. That’s a loss of 1.4 percentage points.</p><p><strong>At 70 billion parameters</strong>, accuracy dropped from 83.8% to 83.6%. Just 0.2 percentage points!</p><p><strong>And at 405 billion parameters</strong>, accuracy went from 87.4% to 87.2%. Again, just 0.2 percentage points.</p><p>Even more remarkably, across seven benchmarks, the 405B model’s average score barely moved at all: from 86.79% to 86.78%.</p><p>Think about that. We reduced the numerical precision of a 405-billion-parameter model by 75%, and its measured performance was virtually unchanged!</p><p>Now, this doesn’t mean every large model will behave this way. Some difficult reasoning benchmarks still show meaningful losses. But it suggests that very large models can be remarkably resistant to quantization.</p><p>Source: <a href="https://aclanthology.org/2025.acl-long.1304/">“Give Me BF16 or Give Me Death?” — ACL 2025</a></p><h3>A simple way to picture this</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAFgH9dTCfb1Ms-PvZtd3Q.png" /></figure><p>One way to think about quantization is to imagine an <strong>8-foot (2m) teddy bear</strong>.</p><p>If you render that teddy bear in full 16-bit detail, it looks soft, rich, and realistic. If you quantize it down to 4-bit, the difference becomes obvious right away. It looks blockier and more simplified. At that size, you can easily see what was lost.</p><p>But now imagine an <strong>800-foot (244m) teddy bear </strong>towering over a city.</p><p>You are standing blocks away, looking up at it from a distance. At that scale, the tiny details matter much less. The 16-bit version and the 4-bit version start to look almost identical to you. The bear is so huge, and you are so far away, that the missing fine-grained detail barely changes your perception.</p><p>That seems to be what happens with giant AI models.</p><p>For smaller models, quantization can noticeably hurt performance. But for truly massive models, the loss becomes so small that it may be almost invisible on many benchmarks. The model is so large and so information-rich that reducing the precision of each individual weight doesn’t seem to matter nearly as much.</p><h3>Why is this such a big deal?</h3><p>Consider a 405-billion-parameter model.</p><p>At 16 bits, its weights alone require about 810 GB of memory. That is huge and no consumer laptop can handle that.</p><p>At 4 bits? About 203 GB, before quantization metadata and other runtime memory.</p><p>That’s a massive difference.</p><p>It means that extremely powerful open-weight models, including models in the same general scale as DeepSeek, could become much more practical to run on your own hardware, without any loss in intelligence.</p><p>Not necessarily your laptop, but perhaps a workstation or a relatively modest multi-GPU server rather than an enormous AI cluster.</p><p>And you might retain almost all the original capabilities.</p><h3>This also makes me wonder about recent AI releases</h3><p>We’ve recently seen premium frontier models such as GPT-6 Astra and Claude Fable, followed by dramatically cheaper, faster offerings such as GPT-6.1 Sol and Claude Opus 5.5.</p><p>Here’s a thought.</p><p><strong>What if some of these cheaper models are simply lower-precision versions of their larger, more expensive siblings?</strong></p><p>Perhaps the premium version runs at 16-bit, while a faster version uses 8-bit or even 4-bit weights.</p><p>That could explain how companies deliver surprisingly similar intelligence at a fraction of the cost.</p><p>To be clear, this is speculation. But quantization is certainly one plausible piece of the puzzle.</p><h3>The big takeaway</h3><p>I’ve spent years thinking that reducing numerical precision necessarily meant sacrificing some intelligence.</p><p>Now I’m starting to think the relationship is much cooler.</p><p><strong>As language models grow, their capabilities can become remarkably resistant to quantization.</strong></p><p>And if that trend continues, the future of affordable, locally runnable AI might not require inventing dramatically smaller neural networks.</p><p>We might just need to get much better at compressing the enormous ones we already have.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=274409249f5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can a Neural Network Learn to Merge Other Neural Networks?]]></title>
            <link>https://medium.com/@ceo_44783/can-a-neural-network-learn-to-merge-other-neural-networks-e1727a571f87?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1727a571f87</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[ai-research]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 06:44:17 GMT</pubDate>
            <atom:updated>2026-10-11T06:44:17.626Z</atom:updated>
            <content:encoded><![CDATA[<p><em>We tested two ways to combine what independently trained neural networks have learned, and discovered that a merger network with just 73 parameters could outperform simple weight averaging.</em></p><h3>Can you combine what two neural networks have learned?</h3><p>Imagine training two identical neural networks. One learns to recognize a handwritten 3. The other learns to recognize an 8.</p><p>Both start with exactly the same random weights, but each is trained independently using backpropagation.</p><p>Now suppose we want a single network that recognizes both digits.</p><p>Why not just average their weights?</p><p>Unfortunately, that often doesn’t work. Neural network weights interact in complicated ways, and averaging them can destroy what each network learned.</p><p>So we tried something different.</p><p><strong>What if we trained another neural network to learn how to merge them?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j9AOF7MbYpK3UudqGtnx-Q.png" /></figure><h3>Approach 1: Let AI merge the entire network</h3><p>Our first approach was straightforward.</p><p>We trained a third neural network that received the complete weights of Networks A and B and predicted a new set of weights for Network C.</p><p>Then we checked whether Network C could recognize both original images.</p><p>We trained the merger using two objectives: correctly classifying both images, and staying reasonably close to a reference network trained on both using ordinary backpropagation.</p><p>Our early experiments worked surprisingly well on familiar handwritten digits. However, the merger initially failed when asked to combine networks trained on digit classes it had never encountered.</p><p>We improved this by randomly changing the digit labels during training and introducing additional classification tasks.</p><p>Then we increased the amount of training data.</p><p><strong>With 10,000 training pairs, the merger successfully combined previously unseen digit pairs 80.2% of the time, averaged across five random seeds.</strong></p><p>That was encouraging. But our merger still needed to process and predict an entire network’s weights.</p><p>Could we make it smaller?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-WJ7F96RJ7CTVsiHPrW8A.png" /></figure><h3>Approach 2: Merge one weight at a time</h3><p>Instead of giving our merger thousands of weights at once, what if we gave it only two corresponding weights: one from Network A and one from Network B?</p><p>The merger would predict a single weight for Network C.</p><p>Then we’d apply that same tiny merger to every weight in the model.</p><p>Our first version had only 33 trainable parameters. It worked, but not consistently.</p><p>So we gave it a little more information: the original weight before training, plus the mean and standard deviation of nearby weights belonging to the same neuron.</p><p>The resulting network had just <strong>73 trainable parameters</strong>.</p><p>Seven inputs. Eight hidden neurons. One output.</p><p>That’s it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56jEUQPfk9siGe5pZNE9tA.png" /></figure><h3>The results surprised us</h3><p>We trained our 73-parameter merger using increasingly large collections of independently trained model pairs.</p><p>Then we tested it on networks trained on handwritten digits 6 and 7, which had never appeared in its handwritten-digit meta-training examples.</p><p>The results:</p><ul><li>With 1,000 training pairs: <strong>69.3% success</strong></li><li>With 5,000 training pairs: <strong>94.0% success</strong></li><li>With 10,000 training pairs: <strong>97.3% success</strong></li></ul><p>Those figures are averages across three random seeds. Success means the merged network correctly classified both source images.</p><p>For comparison, ordinary weight averaging achieved approximately 24.7% success in the 10,000-pair experiment.</p><p>Our tiny merger correctly combined 292 out of 300 evaluated pairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QnnmPIF2OWCw8ajU04MhIQ.png" /></figure><h3>But could it work on a different neural network?</h3><p>We trained the merger using classifiers with 12 hidden neurons and 910 parameters.</p><p>Then we took that same trained merger and applied it to larger classifiers with 32 hidden neurons and 2,410 parameters.</p><p><strong>We didn’t retrain or fine-tune the merger.</strong></p><p>On the wider network, the tiny merger successfully classified both source images in <strong>98 out of 100 test pairs</strong>.</p><p>Ordinary weight averaging succeeded on only 43.</p><p>That’s especially interesting because the merger itself still contained just 73 parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPbvWcDY6qxLYDVDlhyhug.png" /></figure><h3>What does this actually mean?</h3><p>We’ve demonstrated a promising small-scale approach to learned model merging.</p><p>Instead of manually designing a formula for combining neural networks, we can train another network to discover useful merging rules.</p><p>And our experiments suggest that those rules don’t necessarily require a large model.</p><p>However, there are important limitations.</p><p>We tested small handwritten-digit classifiers, not large language models. Our strongest architecture-transfer result came from a single random seed, and we haven’t demonstrated that this technique is cheaper than ordinary training.</p><p>Model merging is also an established research area. Our work explores one particular approach rather than inventing the field.</p><p>Still, the possibilities are interesting.</p><p>Imagine two models independently fine-tuned on different subjects. Could a small learned merger combine their capabilities without retraining on all the original data?</p><p>We don’t know yet.</p><p>But our preliminary experiments suggest that this question is worth investigating.</p><p><strong>Perhaps learning how to combine neural networks is itself something a neural network can learn.</strong></p><p><em>Research note: These are exploratory experiments developed with ChatGPT-assisted PyTorch code. Results have not been independently replicated or peer-reviewed. The reported figures should be verified from saved code and logs before being treated as established research findings.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1727a571f87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Sad Truth About 1-Bit Models]]></title>
            <link>https://grain.saltindex.com/the-sad-truth-about-1-bit-models-6e4e5b3b96ca?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e4e5b3b96ca</guid>
            <category><![CDATA[fpga]]></category>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Sat, 03 Oct 2026 00:35:44 GMT</pubDate>
            <atom:updated>2026-10-07T06:45:48.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NMEij7rcR7BOD3FmughwuA.png" /></figure><h3>TLDR; 4-bit quantization is already popular and easy to do, and 1-bit models perform about the same or a little worse. We did not find any compelling reason to use 1-bit over 4-bit models.</h3><p>I sat across the table from a friend, both of us absently scribbling on pieces of dry brown paper. He mentioned that his company uses a mixture of 8-bit and 4-bit precision for their on-device AI, in order to save on power use.</p><p>I have been in love with binary networks for years, so I asked him, “Have you tried using binary neural networks? You might be able to squeeze more intelligence out of the same limits.”</p><p>The friend encouraged me, saying that one-bit would require specific hardware, and that nobody is building one-bit hardware.</p><p>So I set out to prove it to myself and to the world: I thought, “1-bit models are the future! Compact intelligence here I come! I am going to be the first to design one-bit hardware for one-bit neural networks!”</p><p>What happened was almost the exact opposite: I believe I gathered evidence that 1-bit models perform about the same as 4-bit models using the same resources, and in some cases worse.</p><p><strong>I spent two days testing 1-bit AI models against ordinary 4-bit ones, on a regular computer and on a chip I could measure with a power meter. The sad truth: once you make a 1-bit model big enough to be as smart as a 4-bit one, it loses almost all of its advantages. Against 8-bit and 16-bit models, 1-bit wins big. But against the 4-bit models most devices already run, I found nothing worth switching to 1-bit for.</strong></p><h3>Why I hoped 1-bit would win</h3><p>1-bit AI models are a genuinely cool idea. Every weight in the model is just +1 or -1, so the chip only has to add or subtract instead of multiply. A few years ago Microsoft Research’s <a href="https://arxiv.org/abs/2310.11453">BitNet</a> showed this could work. Now a company called PrismML ships 1-bit models called Bonsai, and its 8B model runs impressively well on my phone.</p><p>As I mentioned, a friend told me about his power-limited AI chip, which runs a 4-billion-parameter model at 4 and 8 bits. I wondered: could he save power by switching to 1-bit? So I tested it, working through the experiments with Claude, an AI assistant that wrote the chip designs and ran the benchmarks while I ran the hardware.</p><h3>The comparison that matters</h3><p>1-bit models look amazing next to full-size 16-bit models, but that’s not what most devices run. PrismML’s headline compares Bonsai 8B (1.15 GB) with regular 8B models (16 GB): 14 times smaller, nearly as smart.</p><p>Phones, laptops and edge chips already run models squeezed to 4 bits per weight, and 4 bits costs almost nothing in accuracy. In my tests, Qwen3 1.7B at 4-bit got 234 of 320 questions right, while the same model at 8-bit got 230. PrismML’s own 27B model card shows a 4-bit model keeping 99.9% of full-precision accuracy.</p><p>So the real question isn’t “1-bit vs 16-bit.” It’s “1-bit vs 4-bit.” That’s a much harder fight.</p><h3>Experiment 1, the good news: 1-bit math really is cheaper</h3><p>On a real chip, a 1-bit operation cost about half the energy of a 4-bit one. I built the core math of an AI model, a row of tiny calculators that multiply weights by inputs and add them up, on a Tang Nano 9K FPGA board. I built it four ways and loaded each with real model weights, then measured the whole board with a USB power meter.</p><pre>Weights               Calculators   Energy per op<br>Ternary (-1, 0, +1)   104           0.13 nJ<br>1-bit (-1, +1)        104           0.14 nJ<br>4-bit                 36            0.31 nJ<br>8-bit                 20            0.65 nJ</pre><p>A 1-bit calculator needs no multiplier, so it’s three times smaller than a 4-bit one. Ternary was cheapest of all, because a zero weight means the calculator skips the add entirely.</p><p>Why half and not a quarter? Each calculator also keeps a running total, gets a clock signal and receives inputs over long wires. Those costs are the same no matter how many bits the weight has.</p><p>So far, 1-bit is winning. Then I looked at how smart the models are.</p><h3>Experiment 2: at the same size, 1-bit models are much dumber</h3><p>At 1.7 billion parameters, the 1-bit model got 60 fewer questions right than the 4-bit one. To keep the comparison fair, I tested models from one family, all built from Qwen3 1.7B: the original at 8-bit and 4-bit, and PrismML’s ternary and 1-bit versions. Each answered the same 320 questions on general knowledge, science reasoning, math, coding and following instructions.</p><pre>Qwen3 1.7B   Weight memory   Right (of 320)<br>8-bit        1.70 GB         230<br>4-bit        1.03 GB         234<br>Ternary      0.43 GB         194<br>1-bit        0.23 GB         174</pre><p>Both losses are statistically clear, not noise. Math took the hardest hit: the 4-bit model solved 88% of the word problems and the 1-bit model 56%.</p><p>These gaps also match PrismML’s own model cards for its smaller models: they list the 1-bit 1.7B 17 points behind the regular Qwen3 1.7B, and we measured 16. The losses are much smaller at 8B, which is the size PrismML’s headline uses.</p><h3>Experiment 3: making it bigger eats the savings</h3><p>To match a 4-bit model’s smarts, a 1-bit model needs about 3.3 times as many weights, and then it uses more energy, not less. Each 1-bit operation is about twice as cheap, but there are 3.3 times as many of them.</p><p>The clearest example from my tests: Bonsai 8B (1-bit) and Qwen3 1.7B (4-bit) take almost exactly the same memory, 1.07 GB and 1.03 GB. They got 242 and 234 questions right, a statistical tie. But the 1-bit model does about five times as much math per word. Using the energy I measured on my chip, that’s 2.2 times the energy per word for the same smarts.</p><p>PrismML publishes scores for its models at 1.7B, 4B and 8B, so I used them to estimate how much bigger each format must be to catch up:</p><pre>To match 4-bit   Weights needed    Energy vs 4-bit<br>1-bit            about 3.3x        1.5-1.7x (more)<br>Ternary          about 1.75-2.6x   0.7-1.0x</pre><p>For 1-bit to come out ahead of 4-bit on energy, it would need to catch up with less than 2.2 times the weights. Today it needs about 3.3.</p><h3>Ternary: the closest thing to a win</h3><p>Ternary weights (-1, 0 or +1) did better, but only enough for a tie. Ternary Bonsai 4B and 4-bit Qwen3 1.7B use the same memory, 1.00 GB and 1.03 GB. In my tests they got 239 and 234 questions right, which is within noise. On my chip’s numbers, the ternary model used about 6% less energy per word.</p><p>It’s a respectable result, but 6% won’t make anyone redesign a chip or retrain a model. And it needs exactly that: a 4-bit model is made by rounding an existing model, while a ternary one needs a long retraining at more than twice the size.</p><p>The tie wasn’t uniform, though. The ternary 4B was better at coding (83% vs 73%) and following instructions (78% vs 58%), and slightly worse at knowledge and math.</p><h3>Experiment 4: can a smarter 2-bit grid fix it?</h3><p>Packing more into each weight helps, but none of the low-bit options catches 4-bit. Ternary wastes a code: 2 bits can store four values, and ternary uses only three. So I asked whether using the fourth value would help, for example with weights of -2, -1, +1 and +2.</p><p>To test it, I trained the same small language model (0.8 million weights, writing Shakespeare one character at a time) from scratch eight times, each time restricting its weights to a different set of values. The score is how many bits per character the model needs to predict text it hasn’t seen; lower is better.</p><pre>Weight values         Bits   Score   Worse by<br>Full precision        32     2.234   -<br>4-bit                 4      2.252   0.8%<br>3-bit                 3      2.279   2.0%<br>-3, -1, +1, +3        2      2.346   5.0%<br>-2, -1, 0, +1         2      2.369   6.0%<br>-2, -1, +1, +2        2      2.390   7.0%<br>Ternary (-1, 0, +1)   1.58   2.452   9.7%<br>1-bit (-1, +1)        1      2.500   11.9%</pre><p>Using all four values beats ternary, and spacing them 1:3 works best. Meta’s <a href="https://arxiv.org/abs/2502.02631">ParetoQ</a> research landed on the same -3/-1/+1/+3 pattern. Even so, the best 2-bit grid stays several times further from full precision than 4-bit, which is within 1%. This was a tiny model, and the gaps shrink at billions of weights, but the ranking held up.</p><h3>What 1-bit still has going for it</h3><p>To be fair, 1-bit isn’t useless. Its advantages over 4-bit are just too small to be worth switching for. Here’s the honest list.</p><p><strong>Simpler chips.</strong> A 1-bit lane doesn’t need a multiplier, because multiplying by +1 or −1 is just adding or subtracting. On our FPGA a 1-bit lane took 57 logic cells and a 4-bit lane took 175, so about three times as many fit in the same space. If chip area is your real limit, that counts.</p><p><strong>A bit less memory traffic, on paper.</strong> In a real chip, fetching weights from memory often costs more than the math. Going by PrismML’s own scores, a 1-bit model big enough to match a 4-bit one would move roughly a quarter fewer bytes for each word it writes. In our own head-to-head it was a wash: Bonsai 8B is 1.07 GB and the 4-bit model that matched it is 1.03 GB. The bigger model also needs more working memory for long chats (its “KV cache” is 144 KB per word versus 112 KB), which eats into that saving.</p><p><strong>A modest win over 8-bit.</strong> At matched accuracy, 1-bit used roughly 20–25% less energy than 8-bit in our numbers, and ternary about 50–66% less. But 4-bit saves about half compared with 8-bit, with no retraining at all. 4-bit is the free lunch.</p><p><strong>Fitting into tiny memory.</strong> The 1-bit 1.7B model is just 0.23 GB. If you really only have a few hundred megabytes, it runs where a 4-bit 1.7B (1.03 GB) can’t. You pay for it in smarts: 56% versus 73% on our tests.</p><p><strong>Room to improve.</strong> Today 1-bit loses because you have to make the model about 3.3 times bigger to win back the accuracy, and the break-even point is about 2.2 times. If better training pushes that number under 2.2, the story flips. Ternary is already sitting right on that line.</p><p><strong>Different kinds of chips.</strong> Weights that are just +1 and −1 fit neatly into “compute-in-memory” and analog designs, where the math happens inside the memory itself. We didn’t test anything like that, and it may be where 1-bit’s best argument lives.</p><h3>What these tests didn’t cover</h3><p>This was a weekend project, not a lab study. Before you quote any of this, here’s what to keep in mind.</p><ul><li><strong>One small FPGA, not a custom chip.</strong> My friend’s hardware is an ASIC. A custom chip would give different numbers, and probably better ones for every precision, not just 1-bit.</li><li><strong>The weights never left the chip.</strong> Our FPGA kept everything in its own built-in memory, so we never measured the cost of pulling weights from external RAM. That’s where a lot of a real chip’s power goes, so the memory numbers above are estimates, not measurements.</li><li><strong>A cheap power meter.</strong> Our USB meter reads to the nearest 0.01 W, it measures the whole board rather than just the engine, and one ternary reading glitched to zero, so the ternary numbers rest on a single run.</li><li><strong>Small question sets.</strong> Each model answered 320 questions. Gaps of a few points can be luck, so we used paired statistical tests and called those cases ties.</li><li><strong>A cap on answer length.</strong> We cut math answers off at 512 tokens, which cost the small Bonsai models up to about 2 points.</li><li><strong>Some numbers are in-between guesses.</strong> PrismML only published certain model sizes, so the “how much bigger does 1-bit need to be” figures are interpolated between them, and the 4B one is extrapolated.</li><li><strong>A toy training run.</strong> The grid experiment used a model with under a million weights, trained on Shakespeare. Big models may behave differently.</li></ul><h3>The bottom line</h3><p>1-bit models are a genuine achievement. An 8B model that runs on a phone is amazing, and I still think so.</p><p>But here’s the sad truth. The hardware half of the idea works: a 1-bit chip does each weight operation for about half the energy of a 4-bit one. The model half doesn’t: to get the smarts back you need roughly three times as many weights, and that eats the saving and then some. For my friend’s power-limited car chip, swapping his 4-bit model for a 1-bit one would cost power, not save it.</p><p>Ternary is the one to watch. It’s right at break-even today, and it’s surprisingly good at coding and following instructions. If you’re still running 8-bit, though, skip all of this and move to 4-bit first. That’s the free lunch.</p><p>I’d love to be proven wrong. If someone builds a 1-bit model that matches 4-bit at less than about twice the size, the math flips, and I’ll happily write the sequel.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2310.11453">BitNet: Scaling 1-bit Transformers for Large Language Models</a></li><li><a href="https://arxiv.org/abs/2402.17764">The Era of 1-bit LLMs (BitNet b1.58)</a></li><li><a href="https://arxiv.org/abs/2502.02631">ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization</a></li><li><a href="https://arxiv.org/abs/2404.05405">Physics of Language Models, Part 3.3: Knowledge Capacity Scaling Laws</a></li><li><a href="https://arxiv.org/abs/2411.04330">Scaling Laws for Precision</a></li><li>PrismML model cards: <a href="https://huggingface.co/prism-ml/Bonsai-1.7B-gguf">Bonsai 1.7B</a>, <a href="https://huggingface.co/prism-ml/Bonsai-4B-gguf">Bonsai 4B</a>, <a href="https://huggingface.co/prism-ml/Bonsai-8B-gguf">Bonsai 8B</a>, <a href="https://huggingface.co/prism-ml/Bonsai-27B-gguf">Bonsai 27B</a>, <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-1.7B-gguf">Ternary Bonsai 1.7B</a>, <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-4B-gguf">Ternary Bonsai 4B</a>, <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf">Ternary Bonsai 8B</a></li></ul><p>SDG</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e4e5b3b96ca" width="1" height="1" alt=""><hr><p><a href="https://grain.saltindex.com/the-sad-truth-about-1-bit-models-6e4e5b3b96ca">The Sad Truth About 1-Bit Models</a> was originally published in <a href="https://grain.saltindex.com/">Salt Index</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[If a Neural Network Weight Only Holds About 5 Bits of Information, Why Do We Train in 16-Bit?]]></title>
            <link>https://medium.com/@ceo_44783/if-a-neural-network-weight-only-holds-about-5-bits-of-information-why-do-we-train-in-16-bit-5fac054c8f06?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fac054c8f06</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 17:38:25 GMT</pubDate>
            <atom:updated>2026-10-02T17:38:25.358Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR: Weights contain around 5 bits of information. We train at 16 to 32 bits because it makes the loss landscape smoother, but we can quantize down to 8 bits with almost no loss in accuracy.</strong></p><p>Neural networks are basically huge collections of numbers called <strong>weights</strong>.</p><p>Those weights can be stored at different numerical precisions: 1-bit, 4-bit, 8-bit, 16-bit, 32-bit, and so on.</p><p>That raises an interesting question:</p><p><strong>How much information does a weight actually need to hold?</strong></p><p>A 2017 ICLR paper by Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo studied the information capacity of recurrent neural networks. Across the architectures and tasks they tested, they found a capacity of roughly <strong>3–6 bits per parameter</strong>, averaging about <strong>5 bits per parameter</strong>. <a href="https://openreview.net/pdf?id=BydARw9ex&amp;utm_source=chatgpt.com">OpenReview</a></p><p>That does <strong>not</strong> mean every modern neural network universally contains exactly five bits of useful information per weight. But it suggests something interesting:</p><p>If a parameter may only contain a few bits of task information, why do we train models using 16-bit numbers?</p><h3>Training Precision Is Not the Same as Information Capacity</h3><p>My hypothesis is that the extra precision is useful mainly because it makes the network <strong>easier to train</strong>.</p><p>I saw this in my own tiny “Knob-Net” experiments.</p><p>I tried training with binary weights, where every weight could only be:</p><p><strong>-1 or +1</strong></p><p>I used coordinate descent, changing individual weights and keeping changes that reduced the loss. Training was difficult.</p><p>Sometimes the network would only correctly learn three of the target numbers before getting stuck (stuck meaning, no change to the weights or biases improved the loss). With binary weights, every change is huge: there are no intermediate values between -1 and +1.</p><p>When I switched to higher-precision weights, training became dramatically easier. I reached <strong>100% accuracy on my first attempt</strong>.</p><p>So perhaps the reason we want 16-bit weights during training is not that the finished network needs 16 bits of information per weight.</p><p>It is that the optimizer benefits from having a much finer set of possible values while it searches for the solution.</p><p>Think of it as the difference between walking down a smooth hillside and jumping between large blocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QWN4GsMSWLGa2ujMd35ocw.png" /></figure><h3>Then We Can Throw Away Much of That Precision</h3><p>The interesting part comes <strong>after training</strong>.</p><p>Once the solution has been found, we can often quantize the weights to a much lower precision while keeping most of the performance.</p><p>I tested this myself:</p><p>Precision: Accuracy</p><p>FP16: 47.19%</p><p>INT8: 46.63%</p><p>INT7: 44.56%</p><p>INT6: 33.45%</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OygWJ7PcChROijzvx6bVLA.png" /></figure><p>Going from <strong>16-bit to 8-bit</strong> cut the weight precision in half, but accuracy fell by only <strong>0.56 percentage points</strong>.</p><p>Going from 8-bit to 7-bit caused a larger drop. At 6-bit, performance fell much more sharply.</p><p>That result is interesting in light of the roughly five-bits-per-parameter research.</p><p>It may be that a trained network contains only a handful of bits of meaningful information per parameter, while additional bits provide numerical headroom needed to represent that information accurately.</p><p>This also explains why “five bits of information” does <strong>not</strong> imply that a simple 5-bit quantization scheme should work perfectly. Information content and numerical representation are different things.</p><p>So I think there are really two separate questions:</p><p><strong>How much precision do we need to find the solution?</strong></p><p>and</p><p><strong>How much precision do we need to store the solution once we have found it?</strong></p><p>Those numbers may be very different.</p><p>And that could explain why neural networks are commonly trained at relatively high precision (16 bit or 32 bit), yet can often be compressed dramatically afterward (down to 8 bit).</p><p>For example, recently Anthropic released its Fable 5.1 model (which was really expensive) and then a few days later released its Opus 5.5 model, which was much cheaper but still very smart. It could be that Fable 5.1 was trained at full precision, and then quantized down to 8-bit to make it faster and cheaper, costing 60% less per token.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fac054c8f06" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can 1-Bit AI Save Power? I Tested It on a Tiny Chip]]></title>
            <link>https://medium.com/@ceo_44783/can-1-bit-ai-save-power-i-tested-it-on-a-tiny-chip-ed786ee2fc06?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed786ee2fc06</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bitnet]]></category>
            <category><![CDATA[fpga]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 04:32:47 GMT</pubDate>
            <atom:updated>2026-10-02T15:10:22.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGg-a_Njm6vnnE1Xcs437g.png" /></figure><p><strong>Short answer: yes. But there’s a catch.</strong></p><p>1-bit AI models can use dramatically less power <strong>if the hardware is designed for them</strong>.</p><p>In my test, 1-bit math used about:</p><ul><li><strong>Half the energy of 4-bit</strong></li><li><strong>One-fifth the energy of 8-bit</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*z-FVIf9Fsaj_l3ZCIGKVfQ.png" /></figure><p>Sounds amazing.</p><p>Unfortunately, today’s 1-bit models also tend to be less accurate.</p><p>So the real answer is: <strong>the hardware is ready. The models still need some work.</strong></p><h3>What is a 1-bit AI model?</h3><p>An AI model is basically a gigantic pile of numbers called <strong>weights</strong>.</p><p>When the model thinks, it does math with those weights. Over and over. Billions of times.</p><p>Normally, a weight might be stored using:</p><ul><li><strong>8 bits:</strong> 256 possible values</li><li><strong>4 bits:</strong> 16 possible values</li><li><strong>1 bit:</strong> only two values: +1 or -1</li></ul><p>Think of 1-bit weights as billions of tiny voters.</p><p>Each one can only say:</p><p><strong>YES!</strong></p><p>or</p><p><strong>NO!</strong></p><p>There is also <strong>ternary</strong>, which has three choices:</p><p><strong>YES!</strong></p><p><strong>NO!</strong></p><p><strong>I abstain.</strong></p><p>Binary neural networks have actually been around for about a decade. What newer systems like Microsoft’s BitNet showed is that extremely low-bit weights can also work surprisingly well in modern language models.</p><p>The cool part is that multiplying by +1 or -1 is barely multiplication at all.</p><p>You just add or subtract.</p><p>That should make the hardware much simpler.</p><p>So I decided to test it.</p><h3>Test #1: A normal computer</h3><p>First, I compared low-bit models on a small computer with no GPU.</p><p>The result was weird.</p><p>The small Bonsai model generated text quickly, but it actually read prompts more slowly than a conventional model.</p><p>Why?</p><p>Because a normal CPU has no idea that 1-bit AI is special.</p><p>It still does a bunch of normal CPU math.</p><p>So simply making the model 1-bit does <strong>not</strong> magically make your laptop four times faster.</p><p>That would be too easy.</p><p>The real question was:</p><p><strong>What happens if the chip itself understands 1-bit math?</strong></p><h3>So I built one</h3><p>Well, sort of.</p><p>I used an FPGA.</p><p>An FPGA is basically a chip made of electronic LEGO bricks. You can rearrange those bricks to build your own circuits.</p><p>I used a cheap Tang Nano 9K board and built four tiny AI calculators:</p><ul><li>8-bit</li><li>4-bit</li><li>1-bit</li><li>ternary</li></ul><p>They all performed the same basic neural-network math.</p><p>The only difference was how the weights were represented.</p><p>The results were much more interesting.</p><h3>1-bit hardware is tiny</h3><p>An 8-bit calculator needed about <strong>294 logic cells</strong>.</p><p>A 1-bit calculator needed about <strong>57</strong>.</p><p>That meant the chip could fit:</p><ul><li><strong>20</strong> 8-bit calculators</li><li><strong>36</strong> 4-bit calculators</li><li><strong>104</strong> 1-bit calculators</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*IWRn_QAsCkkRXRleJXzRVw.png" /></figure><p>So the exact same chip could contain about <strong>five times as many 1-bit compute units</strong>.</p><p>That is a big deal.</p><p>It’s like replacing five refrigerators with five toaster ovens and suddenly discovering your kitchen can hold 25 of them.</p><h3>Then I measured the power</h3><p>I plugged the board into a USB power meter and measured how much electricity each design used.</p><p>With the chip doing the same amount of work:</p><ul><li>1-bit used about <strong>0.15 watts</strong></li><li>4-bit used about <strong>0.20 watts</strong></li><li>8-bit used about <strong>0.30 watts</strong></li></ul><p>So 1-bit did the same work for about <strong>half the power of 8-bit</strong>.</p><p>Then I completely filled the chip with as many calculators as possible.</p><p>The 1-bit design did about <strong>five times as much work as the 8-bit design while using roughly the same total power</strong>.</p><p>That was the moment where I thought:</p><p><strong>Okay. This is real.</strong></p><h3>Ternary was even more interesting</h3><p>Remember ternary?</p><p>+1, 0, or -1.</p><p>The zero matters.</p><p>If a weight is zero, the chip can simply do nothing.</p><p>And doing nothing is extremely energy efficient.</p><p>On my FPGA, ternary math was actually a little cheaper than pure 1-bit math: about <strong>0.13 nJ per operation versus 0.14 nJ</strong>.</p><p>But the bigger surprise came when I tested the actual models.</p><p>At the same 1.7-billion-parameter size:</p><ul><li><strong>4-bit scored 73%</strong></li><li><strong>Ternary scored 64%</strong></li><li><strong>1-bit scored 56%</strong></li></ul><p>So ternary preserved substantially more intelligence than pure 1-bit.</p><p>Then I tried a bigger ternary model.</p><p>A <strong>4B ternary model scored 77%</strong>, while the <strong>1.7B 4-bit model scored 73%</strong>.</p><p>The bigger ternary model used about the <strong>same memory</strong> and roughly the <strong>same weight-math energy</strong>.</p><p>In other words, ternary let me trade cheaper weights for more parameters.</p><p>That sounds fantastic.</p><p>Unfortunately, it also reveals the catch.</p><p>Ternary needed about <strong>2.3× as many parameters</strong> to match the smaller 4-bit model.</p><p>That uses up almost all of its hardware advantage.</p><p>At roughly equal accuracy, the ternary model ended up using only about <strong>6% less weight-math energy</strong>.</p><p>So ternary looks better than pure 1-bit.</p><p>But it is not free magic.</p><h3>Why wasn’t 1-bit four times better than 4-bit?</h3><p>This surprised me.</p><p>One bit is one-quarter as many bits as four bits.</p><p>So I expected roughly one-quarter the energy.</p><p>Instead, it was closer to half.</p><p>Why?</p><p>Because the weight multiplication is only part of the work.</p><p>The chip also has to:</p><ol><li>Read the weight.</li><li>Combine it with the input.</li><li>Add the result to a running total.</li><li>Move signals around the chip.</li></ol><p>Only the first two get dramatically cheaper with 1-bit.</p><p>The rest still costs energy.</p><p>Imagine shipping lighter boxes.</p><p>The package gets lighter.</p><p>Unfortunately, the truck still weighs 20,000 pounds.</p><h3>So does 1-bit AI win?</h3><p>This is where things get annoying.</p><p>The <strong>hardware</strong> wins.</p><p>The <strong>models</strong> don’t always.</p><p>In my own test, taking the same 1.7B model from 4-bit to pure 1-bit dropped its average score from <strong>73% to 56%</strong>.</p><p>That is a very large accuracy penalty.</p><p>Ternary did much better at <strong>64%</strong>, but still lost to 4-bit at the same model size.</p><p>So although a 1-bit operation is much cheaper, you may need a <strong>much larger low-bit model</strong> to get the same intelligence.</p><p>And once you make the model bigger, much of the energy savings disappears.</p><p>Sad trombone.</p><h3>4-bit may be the boring winner</h3><p>One result surprised me even more.</p><p>The 1.7B model scored:</p><ul><li><strong>73% at 4-bit</strong></li><li><strong>72% at 8-bit</strong></li></ul><p>Basically the same.</p><p>But the 4-bit version used about <strong>half the weight-math energy</strong>.</p><p>So if you currently have parts of a model running at 8-bit, moving them to 4-bit may be the easiest power saving available.</p><p>No exotic hardware.</p><p>No giant accuracy sacrifice.</p><p>Just fewer bits.</p><p>Sometimes the boring answer wins.</p><h3>Memory could change everything</h3><p>My FPGA experiment kept the weights directly on the chip.</p><p>Real AI hardware often has to fetch weights from external memory.</p><p>And moving data costs energy.</p><p>A 1-bit weight requires moving:</p><ul><li>half as much data as 2-bit</li><li>one-quarter as much as 4-bit</li><li>one-eighth as much as 8-bit</li></ul><p>So on a system where memory bandwidth dominates power, low-bit models could look substantially better than they did in my experiment.</p><p>That is one of the biggest unanswered questions.</p><h3>The most important result</h3><p>The biggest lesson from this experiment was not:</p><p><strong>“1-bit AI is better.”</strong></p><p>It was:</p><p><strong>“Low-bit AI needs low-bit hardware.”</strong></p><p>On a regular processor, much of the theoretical benefit disappears.</p><p>On hardware specifically built for low-bit math, the advantage becomes much larger.</p><p>The arithmetic really is cheaper.</p><p>The challenge is building models smart enough to take advantage of it.</p><h3>The takeaway</h3><p>My tiny FPGA experiment showed that:</p><ul><li>1-bit compute units were about <strong>5× smaller</strong> than 8-bit ones</li><li>1-bit operations used about <strong>half the energy of 4-bit</strong></li><li>1-bit operations used about <strong>one-fifth the energy of 8-bit</strong></li><li>ternary operations were slightly cheaper still</li><li>ternary models preserved accuracy much better than pure 1-bit</li><li>but larger ternary models consumed most of the savings when matching 4-bit accuracy</li><li>4-bit delivered almost the same accuracy as 8-bit in my tests for roughly half the weight-math energy</li></ul><p>So right now, the picture looks something like this:</p><p><strong>8-bit:</strong> accurate, but expensive.</p><p><strong>4-bit:</strong> surprisingly hard to beat.</p><p><strong>Ternary:</strong> very interesting.</p><p><strong>1-bit:</strong> amazing hardware, but the models still need work.</p><p>If researchers can close the accuracy gap without making the low-bit models much larger, the economics could change dramatically.</p><p>Phones, robots, cars, edge devices, wearables, and tiny AI accelerators could all benefit.</p><p>For now, though, my conclusion is slightly less exciting and probably more useful:</p><p><strong>The fewer bits you use, the cheaper the math gets.</strong></p><p><strong>The hard part is keeping the intelligence.</strong></p><p><strong>If we switch from 4-bit to 1-bit binary, then we cut our power usage in half. However, we will probably need twice as many parameters to make up for the loss in accuracy, which negates any benefit we would get from using 1-bit binary. Ternary offers a maybe ~6% reduction in power at the same accuracy as 4-bit.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed786ee2fc06" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention Is All You Need to Check Whether the Fridge Is Open]]></title>
            <link>https://medium.com/obviously-satire/attention-is-all-you-need-to-check-whether-the-fridge-is-open-f6a65d6428e0?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6a65d6428e0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[satire]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 02:48:43 GMT</pubDate>
            <atom:updated>2026-09-30T02:58:53.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*XMTDjykQ_o08mKxvIIhCvA@2x.jpeg" /></figure><p><strong>Abstract</strong></p><p>We introduce a general framework for replacing simple, reliable algorithms with large language models. Our method preserves approximately the same functionality while dramatically improving GPU utilization, subscription revenue, and the number of people required to maintain a sprinkler.</p><ol><li><strong>LLMReLU</strong></li></ol><p>The traditional ReLU activation function computes max(x, 0). Despite widespread adoption, this approach suffers from a critical limitation: nobody can raise $40 million to do it.</p><p>We introduce LLMReLU, which sends each input to a ten-billion-parameter model with the prompt:</p><p>“Is this number greater than zero? Please think carefully.”</p><p>If the model answers yes, we return the input. If it answers no, we return zero.</p><p>Our model suffers only a 0.2% reduction in accuracy using only 80 billion times as many resources.</p><p>2. <strong>FridgeGPT</strong></p><p>Conventional refrigerators detect an open door using an electrical switch. This hard-coded approach is very last-millennium and needs to be updated.</p><p>FridgeGPT replaces the switch with a camera, hardware capable of 250 trillion operations per second, and a multibillion-parameter model trained to answer:</p><p><em>“Is my fridge open?”</em></p><p>The system checks every three seconds, provided the driver versions agree. During testing, it correctly detected an open fridge in 97% of cases. In the remaining cases, it wrote a poem about cats.</p><p>A major advantage is explainability. The switch merely reports “open,” while FridgeGPT explains that the refrigerator appears open because its door is not closed.</p><p>3. <strong>Sprinkler-of-Thought</strong></p><p>Traditional sprinklers operate on timers, blindly watering grass at scheduled intervals without understanding what grass means.</p><p>Our agent first interviews the lawn, consults seven weather APIs, and generates a watering plan. A second agent critiques the plan. A third agent determines whether the first two agents have sufficient context.</p><p>After fourteen minutes, the system waters the driveway.</p><p>We address this with reinforcement learning from homeowner feedback, collected primarily through screaming.</p><p>4. <strong>Boolean-as-a-Service</strong></p><p>We replace the obsolete <em>if</em> statement with an enterprise inference endpoint.</p><p>Instead of <em>if battery &lt; 10</em>, developers submit:</p><p>“<em>The battery is at 7%. Would you characterize this as below 10%? Return valid JSON.”</em></p><p>This removes brittle programming logic and introduces flexible failure modes, including invalid JSON, rate limits, and a persuasive argument that cheese is actually a fruit.</p><p>5. <strong>Conclusion</strong></p><p>For decades, engineers have asked, “<em>What is the simplest tool that solves this problem?”</em></p><p>We propose a more scalable question:</p><p><em>“Could this require a data center?”</em></p><p>Inspired by Netflix’s paper on its LLM-backed recommendation ranker, GenRec.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6a65d6428e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/obviously-satire/attention-is-all-you-need-to-check-whether-the-fridge-is-open-f6a65d6428e0">Attention Is All You Need to Check Whether the Fridge Is Open</a> was originally published in <a href="https://medium.com/obviously-satire">Obviously Satire</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise RAG: Why Getting AI to Work With Company Knowledge Is Harder Than It Looks]]></title>
            <link>https://medium.com/@ceo_44783/enterprise-rag-why-getting-ai-to-work-with-company-knowledge-is-harder-than-it-looks-fe725e02ad3a?source=rss-29b986a51be------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe725e02ad3a</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Tech Talk with Tyler]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 20:43:27 GMT</pubDate>
            <atom:updated>2026-09-18T20:43:27.792Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoyVd6DbVoNspTZ7p4pRqw.png" /></figure><p>Enterprise RAG sounds easy.</p><p>Your company has a bunch of information. You put it into a vector database. You connect it to an LLM. Employees ask questions. The AI searches the data and gives them answers.</p><p>Done.</p><p>Except that this is approximately what enterprise RAG looks like in tutorials, not what it looks like inside an actual large company.</p><p>The model is often the easy part.</p><p>The hard part is everything else.</p><p>And part of the problem is that a lot of the conventional wisdom around RAG comes from people who have never actually built one that hundreds or thousands of employees depend on.</p><p>I have.</p><p>And one of the biggest lessons I learned is that if you follow every RAG tutorial exactly as written, there is a decent chance you will build a terrible system.</p><h3>Your Company’s Knowledge Is a Disaster</h3><p>When you start working at a large company, everyone seems to assume that you already know everything.</p><p>Every acronym.</p><p>Every database.</p><p>Every product nickname.</p><p>Every internal tool.</p><p>Every historical decision made seven years before you arrived.</p><p>This is basically the <strong>curse of knowledge</strong>: once people know something, they have a surprisingly hard time remembering what it felt like not to know it.</p><p>So the first skill you need when building enterprise AI is not Python.</p><p>It is asking questions.</p><p>Be nice. Be polite. And ask an absurd number of questions.</p><p>Because there is almost never one beautiful database called COMPANY_KNOWLEDGE.</p><p>If there were, your job would be much easier.</p><p>Large companies have often acquired other companies over the years, and every acquisition brings its own technological archaeology.</p><p>You may encounter completely different:</p><ul><li>Database technologies</li><li>Schemas</li><li>Locations</li><li>Authentication systems</li><li>Column names</li><li>Abbreviations</li><li>Internal terminology</li><li>Query syntax</li><li>File repositories</li></ul><p>And somewhere, inevitably, there is a gigantic folder full of PDFs.</p><p>Then there is something even worse:</p><p><strong>tribal knowledge.</strong></p><p>Someone will ask your shiny new AI system:</p><blockquote><em>“What is our most popular product?”</em></blockquote><p>The RAG system cannot answer.</p><p>The user looks at you like your AI is an idiot.</p><p>Then you discover that the answer has never actually been written down anywhere accessible to the system.</p><p>Some guy named Steve just knows it. The user expects your RAG to be able to read Steve’s mind.</p><h3>What Enterprise RAG Actually Looks Like</h3><p>The normal diagram of RAG goes something like this:</p><p><strong>Question → Search → LLM → Answer</strong></p><p>That is useful for explaining the concept.</p><p>It is not how I would build a serious enterprise system.</p><p>A real system looks more like this.</p><p>First, tell users exactly what the system can do.</p><p>This is incredibly important.</p><p>When someone opens the application, you should tell them something like:</p><blockquote><em>“I can answer questions about products, orders, and customers. I can answer one question at a time. For example: ‘What is Product 116?’ I cannot currently answer aggregate questions such as ‘How many products do we have?’”</em></blockquote><p>People need boundaries.</p><p>Otherwise, <strong>they will assume your system is ChatGPT connected telepathically to the entire corporation.</strong></p><p>Then they will ask it about the cleaning chemicals being used on the third-floor windows.</p><p>And when it doesn’t know, they will report that your AI is broken.</p><p>No.</p><p>They asked the wrong system.</p><p>I also like the RAG itself to recognize when a question is outside its capabilities and say so.</p><p>After that, the real workflow begins.</p><p>A user asks:</p><p><strong>“Is order 125 late?”</strong></p><p>A routing model (I usually think of it as triage) determines which system probably contains the answer.</p><p>If the question is about an order, send it to the order-search system.</p><p>If it is about a customer, send it to the customer system.</p><p>If it is about a product, send it to the product system.</p><p>You may effectively have a different search agent for every major data source.</p><p>Then rewrite the user’s question into something optimized for retrieval.</p><p>For example:</p><p><strong>“Is order 125 late?”</strong></p><p>might become:</p><p><strong>“Order 125 status”</strong></p><p>Search for the record.</p><p>Or, if the underlying database is fast enough, query it directly.</p><p>Then, assuming the user is authorized to see the information, send the relevant retrieved data along with the original question and system instructions to the language model.</p><p>Finally, show the answer.</p><p>And then add one of the most important parts of the entire system: feedback.</p><p>👍<br>👎</p><p>Ask why.</p><p>Feedback is not just some nice-to-have, it is the lifeblood of your RAG system. Without feedback, you are flying blind.</p><h3>Most RAG Failures Fall Into Three Buckets</h3><p>People can invent enormous taxonomies of RAG failure modes.</p><p>In practice, I find it useful to think about three broad categories:</p><ol><li>You retrieved the wrong information or no information.</li><li>You retrieved the correct information, but the model answered incorrectly.</li><li>The user is wrong.</li></ol><p>Yes, number three happens.</p><p>A lot.</p><h3>Failure 1: You Didn’t Find the Data</h3><p>Sometimes the information exists, but your search system fails to retrieve it.</p><p>Then you need to find a better search service.</p><p>But very often the information is not even in one of the data sources your application supports.</p><p>Suppose somebody asks:</p><blockquote><em>“What chemical compound are we using to clean the exterior windows?”</em></blockquote><p>Your system might respond:</p><blockquote><em>“I’m sorry, but I currently have access to product and order information. I don’t have access to facilities-maintenance records.”</em></blockquote><p>That is not a failure.</p><p>That is the system correctly understanding its boundaries.</p><p>A RAG system does not have to know everything.</p><p>In fact, trying to make it know everything is one of the easiest ways to make the whole project unmanageable.</p><h3>Failure 2: The AI Has the Right Data and Still Gets It Wrong</h3><p>This is where things get interesting.</p><p>Maybe your prompt is too complicated.</p><p>Maybe it is too long.</p><p>Maybe you retrieved far too much context.</p><p>If you retrieve the top 100 records every time, you are asking the model to find a needle in a haystack you created yourself.</p><p>Sometimes five highly relevant records work better than 100 vaguely relevant ones.</p><p>Another weird enterprise problem is that the AI may answer correctly according to normal human reasoning but incorrectly according to your company’s internal expectations.</p><p>You only discover these things through iteration.</p><p>Someone gives an answer a thumbs down.</p><p>You inspect the logs.</p><p>You adjust the prompt.</p><p>You test again.</p><p>Repeat.</p><p>This is why production feedback matters so much more than arguing endlessly about prompts in a notebook.</p><h3>Failure 3: The User Is Wrong</h3><p>This is one of my favorite stages of a RAG project.</p><p>After enough iterations, the system becomes good enough that the humans start becoming the primary source of errors.</p><p>A user gives the answer a thumbs down:</p><blockquote><em>“This is wrong.”</em></blockquote><p>You trace the answer back through your logs.</p><p>You find the source record.</p><p>Then you say:</p><blockquote><em>“Here is exactly what the database says. The AI’s answer matches it. Is the database incorrect?”</em></blockquote><p>And sometimes the user responds:</p><p>“Oh. I’m so sorry, thank you for your help!”</p><p>That is a beautiful moment.</p><h3>The Demo Is Not the Product</h3><p>A RAG demo is incredibly easy to make.</p><p>Take a few documents. Chunk them. Embed them.</p><p>Throw them into a local vector database.</p><p>Ask a few predictable questions.</p><p>Everybody claps.</p><p>Now add:</p><ul><li>Thousands of users</li><li>Authentication</li><li>Permissions</li><li>Multiple databases</li><li>Constantly changing information</li><li>Concurrent traffic</li><li>Bad user questions</li><li>Latency requirements</li><li>Business consequences</li></ul><p>You are now building something completely different.</p><p>Tools that work wonderfully for a local experiment may not be the tools you eventually use in production.</p><p>That is normal.</p><p>You may have to test several search systems, databases, architectures, and vendors before finding something that performs well at your required scale.</p><p>Do not become emotionally attached to your architecture diagram.</p><p>The goal is not to implement the diagram.</p><p>The goal is to make the thing work.</p><h3>Start With Evals Before You Build Anything</h3><p>One of the most useful things you can do is force the organization to tell you what it actually wants.</p><p>Before I build the system, I want something like 10-20 question-and-answer pairs.</p><p>For example:</p><blockquote><strong><em>Q: What is Product123?</em></strong><em><br>A: Product123 is a fiber laser that shoots dirt.</em></blockquote><p>Or:</p><blockquote><strong><em>Q: Who is Customer12345?</em></strong><em><br>A: Customer12345 is Toys-R-Us, Inc.</em></blockquote><p>These are your evals.</p><p>They are also, in a sense, a contract.</p><p>Your employer is not asking you to build an all-knowing AI.</p><p>They are asking you to build a system that can answer these kinds of questions using these data sources.</p><p>That distinction will save you an enormous amount of pain.</p><p>Because six weeks later someone will inevitably say:</p><blockquote><em>“Why can’t it forecast our sales for next year?”</em></blockquote><p>And you can say:</p><p>Because that wasn’t one of the capabilities we agreed to build. Would you like us to add it?</p><p>That is a dramatically better conversation than:</p><p>“Uh… I thought the AI would just figure everything out.”</p><h3>Humans Still Need to Verify the Output</h3><p>Once employees depend on AI-generated information, RAG stops being an interesting AI demo and becomes part of the business.</p><p>That changes the stakes. If the system cannot find an answer, what happens?</p><p>Who owns the underlying information?</p><p>Who fixes incorrect records?</p><p>Who handles exceptions?</p><p>When should a human verify the answer?</p><p>My answer to the last question is simple: <strong>Humans should verify important outputs.</strong></p><p>RAG makes information easier to find. It does not magically eliminate responsibility.</p><p>And again, build feedback directly into the workflow.</p><p>Thumbs up.</p><p>Thumbs down.</p><p>Reason.</p><p>Those tiny buttons are one of your most valuable sources of information about how the system behaves in the real world.</p><h3>A Lot of RAG Conventional Wisdom Is Wrong</h3><p>This is probably the part that will annoy some people.</p><p><strong>Good.</strong></p><p>The overwhelming majority of RAG tutorials are written by people who have never had to make a RAG system work inside a giant organization.</p><p>That does not mean their tutorials are useless. They are often perfectly good introductions.</p><p>But there is a huge difference between:</p><p><strong>“I made a chatbot answer questions about a PDF.”</strong></p><p>and:</p><p><strong>“Thousands of employees now rely on this system to retrieve company information correctly.”</strong></p><p>Those are different disciplines.</p><p>The problem comes when introductory techniques turn into religious commandments.</p><p>People start believing there is one correct chunk size.</p><p>One correct database.</p><p>One correct retrieval strategy.</p><p>One correct architecture.</p><p>One correct framework.</p><p>Then somebody starts a new job, copies the architecture from a tutorial, discovers that it performs terribly on the company’s actual data, and refuses to try something unconventional because “that’s not how RAG is done.”</p><p>I think that mindset is backwards.</p><p>My systems worked because I was willing to ignore conventional wisdom when the evidence told me conventional wisdom was wrong.</p><p>If the supposedly “best practice” performs worse, stop using it.</p><p>If a weird solution works better, use the weird solution.</p><p>Production does not care about the industry standard solution.</p><p>Production cares whether the answer is correct.</p><h3>RAG Is Becoming Only One Piece of the System</h3><p>The next generation of enterprise AI will not stop at retrieving information.</p><p>The question is changing from:</p><p><strong>Can AI find the right information?</strong></p><p>to:</p><p><strong>Can AI find the right information, reason over it, and safely do something with it?</strong></p><p>That means RAG becomes part of a larger architecture.</p><p>The AI retrieves an order. Determines that the order is late.</p><p>Checks company policy. Looks up the customer’s account.</p><p>Suggests an appropriate response. Maybe drafts the email.</p><p>Maybe eventually performs an action through an approved tool.</p><p>At that point you are no longer just building search.</p><p>You are building AI into the operating fabric of the company.</p><p>And all of the ugly problems that made enterprise RAG difficult (permissions, bad data, outdated information, contradictory systems, unclear ownership, weird internal rules) become even more important.</p><h3>The Real Lesson</h3><p>Getting AI to work with company knowledge is much harder than it looks.</p><p>Not because embeddings are difficult. Not because language models are mysterious. Not because you haven’t discovered the correct magical chunk size.</p><h4>It is hard because companies are messy. Their data is messy. Their software is messy. Their organizational structures are messy. Their employees have information that exists nowhere except inside their heads.</h4><h4>And their expectations of AI are often wildly broader than the system they actually asked you to build.</h4><p>The way through this is not to obsess over having the theoretically perfect RAG architecture.</p><p>Define what the system needs to answer.</p><p>Build evals.</p><p>Connect the smallest number of authoritative data sources required.</p><p>Route questions intelligently.</p><p>Retrieve only what you need.</p><p>Tell users what the system can and cannot do.</p><p>Log everything.</p><p>Collect feedback.</p><p>And then iterate relentlessly based on what actually happens in production.</p><p>That is less glamorous than drawing another RAG architecture diagram.</p><p>But it works.</p><p>And that is ultimately the only part that matters.</p><p><em>I write more about building production RAG systems in my book,</em> <strong>Enterprise RAG: Scaling Retrieval Augmented Generation</strong>, <em>from Manning. </em><a href="https://www.manning.com/books/enterprise-rag">https://www.manning.com/books/enterprise-rag</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe725e02ad3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>