<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by TMD on Medium]]></title>
        <description><![CDATA[Stories by TMD on Medium]]></description>
        <link>https://medium.com/@true.m.medical?source=rss-8dddd66aa9f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vzL5Kz-t5QlbQj54wOYpVQ.png</url>
            <title>Stories by TMD on Medium</title>
            <link>https://medium.com/@true.m.medical?source=rss-8dddd66aa9f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:59:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@true.m.medical/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Real Bottleneck of Local LLMs: It’s Not What You Think]]></title>
            <link>https://medium.com/@true.m.medical/the-real-bottleneck-of-local-llms-its-not-what-you-think-8012a2ca4857?source=rss-8dddd66aa9f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8012a2ca4857</guid>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[apple-silicon]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[TMD]]></dc:creator>
            <pubDate>Thu, 16 Apr 2026 14:48:48 GMT</pubDate>
            <atom:updated>2026-04-16T14:48:48.251Z</atom:updated>
            <content:encoded><![CDATA[<h3>MASA Dev Log #1α — On Speed, Memory Bandwidth, and Hardware Daydreaming</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TNHpC2cqhdxMWB1z" /></figure><h3>Picking Up Where We Left Off</h3><p>Last time, I wrote about building MASA — an integrated local AI system running on a Mac mini M4 with 16GB of RAM.</p><p>The system is actually working pretty well now. Chat responses are solid, the autonomous systems are running, and I’ve beefed up the self-learning module so it studies on its own whenever the system is idle. Development is done almost entirely with Claude Code. Honestly, building something like this solo without AI would be absurd — there’s no reason not to use it. If you’re the type who obsesses over code aesthetics, more power to you, but speed of implementation wins. Get it running first, then customize. Eventually, AI might write cleaner code than any of us anyway.</p><p>I was going to dive into the full system architecture this time, but my brain went off on a tangent. So instead, here’s that tangent.</p><p>Most people know local LLMs exist. But do you have a clear picture of how fast they actually run on consumer hardware like a Mac?</p><h3>LLM Speed: What Actually Matters</h3><p>You’ve probably seen “tokens/s” thrown around. Simple concept: faster means quicker responses and more output per second. But here’s the thing — having lots of memory doesn’t automatically mean you get a fast LLM. Not even close.</p><p><strong>The real bottleneck is memory bandwidth.</strong></p><p>Here’s why. Take gemma4-q5 (5.5GB model size):</p><ul><li>Generating 1 token = reading 5.5GB from memory</li><li>At 100 GB/s bandwidth → 100 ÷ 5.5 ≈ <strong>18 tokens/s</strong> (theoretical max)</li><li>At 120 GB/s bandwidth → 120 ÷ 5.5 ≈ <strong>22 tokens/s</strong> (theoretical max)</li></ul><p>Cache hits and Apple Silicon optimizations add a small boost. Quantization trades quality for speed. But the dominant factor is bandwidth.</p><p>A quick disclaimer: this is back-of-the-envelope math. MoE models behave differently, and I’m sure there are holes in my calculations. This is a late-night ramble, not a research paper. But for rough relative comparisons, it serves its purpose.</p><h3>Where My System Stands</h3><p>On my Mac mini M4 (16GB, ~120 GB/s bandwidth), gemma4-q5 runs at about <strong>25–30 tokens/s</strong>. It’s… a bit sluggish.</p><p>For reference, Claude runs at roughly <strong>80–150 tokens/s</strong> depending on the task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o5ouw00MWqY0SvoR" /></figure><p>Yeah, I want that speed.</p><p>But here’s the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jGkqMkeTGh1zpEMe" /></figure><p>Even with a tiny 5.5GB model like gemma4-q5, hitting Claude-level generation speed on Apple Silicon requires an <strong>M2 Max or higher</strong>. And here’s the kicker — the M4 Max tops out at about 546 GB/s, while the M4 Ultra reaches 800 GB/s. CPU cores barely matter here. Memory bandwidth within the same chip family is essentially fixed.</p><h3>The 70B Dream and Its Reality Check</h3><p>Want to run a 70B model at practical speeds? Let’s do the math.</p><p>A 70B model, say 40GB at full size, Q5-quantized down to ~35GB:</p><ul><li>To hit 80 tokens/s: 35GB × 80 = <strong>2,800 GB/s required</strong></li></ul><p>That’s H100 territory. An NVIDIA H100 80GB. Which currently costs around… $50,000? Maybe more?</p><p>Consumer GPUs like the RTX 4090 have blazing fast VRAM but only 24GB — not enough to load large models. Fast, but too small. That’s the fundamental trade-off in consumer hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_TnS9DcAPnO8zAWf" /></figure><p>Now, MoE (Mixture of Experts) models change the equation significantly. Active parameters are often a fraction of the total — sometimes one-third or less. That means 3–10x speedup in practice. So maybe it’s workable after all. But for dense models, 70B on a Mac is going to feel like wading through mud.</p><h3>A Quick Hardware Comparison</h3><p>For gemma4-q5 (5.5GB), rough theoretical speeds:</p><p>HardwareBandwidthTheoretical tokens/sMac mini M4 (16GB)~120 GB/s~22Mac Studio M4 Max (128GB)~546 GB/s~99RTX 3090 (24GB VRAM)936 GB/s~170RTX 4090 (24GB VRAM)1,008 GB/s~183H100 (80GB)3,350 GB/s~609</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yzNlIUrcR9KcA7Vd" /></figure><p>GPU VRAM is fast. No contest. But Mac’s unified memory has the advantage of capacity — 128GB or 192GB in a single machine, something no consumer GPU can match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4E-FbH1Fjg04szsf" /></figure><h3>A Late-Night Idea</h3><p>While writing this, a thought hit me: the LLM doesn’t have to run on the Mac.</p><p>Mac-specific tasks like Xcode builds and simulator launches stay on the Mac. But the LLM inference? That could run on a separate Linux box. An old RTX 3090 with 24GB of VRAM, running Ollama as a local server over LAN — gemma4’s 26B MoE variant might hit around 270 tokens/s on that setup. Connect it to MASA over the network and boom, the high school kid just got a brain upgrade.</p><p>It’s basically a self-hosted Claude. Not exactly what I originally envisioned, but interesting as an experiment.</p><p>So, how much does a used RTX 3090 cost these days?</p><p>…Wait. They’re HOW much?! Over $1,000?! What happened to cheap used GPUs? My mental model of “pick one up for a few hundred bucks” was apparently years out of date.</p><h3>Back to Reality</h3><p>The idea has potential though. One nice thing about MASA is that everything — the program, models, and learning data — lives on a portable SSD. On startup, a setup checker and mini-installer runs and handles dependencies. Plug the SSD into a different machine and the system adapts to that environment.</p><p>Windows has limitations, of course — no Xcode, no iOS simulator. But the core system runs.</p><p>Still, if I set speed aside, a <strong>Mac Studio M4 Max with 128GB</strong> might be the sweet spot after all. It would let me test the multi-model parallel architectures I originally designed but couldn’t run on 16GB. Speculative decoding, simultaneous agent pipelines, mixed model routing — all suddenly possible.</p><p>Hmm… hmm…</p><p>I stare at the “Buy” button, daydream for a while, then turn back to my Mac mini M4 with its 16GB and give it an appreciative pat.</p><p>You’re doing your best, little guy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8012a2ca4857" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Local LLM System on 16GB of RAM — And Why It Actually Works]]></title>
            <link>https://pub.towardsai.net/how-i-built-a-local-llm-system-on-16gb-of-ram-and-why-it-actually-works-c289ecd2020f?source=rss-8dddd66aa9f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c289ecd2020f</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[apple-silicon]]></category>
            <dc:creator><![CDATA[TMD]]></dc:creator>
            <pubDate>Thu, 16 Apr 2026 14:26:30 GMT</pubDate>
            <atom:updated>2026-04-17T04:59:45.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>A 9-Layer AI System Built With Ollama on a Mac Mini M4. Architecture, Failures, and What Actually Works. How I Built a Local LLM System on 16GB of Ram — And Why It Actually Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*10eteZJJZt-XMDH_" /></figure><h3>1) First of All — Claude Is Incredible</h3><p>Let’s get this out of the way: recent AI, especially Claude, is absurdly powerful.</p><p>App development, multi-agent workflows, web automation, Mac control — we’re approaching the point where Claude can handle most of what a computer does.</p><p>For example, corporate tax filing. I tried handing it off to Claude using MoneyForward Cloud and Zenryoku Hojinzei, and it just… worked. Photos of product tags saved to a folder? End-of-month invoices generated automatically. All I do is review. Journal entries, audits — Claude handles it. I’m honestly starting to wonder if I even need an accountant.</p><p>Same goes for programming. If you have an idea, you can take it to completion through back-and-forth with Claude. App Store rejection? Paste the reason, and Claude drafts both the fix and the English response. Blog auto-posting, SNS automation — all easily built in Python. Forget textbooks. Just ask AI what you want to build, then build it.</p><p>Claude is a miracle worker. But during times of rapid technological change, curiosity takes over.</p><p>If you’re in your 40s or 50s, you probably lived through the early days of PCs and the internet with a sense of wonder. The thrill of tinkering, of making things work yourself. I suspect many of you are feeling that same spark right now.</p><p>So yes, Claude is amazing. But I still wanted to run something on my own machine.</p><p>“Why not just use Claude?” — Fair point. It handles everything. But there are privacy concerns, and even the Max plan has usage limits. And honestly, building a local AI system is mostly about curiosity and self-satisfaction.</p><p>Or so I thought — until I realized: this thing might actually be useful.</p><h3>2) What Is MASA v18.5?</h3><p>It started as an experiment: how much can you do with a Mac mini’s 16GB of RAM?</p><p>I named the system “MASA” — it’s now at version 18.5.</p><p>What I built is an integrated local AI system that combines chat, autonomous coding, web and Mac automation, and a learning system.</p><p>The goal was to replicate what Claude can do — locally. It’s not all the way there yet, but most of it works. There are three main modes: Chat for conversations, Code for programming tasks, and Agent for autonomous browser and file operations. For browser control, I built a custom Chrome extension.</p><p>Past conversations are stored with full history, organized into folders. With auto-sort enabled, the system categorizes chats by content automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2fT5pP6KuWyX1kOX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*feQtgWsD3nzPfjzU" /></figure><h3>Claude vs. MASA: Superman vs. a High School Kid</h3><p>If Claude is a superhuman genius with unlimited capacity who handles everything for you, then MASA v18.5 is a high school part-timer — decent knowledge, decent brains, makes plenty of mistakes.</p><p>My job is to set up the environment, write the manuals, prepare the knowledge sources, and get this kid to do actual work.</p><p>At first I thought, “Is this thing useless?” But gradually, signs of competence started showing up.</p><h3>Fighting with 16GB</h3><p>Running everything on a Mac mini M4 with 16GB means tough choices.</p><p>I tried various models and approaches. Routing between models, running multiple models in parallel — all the ideas from research papers sounded great. But with only 16GB, gathering three kindergartners in a room just creates chaos. Better to have one capable high schooler doing everything.</p><p>The current main model is gemma4-q5. Open-weight LLMs are dominated by Chinese models these days, but I went with Google’s Gemma 4 — it offers the best balance of speed and memory usage at 16GB. The q5 variant lacks vision capabilities though, so I switch to the base gemma4 for Agent mode.</p><p>The beauty of MASA is that the model is swappable. On a 128GB machine, that high school kid becomes near-Superman. Swap in a more powerful model and you get Claude-level performance. The architecture supports it.</p><p>But for now, we fight with 16GB.</p><h3>3) What Can It Do?</h3><p>Here’s what MASA currently offers:</p><p><strong>Chat</strong> — RAG-powered responses with integrated web search and academic paper search. 165 domain-specific knowledge packs. PDF, image, and video analysis. 16 reasoning engines that auto-switch based on question difficulty.</p><p><strong>Autonomous Coding &amp; Agent</strong> — Say “build a calculator” and it goes: spec document, code generation, build, error fix, simulator launch — all automatic. A database of 335 error patterns enables fast fixes without calling the LLM. Existing projects can be modified through natural language. Slide creation, file operations, and other tasks are decomposed and executed autonomously.</p><p><strong>Autonomous Mode</strong> — Continuously monitors system health and auto-repairs on failure. Three autonomy levels: monitor only, suggest fixes, or auto-execute.</p><p><strong>Tool Integration</strong> — Mac file operations, browser control via Chrome extension, automated blog generation and WordPress posting.</p><p><strong>Fully Local</strong> — No cloud APIs. Your data never leaves your machine. No usage limits, no fees (electricity and hardware costs not included…).</p><h3>4) Building an App with MASA’s Autonomous Coding</h3><p>In Code mode, when I say “build a calculator in Xcode,” MASA first generates a spec document. After approval, it proceeds through project setup, coding, building, error correction, and simulator launch — all autonomously.</p><p>Sometimes the build passes on the first try. Other times you need to say “fix the errors.” After that, instructions like “make the buttons blue” or “add this feature” trigger another autonomous cycle of modify, build, and verify.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CfmJjX1H-9DavmbT" /></figure><p>The center panel shows the chat interaction, while the right panel displays a real-time log of what the system is doing.</p><p>What I’ve successfully built so far: a calculator iOS app, an Othello game iOS app, and a Python GUI image resizer — all generated automatically. The system handles Python, Java, Swift, and other languages.</p><p>These are simple apps. But the fact that a local LLM on 16GB can autonomously build working applications — that’s significant.</p><p>For complex code, task decomposition becomes critical, and performance varies significantly with the LLM’s capability. Error handling is supported by learned patterns, but more powerful models undeniably produce better fixes.</p><p>Still, even in this modest environment, basic programming workflows succeeded. Apps were built.</p><h3>Summary So Far</h3><ul><li>Claude is overwhelming. But I still wanted to run AI on my own hardware.</li><li>A reasonably functional local AI system runs on a 16GB Mac mini.</li><li>Autonomous coding successfully generated iOS apps.</li><li>The architecture can scale to Claude-level performance with a better model.</li></ul><p>“Local AI might actually be useful?”</p><h3>What’s Coming in the Paid Series</h3><p>Starting from Part 2, I’ll cover how MASA was actually built — technical details, architecture decisions, and real development stories. I’m planning roughly five installments:</p><p><strong>System Design &amp; Architecture</strong> — 80 files, 60+ API endpoints. 16 reasoning engines and 6 router types. SmartRouter’s feedback learning system.</p><p><strong>Autonomous Coding Internals</strong> — The 3-stage error correction pipeline (pattern DB, static analysis, LLM). Plan-based approach in chat_modify. BuildAnalyzer’s 13-category auto-fix.</p><p><strong>Knowledge Pack System</strong> — Automated pipeline for 165 packs. 7 collector types (Wikipedia, arXiv, PubMed, etc.). Hybrid search (BM25 + Graph RAG).</p><p><strong>Open-Weight LLM Selection</strong> — Comparison table of 14 models by country and performance. Real benchmarks for models that run on 16GB. Why I chose gemma4-q5.</p><p><strong>Setup &amp; Operations</strong> — Ollama + Gemma 4 setup guide. Tuning for 16GB. Pitfalls and solutions.</p><p>Development is ongoing, so content may evolve. If you’re interested, come back and check in.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c289ecd2020f" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-i-built-a-local-llm-system-on-16gb-of-ram-and-why-it-actually-works-c289ecd2020f">How I Built a Local LLM System on 16GB of RAM — And Why It Actually Works</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>