<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saman Chitsazian on Medium]]></title>
        <description><![CDATA[Stories by Saman Chitsazian on Medium]]></description>
        <link>https://medium.com/@samanch70?source=rss-ad7ad221e4d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BM-TodwSVlSKv_2jdXshhw@2x.jpeg</url>
            <title>Stories by Saman Chitsazian on Medium</title>
            <link>https://medium.com/@samanch70?source=rss-ad7ad221e4d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:59:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samanch70/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Harness Engineering: How to Actually Trust an AI Coding Agent]]></title>
            <link>https://medium.com/@samanch70/harness-engineering-how-to-actually-trust-an-ai-coding-agent-4f0343b164f1?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f0343b164f1</guid>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agent-harness]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 05:49:20 GMT</pubDate>
            <atom:updated>2026-04-29T05:49:20.450Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Why the next leap in AI-assisted development isn’t a smarter model — it’s the system you wrap around it.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*gK3g4VWZeLjpLtHVK130KA.png" /></figure><p>If you’ve spent any real time pair-programming with a coding agent, you’ve felt the same friction. The model is impressive in flashes and frustrating in stretches. It writes a clean function, then quietly invents an API that doesn’t exist. It refactors a module elegantly, then breaks an architectural boundary nobody told it about. It passes its own tests. tests it also wrote.</p><p>We treat this as a model problem. “Wait for the next version.” “Try a stronger reasoner.” But after a year of watching teams operationalize these tools, I’m convinced the more interesting bottleneck has shifted somewhere else: into the environment the agent works inside.</p><p>That environment now has a name. People are calling it the harness, and the discipline of designing it well is being called harness engineering. Birgitta Böckeler at Thoughtworks recently published a thoughtful framing of the idea on Martin Fowler’s site, and it captures something a lot of us have been groping toward without a vocabulary. This piece is my attempt to distill that framing, connect it to what’s already happening in practice, and explain why I think it’s the most important conceptual shift in AI-assisted engineering right now.</p><blockquote><strong><em>“Agent = Model + Harness”</em></strong></blockquote><p>The cleanest definition I’ve seen comes from the LangChain folks: an agent is a model plus a harness. The harness is everything that isn’t the LLM — the system prompt, the tool registry, the retrieval layer, the orchestration loop, the guardrails, the post-processing.</p><p>In a coding agent specifically, some of that harness is shipped to you by the vendor. It’s baked into how the agent reads files, runs commands, and structures its turns. But, and this is the part most teams underuse — every coding agent also exposes hooks for you to build an outer harness around it: rules files, custom skills, MCP servers, pre-commit hooks, review subagents, fitness functions, the works.</p><p>Think of it as concentric rings. The model is the core. The vendor’s harness is the middle ring. The harness you build for your codebase is the outer ring. Harness engineering is the practice of designing that outer ring with intent.</p><p>The goal isn’t merely to catch bad output after it happens. A well-designed outer harness does two things at once:</p><p><em>1 — It raises the probability that the agent gets it right on the first try.<br> 2 — It catches and self-corrects what slips through, before a human ever has to look at it.</em></p><p>Get both right and you cut review toil, raise codebase quality, and stop burning tokens on rework loops.</p><h4>Two Directions of Control: Guides and Sensors</h4><p>Böckeler’s framing splits the harness into two complementary halves, and I find this distinction genuinely useful for thinking about which gaps in your setup matter.</p><p><strong>Guides are feedforward controls.</strong> They act before the agent does, shaping behavior up front. Coding conventions in an <strong>AGENTS.md</strong>, a skill that explains how your repo is bootstrapped, an MCP server that exposes your team’s API docs all guides. They reduce the variance of the first attempt.</p><p><strong>Sensors are feedback controls.</strong> They observe after the agent acts, and ideally they speak back to the agent in a form it can act on. A linter that emits a message phrased as an instruction (“Replace this with the <strong><em>Result&lt;T, E&gt;</em></strong> <em>pattern from src/lib/errors.rs</em>”) is doing more work than a linter that just emits an error code. Think of it as constructive prompt injection — using the agent’s own input channel to course-correct it.</p><p>The failure mode of using only one is interesting. Sensor-only setups produce agents that bang their head against the same mistake every run. Guide-only setups produce agents that follow the rulebook but never find out whether the rulebook is actually working. You need both, and you need them in dialogue.</p><h4>Two Execution Modes: Computational vs. Inferential</h4><p>The other axis cuts orthogonally to feedforward/feedback, and it’s where a lot of the practical engineering decisions live.</p><p><strong>Computational controls are deterministic.</strong> CPU-bound. Fast. Type checkers, linters, structural tests, codemods, ArchUnit-style boundary checks. They run in milliseconds and produce reliable, reproducible signals. You can afford to run them on every save.</p><p><strong>Inferential controls are probabilistic.</strong> Usually GPU-bound. Slower and pricier. AI code review, “LLM-as-judge” passes, semantic duplicate detection, agent-driven architecture audits. They handle the things deterministic tools can’t — semantic intent, taste, “does this even make sense” but you pay for that flexibility in latency, cost, and non-determinism.</p><p>The honest engineering answer is almost always both. Computational sensors are cheap enough to run everywhere; inferential sensors are powerful enough to catch what computation can’t see. The art is putting each in the right place in your pipeline.</p><p>A rough mapping that’s served me well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9-OtEsYy8-lS66HQYpEDA.png" /></figure><h4>The Steering Loop</h4><p>Here’s where the human role gets sharper. Your job, increasingly, is not to write every line, it’s to steer the agent by iterating on the harness itself.<br>Whenever the same failure mode recurs, that’s a signal. Not a signal to write a better prompt next time. A signal to upgrade a guide, add a sensor, or rewire the loop. The harness is the durable artifact; the prompt is just one turn.</p><p>And meta-pleasingly, the agent itself is increasingly useful for building its own harness. Use it to scaffold custom linters from observed mistake patterns. Use it to draft architecture fitness tests from a codebase walk. Use it to mine your repo history for conventions that should be made explicit. Custom static analysis used to be too expensive to write for niche rules. It isn’t anymore.</p><h4>Shift Quality Left — Again</h4><p>The continuous-integration crowd has been preaching “shift left” for two decades. The harness-engineering version of that argument is the same shape, but the available tools are different.</p><p>Cheap, fast checks should run as far left as possible, pre-commit, pre-integration, inside the agent’s own self-correction loop. That’s the home of LSPs, fast linters, type checks, and lightweight review agents. Expensive, broader checks belong in the pipeline: mutation testing, full architecture reviews, slower inferential audits.</p><p>Then there’s a category that doesn’t fit neatly into the change lifecycle at all <strong><em>continuous drift sensors</em></strong>. Dead-code detection, dependency hygiene, test-coverage quality scanning, runtime SLO monitoring with agents that propose fixes when latency degrades. These run against the codebase, not against a specific change. They catch the slow rot that no PR-level check ever sees.</p><p>If you map your current setup against this, the gaps usually pop out fast. Most teams I talk to are heavy on pre-commit linting, light on inferential review, and almost completely missing on continuous drift detection.</p><h4>Three Dimensions of What You’re Regulating</h4><p>Böckeler proposes three useful categories for what a harness is regulating, and the harnessability of each varies wildly.</p><p><strong>1. The Maintainability Harness</strong></p><p>This is where almost all the existing tooling lives — code quality, complexity, duplication, style, test coverage, architectural boundaries. Computational sensors handle the structural stuff well: cyclomatic complexity, dependency direction, dead code. Inferential sensors can extend into semantic territory: detecting that two functions are duplicates even when they don’t look alike, or that a “fix” is actually a brute-force workaround.</p><p>What neither catches reliably is the high-impact, judgment-heavy stuff: misdiagnosis of a problem, over-engineering, scope creep, misread instructions. If the human didn’t specify the problem clearly, no sensor downstream will save the agent from solving the wrong thing.</p><p><strong>2. The Architecture Fitness Harness</strong></p><p>This is fitness functions in their original sense — guides and sensors that encode and enforce architectural characteristics. Performance budgets that the agent is told about and tested against. Logging conventions plus a debugging skill that asks the agent to reflect on whether the logs it had were good enough. Observability standards that get verified, not just documented. Architecture fitness has been a niche practice for a long time. Coding agents make it suddenly economical to build out, because the cost of writing custom checks has collapsed.</p><p><strong>3. The Behaviour Harness</strong></p><p>This is the open problem, and it’s worth being honest about. How do you guide and verify that the application functionally does what it should? The current default — “feed in a spec, check that the AI-written tests pass, supplement with manual QA” — puts an uncomfortable amount of weight on a test suite the agent wrote itself. Coverage metrics don’t help much when the tests are testing the wrong thing.</p><p>There’s interesting work happening here — the “approved fixtures” pattern, characterization tests, golden-master techniques — but nothing yet that resembles a clean, general answer. If you’re operating coding agents at high autonomy in production, this is where I’d focus the next year of investment.</p><h4><strong>Not Every Codebase Is Equally Harnessable</strong></h4><p>A subtle point that doesn’t get enough airtime: some codebases are simply easier to wrap a harness around than others.</p><p>A strongly-typed language gives you a free continuous sensor. Clear module boundaries make architectural rules expressible. Convention-heavy frameworks (Spring, Rails, NestJS) absorb decisions the agent would otherwise have to make and probably get wrong. Ned Letcher (also at Thoughtworks) calls these properties ambient affordances, structural features of the environment that make it legible and tractable to an agent.</p><p><strong>The greenfield/legacy split here is brutal:</strong></p><p><strong>Greenfield teams</strong> can choose harnessability from day one. Picking Rust over a dynamic language, picking a framework with strong conventions, drawing crisp module boundaries — these are now also decisions about how governable your codebase will be by AI agents.</p><p><strong>Legacy teams</strong> face the inverse problem. The codebases that need a harness most — sprawling, convention-light, debt-heavy systems — are precisely the ones where building a harness is hardest. Type information is missing. Boundaries are nominal. Conventions are implicit and contradictory.</p><p>If you’re working in legacy territory, my read is that the first investment is often making the codebase more harnessable, not harnessing it as-is.</p><h4>Toward Harness Templates</h4><p>Here’s a forward-looking idea I find genuinely exciting. Most engineering organizations have a small number of recurring service topologies — the CRUD-with-API service, the event-processing worker, the dashboard frontend, the data pipeline, that cover most of what they build. Many already codify these as service templates or scaffolding tools.</p><p>The next evolution might be harness templates: a topology bundled with the guides and sensors that constrain an agent to its conventions. Pick “JVM CRUD service” and you get the Spring scaffold plus the AGENTS.md, plus the architecture tests, plus the review skills, plus the runtime SLO monitors — all wired up and version-controlled.</p><p>There’s a cybernetics argument hiding underneath this idea. Ashby’s Law of Requisite Variety says a regulator must have at least as much variety as the system it regulates. An LLM can produce nearly anything, which makes building a comprehensive regulator for “all possible code” a losing game. Committing to a topology is a variety reduction move, narrow what the agent can do, and a complete harness becomes achievable. Constraint, paradoxically, is what makes higher autonomy safe.</p><p>The pitfalls are familiar from service templates: drift between the template and the instances, version skew, contribution flow. With non-deterministic inferential controls in the mix, those problems may even get worse before they get better. But the prize is real and I’d bet on this being a pattern that mature platform teams ship within the next 12–18 months.</p><h4>What the Human Still Brings</h4><p>It’s worth being clear-eyed about what isn’t in the harness, no matter how well you build it.</p><p>Human engineers carry an enormous amount of implicit context that doesn’t fit into a rules file. Aesthetic disgust at a 300-line function. The instinct that “we don’t do it that way here.” Awareness of which technical debt is tolerated for business reasons and which is a ticking bomb. Knowledge of what the team is actually trying to achieve this quarter. Social accountability your name is on the commit. The slow human pace itself, which creates the cognitive space for that experience to fire.</p><p>Coding agents have none of this. A harness is an attempt to externalize some of it into explicit guides and sensors. But the goal isn’t to eliminate human judgment it’s to direct human judgment to where it actually matters most, instead of spending it on style nits and convention reminders that machines can enforce.</p><p>That reframing changes what a senior engineer’s job looks like. Less line-by-line review. More designing the system that does the line-by-line review. More choosing what not to encode. More noticing when the harness itself is drifting out of sync with what the team actually values.</p><p><strong>Open Questions ?</strong></p><p>I’ll close where Böckeler does with the questions that don’t have clean answers yet, because I think they’re the interesting ones.</p><p><strong>Coverage of the harness itself. </strong>We have code coverage and mutation testing for tests. What’s the equivalent for harness completeness? When sensors stop firing, is that a sign of high quality or of detection gone stale?</p><p><strong>Internal consistency. </strong>As the harness grows, how do guides and sensors stay coherent? What stops a rule in one skill from quietly contradicting a check in another?</p><p><strong>Trade-off arbitration.</strong> When feedforward instructions and feedback signals point in different directions, how much do we trust the agent to make the right call? How much should we?</p><p><strong>Tooling for the harness as a system. </strong>Right now controls are scattered across pre-commit hooks, CI configs, agent rules files, MCP servers, and skills directories. There’s clearly a layer missing that lets you reason about all of it as one thing.</p><p>The takeaway, for me, is that this isn’t a one-time setup. Building and maintaining the outer harness is becoming an ongoing engineering practice in its own right — somewhere between platform engineering, developer experience, and quality engineering, with elements of each.<br>The teams who’ll get the most leverage from coding agents over the next two years aren’t going to be the ones with the best prompts. They’ll be the ones who built the best harnesses.</p><p>This article is a synthesis and personal take on Birgitta Böckeler’s “<a href="https://martinfowler.com/articles/harness-engineering.html">Harness engineering for coding agent users</a>” (Martin Fowler, April 2026), which introduced the framing I’ve leaned on throughout. Any oversimplifications or hot takes are mine; the underlying framework is hers and the Thoughtworks team’s. Worth reading the original if any of this resonated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f0343b164f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brain in a Jar, No More: How to Give Your Local LLM Superpowers with Tools]]></title>
            <link>https://medium.com/@samanch70/brain-in-a-jar-no-more-how-to-give-your-local-llm-superpowers-with-tools-42ba95c2e39f?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/42ba95c2e39f</guid>
            <category><![CDATA[local-gpt]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 08:48:52 GMT</pubDate>
            <atom:updated>2025-07-27T08:48:52.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>Your offline AI is smart, but it’s disconnected. Let’s give it the power to interact with the real world — all while keeping your data 100% private.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KyWAa8nust9NP34lwb_N_A.png" /></figure><p>You’ve done it. You’ve set up a powerful Large Language Model like Llama 3 or Mistral to run right on your own computer. There’s a certain magic to it, isn’t there? A private, uncensored intelligence that’s all yours.</p><p>But you quickly hit a wall.</p><p>Ask it about today’s weather, and it apologizes for not having real-time data. Ask it to organize your files, and it can only give you the code to do it yourself. Your powerful AI is a brilliant brain in a jar — all thought, no action.</p><p>What if you could give it hands? What if you could connect it to the world, to your apps, and to your data, all without sacrificing an ounce of privacy? You can. The key is <strong>tool use</strong>.</p><h3>So, What’s the Big Deal with “Tools”?</h3><p>Think of tools as specific superpowers you grant your LLM. Instead of being a passive conversationalist, it becomes an active agent. This isn’t just a minor upgrade; it’s a fundamental transformation.</p><p>Here’s what it unlocks:</p><ul><li>🌐 <strong>Real-Time Knowledge:</strong> Your LLM’s knowledge is frozen in time. A web_search tool lets it break free, fetching current news, stock prices, or the score of last night&#39;s game.</li><li>💻 <strong>Local Environment Control:</strong> This is the game-changer for a <em>local</em> LLM. Imagine telling your AI, “Find every PDF in my Documents folder that mentions ‘Project Phoenix’ and move them to a new folder.” A tool can execute that shell command for you.</li><li>🧮 <strong>Flawless Execution:</strong> LLMs are creative wordsmiths, but they can struggle with precise math or complex logic. By giving it a calculator or code_interpreter tool, it can offload the task to a system that <em>never</em> makes a mistake.</li><li>🔒 <strong>Unbreakable Privacy:</strong> This is the crown jewel. You can build a tool that reads your local calendar or summarizes your private notes. Because both the model and the tool are running on your machine, your sensitive data never travels across the internet to a third-party server.</li></ul><h3>The Magic Trick: How It Actually Works</h3><p>It sounds like you’re giving a ghost control of your machine, but the process is far safer and more elegant than that. The LLM never actually <em>runs</em> the code. It acts as a brilliant dispatcher, figuring out the right tool for the job.</p><p>It’s a simple loop:</p><ol><li><strong>The Prompt:</strong> You ask a question that requires action. <em>“What’s the weather like in Tehran right now?”</em></li><li><strong>The Decision:</strong> The LLM, which has been shown a menu of its available tools, analyzes your request. It recognizes that this requires the get_weather function and identifies &quot;Tehran&quot; as the necessary input.</li><li><strong>The Instruction:</strong> The model doesn’t fetch the weather. Instead, it outputs a clean, simple piece of JSON, which is just a structured instruction.</li><li>{ “tool_to_call”: “get_weather”, “parameters”: { “city”: “Tehran” } }</li><li><strong>The Action:</strong> The application running your LLM (like Ollama or a custom Python script) sees this instruction. It says, “Ah, I know what to do!” It then runs the <em>actual</em> get_weather function on your machine, which calls a weather API and gets the result: &quot;28°C and sunny&quot;.</li><li><strong>The Final Word:</strong> The result is fed back into the LLM. Now armed with the missing piece of information, the model generates the friendly, natural language response you were waiting for.</li></ol><blockquote><strong><em>“The current weather in Tehran is 28°C and sunny.”</em></strong></blockquote><h3>Your Toolkit for Building</h3><p>Ready to get your hands dirty? The community has built incredible frameworks to make this easy.</p><ul><li><strong>Ollama:</strong> Quickly becoming the go-to for running models locally. Its recent updates include increasingly sophisticated native support for tool calling, simplifying the entire process.</li><li><strong>LangChain / LlamaIndex:</strong> These are the heavy-duty, high-level frameworks. They provide a rich ecosystem of pre-built tools and “agents” that can chain multiple tool calls together to solve complex problems.</li><li><strong>Llama.cpp:</strong> For the power user who wants ultimate control. Using grammar-based sampling (like GBNF), you can force the model to output perfectly-formed JSON every single time, ensuring your tools are called reliably.</li></ul><h3>A Quick Word of Warning: With Great Power…</h3><p>You know the rest. Granting an AI the ability to interact with your system is serious business. Never, ever give it an open-ended tool that can execute arbitrary code or delete files without strict guardrails.</p><blockquote><strong><em>Always sandbox your tools. Start with read-only functions. Vet every tool you add. The goal is a capable assistant, not a rogue agent.</em></strong></blockquote><h3>The Dawn of the Truly Personal Agent</h3><p>By embedding tools, we are witnessing the most exciting evolution in local AI. We are moving beyond simple chatbots and are starting to build truly personal, context-aware agents that can understand <em>and</em> act on our behalf.</p><p>Your brain in a jar is finally getting its hands. The only question left is: what will you have it do first?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42ba95c2e39f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Goodbye, VRAM Limits: How to Run Massive LLMs Across Your GPUs ]]></title>
            <link>https://medium.com/@samanch70/goodbye-vram-limits-how-to-run-massive-llms-across-your-gpus-b2636f6ae6cf?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2636f6ae6cf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[deepl]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 15:43:53 GMT</pubDate>
            <atom:updated>2025-07-16T15:45:05.538Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stop offloading to your CPU. It’s time to unleash the full parallel power of your rig for massive local LLMs.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XHSDBJIlabphE7exeAp_2Q.png" /></figure><p>The pace of AI is relentless. We’ve barely finished exploring the capabilities of Llama 3, and already the community is buzzing about what comes next. Imagine a model like “Llama 4: Maverick” — a hypothetical, ultra-capable model that pushes the VRAM limits of even the beefiest consumer GPUs.</p><p>Running a 70B model on a single 24GB card is already a stretch. So how can a developer or enthusiast hope to run a future 100B+ parameter model locally? The answer isn’t necessarily a single, prohibitively expensive NVIDIA H100. The answer is likely already in your machine: multiple GPUs.</p><p>Thanks to the elegant simplicity of <strong>Ollama</strong>, harnessing the combined power of two, three, or even four GPUs to run a single, massive model is easier than you think. Let’s explore how to achieve true model parallelism.</p><h3>Why Ollama is a Game-Changer</h3><p>If you’re not yet familiar with it, <a href="https://ollama.com/">Ollama</a> is a powerful tool that streamlines running open-source LLMs locally. It packages everything you need — model weights, configuration, and a server — into a simple, all-in-one experience. What used to be a complex process of wrangling Python dependencies and CUDA versions now boils down to a single command:</p><pre>ollama run mistral</pre><p>Its true power, however, lies in its flexible and straightforward configuration.</p><h3>The Core Concept: Splitting the Load with a Modelfile</h3><p>By default, when Ollama loads a model, it tries to fit all the model’s layers onto a single GPU. If the model is too big, it places as many layers as it can on gpu0 and then offloads the rest to your system&#39;s RAM, relying on the CPU to process them. This CPU offloading is a major bottleneck and leads to painfully slow inference speeds.</p><p>The solution is to tell Ollama to explicitly distribute the layers across <em>all</em> available GPUs. This is achieved using a custom configuration file called a Modelfile.</p><p>The Modelfile is a plain text recipe that tells Ollama how to build or modify a model. To enable multi-GPU support, we only need two key instructions:</p><ol><li>FROM: This specifies the base model we want to use from the Ollama library.</li><li>PARAMETER num_gpu: This is the magic instruction. It tells Ollama how many layers of the model to offload to the GPU(s).</li></ol><p>You might think you need to calculate the perfect number of layers per GPU, but Ollama is smarter than that. By setting num_gpu to a very high number (e.g., 999), you essentially tell Ollama: &quot;Use all the GPU power you can find. Distribute the layers across all available GPUs as efficiently as possible.&quot;</p><p>Here’s what the Modelfile looks like to create a multi-GPU version of a hypothetical llama4:maverick model.</p><pre># Modelfile to run llama4:maverick on all available GPUs<br><br># Start from the base model<br>FROM llama4:maverick<br><br># Set the number of GPU layers to a high value<br># Ollama will automatically split these layers across all detected GPUs.<br>PARAMETER num_gpu 999</pre><p>Ollama will then handle the complex task of splitting the model’s tensor layers evenly. With two identical GPUs, it will put roughly 50% of the layers on the first GPU and 50% on the second.</p><h3>Practical Walkthrough: Building and Running Your Multi-GPU Model</h3><p>Let’s turn theory into practice. Assume you have a model named llama4:maverick available in your Ollama library.</p><h4>Step 1: Create the Modelfile</h4><p>In an empty directory, create a new file named Modelfile (no file extension) and add the two lines of configuration from the example above.</p><h4>Step 2: Create the New Model Version</h4><p>Open your terminal in the same directory as your Modelfile. Use the ollama create command to build a new, multi-GPU-enabled version of the model. We&#39;ll give it a descriptive name like llama4:maverick-multi.</p><pre>ollama create llama4:maverick-multi -f ./Modelfile</pre><p>Ollama will instantly create a new model entry that inherits from llama4:maverick but applies your new PARAMETER setting.</p><h4>Step 3: Run the Parallelized Model</h4><p>You can now run your powerful new model with the standard run command.</p><pre>ollama run llama4:maverick-multi</pre><p>When Ollama starts, it will detect all your GPUs and begin loading the model layers, distributing them across each card’s VRAM.</p><h3>Verification: Is It Actually Working?</h3><p>Don’t just take our word for it — verify it! While the model is loading or running, open another terminal and use your system’s GPU monitoring tool.</p><ul><li><strong>For NVIDIA GPUs:</strong> watch -n 1 nvidia-smi</li><li><strong>For AMD GPUs:</strong> watch -n 1 rocm-smi</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*XmVWyzj0uNKzz8C0gNzmWw.png" /></figure><p>The watch command will refresh the stats every second. You should see two key indicators:</p><ul><li><strong>VRAM Usage:</strong> Both (or all) of your GPUs will show significant VRAM being consumed. This is the model being loaded onto each card.</li><li><strong>GPU Utilization:</strong> As you send prompts to the model, you should see the GPU-Util percentage spike on all cards simultaneously.</li></ul><p>If you see activity across multiple GPUs, you’ve done it! You are now running a single, massive LLM in parallel. Congratulations, you’ve unlocked a new level of local AI capability. 🚀</p><p>Sources:</p><ul><li>Ollama Team. (2025). <em>Ollama</em>. Retrieved July 16, 2025, from <a href="https://ollama.com/">https://ollama.com/</a></li><li>Ollama Team. (2025). <em>ollama/ollama</em>. GitHub. Retrieved July 16, 2025, from <a href="https://github.com/ollama/ollama">https://github.com/ollama/ollama</a></li><li>Ollama Team. (2025). <em>Ollama Discord Server</em>. Discord. Retrieved July 16, 2025, from <a href="https://www.google.com/search?q=https://discord.gg/ollama">https://discord.gg/ollama</a></li><li>NVIDIA Corporation. (2025). <em>NVIDIA System Management Interface (nvidia-smi)</em>. NVIDIA Developer. Retrieved July 16, 2025, from <a href="https://developer.nvidia.com/nvidia-system-management-interface">https://developer.nvidia.com/nvidia-system-management-interface</a></li><li>Advanced Micro Devices, Inc. (2025). <em>ROCm System Management Interface (rocm-smi)</em>. AMD ROCm™ Documentation. Retrieved July 16, 2025, from <a href="https://www.google.com/search?q=https://rocm.docs.amd.com/en/latest/tools/rocm-smi.html">https://rocm.docs.amd.com/en/latest/tools/rocm-smi.html</a></li><li>Hugging Face, Inc. (2025). <em>Hugging Face Model Hub</em>. Retrieved July 16, 2025, from <a href="https://huggingface.co/models">https://huggingface.co/models</a></li><li>r/LocalLLaMA Community. (2025). <em>LocalLLaMA Subreddit</em>. Reddit. Retrieved July 16, 2025, from <a href="https://www.reddit.com/r/LocalLLaMA/">https://www.reddit.com/r/LocalLLaMA/</a></li><li>Gerganov, G., et al. (2023). <em>GGUF File Format Documentation</em>. GGML Repository on GitHub. Retrieved July 16, 2025, from <a href="https://www.google.com/search?q=https://github.com/ggerganov/ggml/blob/master/docs/gguf.md">https://github.com/ggerganov/ggml/blob/master/docs/gguf.md</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2636f6ae6cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[⚡️Unleash the Beast: How vLLM Delivers 24x Faster LLM Inference]]></title>
            <link>https://medium.com/@samanch70/%EF%B8%8Funleash-the-beast-how-vllm-delivers-24x-faster-llm-inference-78a32302e9b2?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/78a32302e9b2</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 20:09:55 GMT</pubDate>
            <atom:updated>2025-07-14T20:26:26.437Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>A deep dive into the high-throughput serving engine that’s revolutionizing LLM deployment, complete with a hands-on tutorial to get you started.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_gCKHmFSJnutUbwjycGNA.png" /><figcaption>vLLM</figcaption></figure><p>· <a href="#cd7d">The Secret Sauce: What Makes vLLM So Fast?</a><br>· <a href="#2f40">PagedAttention: A Smarter Way to Manage Memory</a><br>· <a href="#3ef4">Continuous Batching: Keeping the GPU Busy</a><br>· <a href="#4ce5">Getting Your Hands Dirty: A vLLM Tutorial</a><br>· <a href="#ae16">vLLM in the Real World: Use Cases and Impact</a><br>· <a href="#d3fc">vLLM vs. The Competition</a><br>· <a href="#ad20">The Road Ahead for vLLM</a><br>· <a href="#75bd">Conclusion</a><br>· <a href="#9a61">Resources</a></p><p>The world of large language models (LLMs) is expanding at an explosive pace. As these models become more powerful and integrated into our daily applications, the efficiency of their deployment becomes a critical bottleneck. Enter vLLM, a groundbreaking open-source library designed to supercharge LLM inference and serving. With its innovative techniques, vLLM boasts significantly higher throughput and more efficient memory management compared to traditional solutions, making it a game-changer for developers and researchers alike.</p><p>This article will guide you through the intricacies of vLLM, from its core concepts to a practical, step-by-step tutorial. We’ll explore why vLLM is rapidly becoming the go-to solution for deploying LLMs in production environments and provide you with the resources to start your own high-performance inference journey.</p><h3>The Secret Sauce: What Makes vLLM So Fast?</h3><p>At the heart of vLLM’s remarkable performance are two key innovations: <strong>PagedAttention and continuous batching</strong></p><h3><em>PagedAttention: A Smarter Way to Manage Memory</em></h3><p>Traditional LLM serving systems often struggle with inefficient memory management for the key-value (KV) cache, a crucial component in the attention mechanism of Transformer models. The KV cache stores the attention information for previously generated tokens, and its size can grow very large, especially for long sequences.</p><p>Inspired by the concept of virtual memory and paging in operating systems, vLLM introduces <strong>PagedAttention</strong>. Instead of allocating a contiguous block of memory for the entire KV cache, PagedAttention partitions it into smaller, non-contiguous blocks, or “pages.” This allows for more flexible and efficient memory allocation, significantly reducing internal fragmentation and wasted memory. The result is the ability to serve much larger batches of requests and longer sequences on the same hardware.</p><h3><strong>Continuous Batching: Keeping the GPU Busy</strong></h3><p>In a typical LLM inference scenario, requests arrive asynchronously with varying prompt lengths. Traditional static batching methods wait for a full batch of requests to be ready before processing them, leading to idle GPU time and increased latency.</p><p>vLLM’s <strong>continuous batching</strong> mechanism addresses this by processing requests as they arrive. It dynamically creates batches on the fly, adding new requests to the batch as soon as there is available capacity. When a sequence in the batch finishes generating, it is immediately replaced with a new one. This ensures that the GPU is constantly utilized, leading to a substantial increase in throughput.</p><h3>Getting Your Hands Dirty: A vLLM Tutorial</h3><p>Now, let’s move from theory to practice. This tutorial will walk you through installing vLLM and using it to run inference on a pre-trained language model.</p><p><strong>Prerequisites:</strong></p><ul><li>A Linux system with a compatible NVIDIA GPU (Compute Capability 7.0 or higher).</li><li>Python 3.8 or newer.</li><li>pip installed.</li></ul><p><strong>Step 1: Installation</strong></p><p>It is highly recommended to use a virtual environment to avoid conflicts with other Python packages.</p><pre>python3 -m venv vllm-env<br>source vllm-env/bin/activate</pre><p>Now, you can install vLLM using pip:</p><pre>pip install vllm</pre><p>vLLM comes with pre-compiled CUDA kernels. If you have a different CUDA version, you might need to build from source. Please refer to the official vLLM documentation for detailed instructions.</p><p><strong>Step 2: Running Offline Inference</strong></p><p>With vLLM installed, you can easily run inference on a Hugging Face model. Let’s try it with a smaller, yet powerful model like EleutherAI/gpt-neo-1.3B.</p><p>Create a Python script named test_vllm.py and add the following code:</p><pre>from vllm import LLM, SamplingParams<br><br># A list of prompts to process<br>prompts = [<br>    &quot;The capital of France is&quot;,<br>    &quot;The future of AI is&quot;,<br>    &quot;Write a short story about a robot who discovers music.&quot;,<br>]<br><br># Initialize the LLM with the desired model<br>llm = LLM(model=&quot;EleutherAI/gpt-neo-1.3B&quot;)<br><br># Define the sampling parameters for generation<br>sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=100)<br><br># Generate text for the prompts<br>outputs = llm.generate(prompts, sampling_params)<br><br># Print the generated outputs<br>for output in outputs:<br>    prompt = output.prompt<br>    generated_text = output.outputs[0].text<br>    print(f&quot;Prompt: {prompt!r}, Generated text: {generated_text!r}&quot;)</pre><p>Now, run the script from your terminal:</p><pre>python test_vllm.py</pre><p>You should see the model generate completions for each of the prompts.</p><p><strong>Step 3: Setting Up an OpenAI-Compatible Server</strong></p><p>One of vLLM’s most powerful features is its ability to serve LLMs through an OpenAI-compatible API server. This allows you to easily integrate vLLM with existing tools and applications that are designed to work with the OpenAI API.</p><p>To start the server, run the following command in your terminal:</p><pre>python -m vllm.entrypoints.openai.api_server --model &quot;EleutherAI/gpt-neo-1.3B&quot;</pre><p>This will start a web server on http://localhost:8000. You can now send requests to this server using tools like curl or any programming language with an HTTP client.</p><p>Here’s an example of how to send a completion request using curl:</p><pre>curl http://localhost:8000/v1/completions \<br>    -H &quot;Content-Type: application/json&quot; \<br>    -d &#39;{<br>        &quot;model&quot;: &quot;EleutherAI/gpt-neo-1.3B&quot;,<br>        &quot;prompt&quot;: &quot;The best way to learn a new programming language is&quot;,<br>        &quot;max_tokens&quot;: 50,<br>        &quot;temperature&quot;: 0.7<br>    }&#39;</pre><p>You will receive a JSON response containing the generated text, similar to the OpenAI API.</p><h3>vLLM in the Real World: Use Cases and Impact</h3><p>The high-throughput and efficiency of vLLM have made it a popular choice for a wide range of applications, including:</p><ul><li><strong>Chatbots and Conversational AI:</strong> vLLM can handle a large number of concurrent users with low latency, providing a smooth and responsive user experience.</li><li><strong>Content Generation:</strong> From writing articles and marketing copy to generating creative text formats, vLLM can significantly accelerate content creation pipelines.</li><li><strong>Code Generation and Assistance:</strong> Developers can leverage vLLM to power intelligent code completion, bug detection, and a variety of other coding-related tasks.</li><li><strong>Research and Development:</strong> Researchers can use vLLM to speed up their experiments and iterate faster on new model architectures and applications.</li></ul><h3>vLLM vs. The Competition</h3><p>While other libraries like Hugging Face’s Text Generation Inference (TGI) also aim to optimize LLM serving, vLLM often demonstrates superior performance in terms of throughput, especially under high concurrency. The choice between vLLM and other solutions will depend on the specific use case, hardware, and desired level of customization. However, for many demanding production workloads, vLLM’s PagedAttention and continuous batching provide a distinct advantage.</p><h3>The Road Ahead for vLLM</h3><p>The development of vLLM is ongoing, with a vibrant open-source community constantly contributing to its improvement. Future developments are likely to include support for more model architectures, further performance optimizations, and enhanced features for distributed inference and model quantization.</p><h3>Conclusion</h3><p>vLLM has emerged as a powerful and essential tool for anyone looking to deploy large language models efficiently and at scale. Its innovative memory management and batching techniques address some of the most significant challenges in LLM serving, paving the way for a new generation of high-performance AI applications. By following the tutorial in this article, you’ve taken your first step into the world of high-throughput LLM inference. The resources below will help you continue your journey and unlock the full potential of vLLM.</p><h3>Resources</h3><ul><li><em>vLLM GitHub Repository: </em><a href="https://github.com/vllm-project/vllm"><em>https://github.com/vllm-project/vllm</em></a></li><li><em>vLLM Documentation: </em><a href="https://www.google.com/search?q=https://vllm.readthedocs.io/en/latest/"><em>https://vllm.readthedocs.io/en/latest/</em></a></li><li><em>PagedAttention Paper: </em><a href="https://arxiv.org/abs/2309.06180"><em>https://arxiv.org/abs/2309.06180</em></a></li><li><em>vLLM Blog: </em><a href="https://www.google.com/search?q=https://vllm.ai/blog/"><em>https://vllm.ai/blog/</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78a32302e9b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Efficiency in Large Language Models: A Look at LoRA]]></title>
            <link>https://medium.com/@samanch70/unlocking-efficiency-in-large-language-models-a-look-at-lora-29016a2771f8?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/29016a2771f8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[low-rank-adaptation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Sat, 12 Jul 2025 08:05:13 GMT</pubDate>
            <atom:updated>2025-07-12T08:05:13.924Z</atom:updated>
            <content:encoded><![CDATA[<h4>Large Language Models (LLMs) have revolutionized AI, powering everything from advanced chatbots to sophisticated content generation. However, fine-tuning these colossal models for specific tasks is often a resource-intensive endeavor, demanding significant computational power and vast datasets. This is where <strong>LoRA (Low-Rank Adaptation)</strong> steps in, offering an ingenious solution to make LLM adaptation more efficient and accessible.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s04F7AlT4ivUi5y-pEo0rQ.png" /><figcaption>LoRA</figcaption></figure><p>· <a href="#712f">The Challenge of Fine-Tuning LLMs</a><br>· <a href="#e21b">Enter LoRA: A Low-Rank Solution</a><br>· <a href="#2b6b">Why does this work? </a><br>· <a href="#f3e4">The Benefits of LoRA</a><br>· <a href="#b157">LoRA in Practice</a><br>· <a href="#e1b5">The Future of LLM Adaptation</a></p><h3>The Challenge of Fine-Tuning LLMs</h3><p>Traditional fine-tuning involves adjusting all, or a large portion, of an LLM’s millions (or even billions) of parameters. Imagine trying to subtly alter the entire architecture of a skyscraper just to change the color of a few windows — it’s overkill and incredibly costly. Similarly, full fine-tuning of an LLM leads to several challenges:</p><ul><li><strong>Computational Expense:</strong> Training billions of parameters requires powerful GPUs and substantial energy.</li><li><strong>Storage Requirements:</strong> Each fine-tuned version of an LLM creates a complete copy of the model, leading to massive storage overhead.</li><li><strong>Catastrophic Forgetting:</strong> When fine-tuning on new data, LLMs can sometimes “forget” previously learned information, degrading performance on general tasks.</li></ul><h3>Enter LoRA: A Low-Rank Solution</h3><p>LoRA addresses these issues by proposing a more surgical approach to adaptation. Instead of modifying all existing weights in the pre-trained model, LoRA injects small, trainable matrices into each layer of the LLM.</p><p>Here’s the core idea:</p><p>Imagine a large weight matrix W0​ in a neural network layer. When fine-tuning, we want to update this matrix by a small amount, ΔW. LoRA proposes that this update matrix ΔW can be approximated by the product of two much smaller matrices, A and B, i.e., ΔW=BA.</p><h3><strong><em>Why does this work?</em></strong></h3><p>This is based on the principle of <strong>low-rank approximation</strong>. Many complex transformations can be effectively represented by combining simpler, lower-dimensional transformations. By using A (a d×r matrix) and B (an r×k matrix), where r (the “rank”) is much smaller than d or k, the number of trainable parameters in BA is significantly reduced compared to directly training ΔW.</p><p>During fine-tuning, the original pre-trained weights W0​ are <strong>frozen</strong> and remain unchanged. Only the newly introduced matrices A and B are trained.</p><h3>The Benefits of LoRA</h3><p>The implications of this simple yet powerful idea are profound:</p><ul><li><strong>Drastically Reduced Trainable Parameters:</strong> By only training the small A and B matrices, LoRA significantly cuts down the number of parameters that need to be updated. This translates to faster training times and lower computational costs.</li><li><strong>Lower Memory Footprint:</strong> Since the original model weights are frozen, LoRA only needs to store the much smaller A and B matrices for each fine-tuned task. This makes it feasible to store multiple specialized versions of an LLM without prohibitive memory demands.</li><li><strong>Faster Deployment and Switching:</strong> Swapping between different fine-tuned LoRA models is incredibly fast, as you only need to load the small A and B matrices, not the entire base model.</li><li><strong>Mitigation of Catastrophic Forgetting:</strong> By keeping the original pre-trained weights frozen, LoRA helps preserve the general knowledge and capabilities of the base LLM, reducing the risk of catastrophic forgetting.</li><li><strong>Democratization of LLM Fine-Tuning:</strong> With its efficiency gains, LoRA makes it more practical for individuals and smaller organizations with limited resources to fine-tune LLMs for their specific applications.</li></ul><h3>LoRA in Practice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Dve095jjg047bT1uSStqw.png" /></figure><p>LoRA has gained immense popularity in the AI community, becoming a go-to method for efficiently adapting LLMs for various downstream tasks like:</p><ul><li><strong>Domain-specific chatbots:</strong> Training an LLM to excel in a particular industry like healthcare or finance.</li><li><strong>Sentiment analysis:</strong> Adapting an LLM to identify positive or negative sentiment in text with higher accuracy for specific use cases.</li><li><strong>Code generation:</strong> Fine-tuning an LLM to generate code in a particular programming language or style.</li></ul><p>Tools and libraries like Hugging Face’s PEFT (Parameter-Efficient Fine-Tuning) integrate LoRA, making it relatively straightforward for developers to implement.</p><h3>The Future of LLM Adaptation</h3><p>LoRA represents a significant leap forward in making LLMs more adaptable and accessible. It underscores a growing trend in AI research: finding innovative ways to achieve powerful results with greater efficiency. As LLMs continue to grow in size and complexity, techniques like LoRA will be crucial for unlocking their full potential across a wider range of applications, empowering more users to harness the power of AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29016a2771f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Fast Beginner’s Guide to Ollama and LLMs: Unlocking the Power of Local AI]]></title>
            <link>https://medium.com/@samanch70/a-fast-beginners-guide-to-ollama-and-llms-unlocking-the-power-of-local-ai-f384a7422eb5?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f384a7422eb5</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Fri, 09 May 2025 10:41:46 GMT</pubDate>
            <atom:updated>2025-05-09T10:41:46.334Z</atom:updated>
            <content:encoded><![CDATA[<p>Artificial intelligence is becoming increasingly accessible to everyday users, and Ollama stands at the forefront of this democratization movement. This powerful tool allows you to run sophisticated Large Language Models (LLMs) directly on your personal computer, without requiring constant internet connections or sending your data to third-party services. In this beginner-friendly guide, we’ll walk through everything you need to know to get started with Ollama and harness the capabilities of AI on your local machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vzlcBknC_jS63n_JsxPP-w.png" /><figcaption>(AI Generated)</figcaption></figure><h3><strong>What is Ollama and Why Should You Care?</strong></h3><p>Ollama is an open-source application that enables you to run large language models locally on your computer. If you’ve ever used ChatGPT or similar AI services, Ollama lets you bring that same functionality to your desktop, but with greater privacy, no usage limits, and no subscription fees.</p><h3>The key benefits of using Ollama include:</h3><p>1. <strong>Privacy</strong>: Your data never leaves your device</p><p>2. <strong>No internet dependency</strong>: Once models are downloaded, you can use them offline</p><p>3. <strong>Customization</strong>: You can fine-tune models for specific tasks</p><p>4. <strong>Cost-effectiveness</strong>: No subscription fees or API costs</p><p>5. <strong>Learning</strong>: Direct experience with how LLMs work</p><p>With Ollama, you can run powerful AI models like Llama 3.2, which was previously only accessible through cloud services, right on your personal computer.</p><h3><strong>Getting Started: Installing Ollama</strong></h3><p>One of Ollama’s strengths is its straightforward installation process. Let’s look at how to get it running on macOS, though the process is similar for other platforms.</p><h4><strong>Installing</strong></h4><p>1. <strong>Download the application</strong>: Visit the Ollama download page and click on the “Download for [OS]”</p><p>2. <strong>Extract and install</strong>: Once downloaded, locate the .zip file in your Downloads folder, double-click to extract, and install the Ollama.</p><p>3. <strong>Launch Ollama</strong>: Open Ollama from your Applications folder on MacOS or start menu on Windows.</p><p>4. <strong>Complete setup</strong>: Follow the installation wizard, which will also set up the command-line version of Ollama.</p><p>The installation process is remarkably simple compared to many other AI tools. As one user commented, “I got a troll comment suggesting one of the tools that is hard to install is easier than Ollama. OMG. Totally a troll.”</p><h3><strong>Running Your First LLM</strong></h3><p>After installing Ollama, you’re ready to download and run your first language model. Ollama makes this process remarkably simple.</p><p><strong>Downloading and Running a Model</strong></p><p>1. <strong>Open Terminal</strong>: Launch the Terminal application on your Mac or PC.</p><p>2. <strong>Run your first model</strong>: Type the following command and press Enter:</p><pre>ollama run llama3</pre><p>3. <strong>Wait for download</strong>: The first time you run this command, Ollama will download the model. This may take some time depending on your internet speed.</p><p>4. <strong>Start chatting</strong>: Once the model is loaded, you’ll see a prompt like:</p><pre>&gt;&gt;&gt; Send a message (/? for help)</pre><p>You can now start typing messages and interacting with the AI.</p><p>This simple process gets you up and running with a powerful LLM in just a few minutes-no complex configuration required.</p><h3>Beginner-Friendly Models to Explore with Ollama</h3><p>The Ollama model library offers a wide selection of LLMs, which can be overwhelming for newcomers. Here are a few recommendations for beginner-friendly models that strike a good balance between size, speed, and overall capabilities :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*grXr-zYWJRv10C9MU_BLhw.png" /></figure><h3><strong>Basic Commands and Interactions</strong></h3><p>Once you have Ollama running, there are several basic commands that will help you manage and interact with your models.</p><p><strong>Managing Your Models</strong></p><p>To see what models you have installed:</p><pre>ollama list</pre><p>To remove a model and free up disk space:</p><pre>ollama rm llama3</pre><p>To get help with available commands:</p><pre>ollama</pre><p>These simple commands give you complete control over what models are installed on your system.</p><h3><strong>Chatting with Models</strong></h3><p>When interacting with a model, you can simply type your messages at the prompt. The model will respond based on its training. To get help during a chat session, type /? and press Enter<a href="#fn1">[1]</a>.</p><p>The chat interface is intentionally minimal, allowing you to focus on your conversation with the AI without distractions.</p><h4><strong>Advanced Usage: The Modelfile</strong></h4><p>As you become more comfortable with Ollama, you might want to customize your models. Ollama uses a special file format called a “Modelfile” that acts as a blueprint for creating and sharing models<a href="#fn3">[3]</a>.</p><h4><strong>Understanding the Modelfile</strong></h4><p>A Modelfile has a simple structure:</p><pre># comment<br>INSTRUCTION arguments</pre><p>Key instructions include:</p><p>· FROM (required): Defines the base model to use</p><p>· PARAMETER: Sets parameters for how Ollama will run the model</p><p>· TEMPLATE: The full prompt template sent to the model</p><p>· SYSTEM: Specifies the system message in the template</p><h3><strong>Creating a Custom Model</strong></h3><p>To use a Modelfile:</p><p>1. Save your Modelfile (for example, as Modelfile)</p><p>2. Create your custom model with:</p><pre>ollama create my-custom-model -f ./Modelfile</pre><p>3. Run your custom model:</p><pre>ollama run my-custom-model</pre><p>This allows you to customize how models behave, giving them specific personalities or specialized knowledge<a href="#fn3">[3]</a>.</p><h3><strong>Integrating Ollama with Python</strong></h3><p>For developers, Ollama offers a Python library that makes integrating AI capabilities into your projects remarkably straightforward.</p><h4><strong>Installing the Python Library</strong></h4><pre>pip install ollama</pre><h4><strong>Basic Python Integration</strong></h4><p>Here’s a simple example of using Ollama from Python:</p><pre>from ollama import chat<br>from ollama import ChatResponse<br><br>response: ChatResponse = chat(<br>    model=&#39;llama3.2&#39;, <br>    messages=[{<br>        &#39;role&#39;: &#39;user&#39;,<br>        &#39;content&#39;: &#39;Why is the sky blue?&#39;,<br>    }]<br>)<br><br>print(response[&#39;message&#39;][&#39;content&#39;])</pre><p>This code snippet demonstrates how easy it is to integrate Ollama’s capabilities into your Python applications<a href="#fn4">[4]</a>.</p><h3><strong>User-Friendly Interfaces for Ollama</strong></h3><p>While the command line interface is powerful, there are several alternative interfaces that can enhance your experience with Ollama.</p><h4><strong>Open WebUI</strong></h4><p>Open WebUI provides a graphical interface for Ollama, making it more accessible for those who prefer not to use the command line. It can be installed with Docker:</p><pre>docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main</pre><p>After installation, you can access Open WebUI at <a href="http://localhost:3000/">http://localhost:3000</a>, providing a more visual and intuitive way to interact with your models<a href="#fn5">[5]</a>.</p><h4><strong>Terminal Interface with oterm</strong></h4><p>If you prefer terminal-based interfaces but want something more feature-rich than the basic command line, oterm offers:</p><p>· Multiple persistent chat sessions</p><p>· Support for Model Context Protocol (MCP) tools &amp; prompts integration</p><p>· Easy customization of model parameters</p><p>· Support for displaying images in the terminal</p><p>You can install oterm with a simple command:</p><pre>uvx oterm</pre><p>This gives you a more powerful terminal interface while maintaining the simplicity that makes Ollama appealing.</p><h3><strong>Learning Resources</strong></h3><p>To deepen your understanding of Ollama and how to craft effective prompts, there’s a comprehensive course structure available that includes:</p><p>· 9 chapters with accompanying exercises</p><p>· Beginner topics like basic prompt structure and being clear and direct</p><p>· Intermediate topics like separating data from instructions and formatting output</p><p>· Advanced topics like avoiding hallucinations and building complex prompts</p><p>Each lesson includes an “Example Playground” area where you can experiment with the examples and see how changing prompts affects responses.</p><h3>Cool Things You Can Do: Engaging Use Cases for Beginners</h3><p>Once you have a model up and running, you can start exploring the exciting possibilities of local AI. Here are a few simple and engaging use cases that beginners can easily try out :</p><ul><li><strong>Writing Assistant:</strong> You can ask your local LLM to help you draft emails, brainstorm ideas for a blog post, or even outline an article. It can also assist with checking grammar and spelling in your writing.</li><li><strong>Coding Helper:</strong> If you’re learning to code, you can ask the model to explain code snippets, generate simple code for you, or help you identify potential bugs in your scripts (though always double-check its suggestions for accuracy).</li><li><strong>Learning and Research:</strong> Local LLMs can be excellent learning tools. You can ask them to summarize information on a particular topic, explain complex concepts in simpler terms, or even generate quiz questions to test your understanding.</li><li><strong>Creative Writing:</strong> Unleash your creativity by prompting the model to generate stories, poems, or even scripts for fun.</li><li><strong>Personal Productivity:</strong> You can use your local AI to create to-do lists, brainstorm solutions to everyday problems, or even help you organize your thoughts.</li><li><strong>Role-playing and Fun:</strong> For a more interactive experience, try engaging in role-playing scenarios with the model, asking it to adopt different personas.</li><li><strong>Simple Data Analysis:</strong> If you have a small dataset or report, you can try asking the model to summarize the key findings or identify trends.</li></ul><p>These are just a few examples to get you started. The true power of having a local LLM lies in your ability to experiment and discover how it can best assist you in your own unique needs and interests. For developers, local LLMs offer a fantastic way to test the functionality of applications before incurring costs with paid cloud-based APIs.</p><h3>Taking it Further: A Glimpse into Customization and Integration</h3><p>As you become more comfortable with the basics of Ollama, you might want to explore some of its more advanced features. <strong>Modelfiles</strong> are a powerful tool that allows you to customize the behavior of your language models. Using a Modelfile (which has a syntax similar to a Dockerfile), you can define system prompts that instruct the model to behave in a certain way, adjust parameters like temperature (which controls the randomness of the output), and even build upon existing models to create specialized versions. For example, you could create a model that always responds in the style of a pirate or one that is specifically tailored to answer questions about a particular topic.</p><p>For users who prefer a graphical interface over the command line, Ollama can be used in conjunction with web interfaces like <strong>Open WebUI</strong>. These web UIs provide a more visually appealing and user-friendly way to interact with your local LLMs.</p><p>Furthermore, for developers, Ollama offers seamless <strong>integration with programming languages</strong> through official libraries for Python and JavaScript. These libraries allow you to easily incorporate the power of local LLMs into your own applications and scripts. For instance, with just a few lines of Python code using the Ollama library, you can send prompts to a model and receive responses programmatically. This opens up a world of possibilities for building custom AI-powered tools that run entirely on your local machine.</p><h3>Troubleshooting Tips for New Ollama Users</h3><p>While Ollama is designed to be user-friendly, beginners might occasionally encounter some issues. Here are a few common problems and potential solutions :</p><ul><li><strong>Installation issues:</strong> Ensure that you have downloaded the correct installer for your operating system and that you have run it properly. Double-check that your computer meets the minimum system requirements, particularly regarding RAM and storage space. On Windows, try running the installer as an administrator.</li><li><strong>“ollama” command not recognized:</strong> If you type ollama in the terminal and it&#39;s not recognized, try restarting your computer. On Windows, ensure that the Ollama installation directory has been added to your system&#39;s PATH environment variable.</li><li><strong>Model download fails or is slow:</strong> Check your internet connection to make sure it’s stable. Also, verify that you have enough free disk space on your computer to accommodate the model you are trying to download.</li><li><strong>Model runs slowly or not at all:</strong> Ensure that your computer has enough RAM to run the model. Larger models require more memory. Consider starting with smaller models. If you have a compatible NVIDIA or AMD GPU, make sure you have the latest drivers installed, as Ollama can leverage the GPU for faster processing.</li><li><strong>Unexpected or nonsensical answers:</strong> Sometimes, the model’s response might not be what you expected. Try rephrasing your prompt or asking the question in a different way. You could also try experimenting with a different language model.</li></ul><p>For more in-depth troubleshooting and to connect with other Ollama users, refer to the official Ollama documentation and community resources, such as the r/ollama subreddit</p><h3><strong>Conclusion: Your AI Journey Has Just Begun</strong></h3><p>Ollama represents an exciting step forward in making AI accessible to everyone. By running language models locally, you maintain control over your data while having the freedom to experiment and learn about AI capabilities firsthand.</p><p>Starting with the simple installation process and progressing through running your first model, you’ve now learned the basics of using Ollama. As you grow more comfortable, you can explore more advanced features like custom Modelfiles, Python integration, and alternative user interfaces.</p><p>The AI revolution is no longer limited to large corporations or cloud services-it’s running right on your desktop, waiting for you to explore its capabilities. Whether you’re looking to boost your productivity, create new applications, or simply learn about AI, Ollama provides an ideal entry point into the world of local LLMs.</p><p>So what will you build with your new AI assistant? The possibilities are limited only by your imagination.</p><p>⁂ Resources:</p><p>1. <a href="https://www.metriccoders.com/post/how-to-install-and-run-ollama-on-macos">https://www.metriccoders.com/post/how-to-install-and-run-ollama-on-macos</a></p><p>2. <a href="https://www.youtube.com/watch?v=oI7VoTM9NKQ">https://www.youtube.com/watch?v=oI7VoTM9NKQ</a></p><p>3. <a href="https://www.restack.io/p/ollama-answer-model-file-tutorial-cat-ai">https://www.restack.io/p/ollama-answer-model-file-tutorial-cat-ai</a></p><p>4. <a href="https://github.com/ollama/ollama-python">https://github.com/ollama/ollama-python</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f384a7422eb5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Beyond Data  Parallelism: A Beginner-Friendly Tour of Model, Pipeline, and Tensor Multi-GPU…]]></title>
            <link>https://medium.com/@samanch70/beyond-data-parallelism-a-beginner-friendly-tour-of-model-pipeline-and-tensor-multi-gpu-a9fdf2e8176d?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9fdf2e8176d</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Tue, 06 May 2025 14:55:39 GMT</pubDate>
            <atom:updated>2025-05-06T14:55:39.619Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀 Beyond Data Parallelism: A Beginner-Friendly Tour of Model, Pipeline, and Tensor Multi-GPU Parallelism</h3><p>Scaling up deep learning often means using multiple GPUs in parallel. This article explores four major multi-GPU training strategies — <strong>Data Parallelism</strong>, <strong>Model Parallelism</strong>, <strong>Tensor Parallelism</strong>, and <strong>Pipeline Parallelism</strong> — explaining each with examples in PyTorch and TensorFlow. We compare their trade-offs, show code snippets, and discuss when and how to combine them for training very large models or datasets. Tools like NVIDIA’s NCCL, PyTorch’s DDP, and TensorFlow’s distribution strategies will also be covered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZS0PrJ6wS7Q5D0sz0ULE2w.png" /><figcaption>(Generated by ChatGPT)</figcaption></figure><h3>Why Multi-GPU?</h3><p>As models and datasets grow, single-GPU training becomes infeasible. For example, very large models <strong>“don’t fit on a single GPU”</strong> unless you distribute them across devices. Multi-GPU strategies split either the data or the model (or both) so that GPUs can work in parallel. The right strategy depends on your setup and goals. Generally: if your model <em>fits</em> on one GPU but you need more speed or memory, <strong>data parallelism</strong> or an optimizer like ZeRO is easiest. If your model <em>doesn’t fit</em> on one GPU, you need to split it (via <strong>pipeline</strong> or <strong>tensor</strong> parallelism) or use memory-saving techniques like ZeRO.</p><h3>Data Parallelism</h3><p><strong>Data Parallelism</strong> is the most common and simplest approach. Each GPU holds a <em>complete copy</em> of the model and processes a different mini-batch of data. After each forward/backward pass, the gradients are averaged (all-reduced) so that all model copies stay in sync. This effectively parallelizes training along the batch dimension.</p><ul><li><strong>How it works:</strong> Split each training batch across GPUs. Each GPU computes forward/backward independently. Then synchronize gradients across GPUs (typically with an <strong>all-reduce</strong> operation) before the optimizer step.</li><li><strong>Pros:</strong> Easy to implement; scales well to many GPUs or nodes. (For example, PyTorch’s DistributedDataParallel and TensorFlow’s MirroredStrategy/MultiWorkerMirroredStrategy automate this.) It gives nearly linear speedups when communication is fast (e.g. with NVLink/NCCL).</li><li><strong>Cons:</strong> Each GPU must have enough memory to store the <em>entire</em> model and optimizer states. Gradient synchronization can become a bottleneck if the network is slow or the model is very large. Data parallel scaling is limited by communication overhead as you add GPUs.</li></ul><blockquote><strong><em>Tip:</em></strong><em> Always prefer optimized distributed APIs over naïve replication. In PyTorch, use </em><em>DistributedDataParallel (DDP) rather than the older </em><em>DataParallel module. DDP launches one process per GPU and overlaps gradient synchronization with the backward pass. In fact, PyTorch’s own docs note: “DDP is recommended because it reduces communication overhead … and scales to more than one machine”. Similarly, TensorFlow’s </em><em>MirroredStrategy or </em><em>MultiWorkerMirroredStrategy set up all-reduce (often via NCCL) under the hood.</em></blockquote><p><strong>Example (PyTorch DDP):</strong> Wrap your model with DistributedDataParallel, one process per GPU. Each GPU does a forward/backward on its chunk of data; DDP automatically all-reduces gradients across GPUs in the backward pass. Below is a minimal example (conceptually run via torch.multiprocessing.spawn):</p><pre># PyTorch: Data Parallel (using DistributedDataParallel)<br>import torch, torch.nn as nn, torch.distributed as dist<br>from torch.nn.parallel import DistributedDataParallel as DDP<br><br>def setup(rank, world_size):<br>    dist.init_process_group(&quot;nccl&quot;, rank=rank, world_size=world_size)<br>    torch.cuda.set_device(rank)<br><br>def cleanup():<br>    dist.destroy_process_group()<br><br>class ToyModel(nn.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.net = nn.Sequential(nn.Linear(1000, 1000), nn.ReLU(), nn.Linear(1000, 10))<br>    def forward(self, x):<br>        return self.net(x)<br><br>def demo_ddp(rank, world_size):<br>    setup(rank, world_size)<br>    model = ToyModel().to(rank)<br>    ddp_model = DDP(model, device_ids=[rank])    # Wrap model<br>    optimizer = torch.optim.SGD(ddp_model.parameters(), lr=0.001)<br>    x = torch.randn(64, 1000).to(rank)<br>    target = torch.randint(0, 10, (64,)).to(rank)<br>    optimizer.zero_grad()<br>    loss = nn.functional.cross_entropy(ddp_model(x), target)<br>    loss.backward()   # Gradients are all-reduced here<br>    optimizer.step()<br>    cleanup()<br><br># (In practice, use torch.multiprocessing.spawn to launch demo_ddp on each GPU.)</pre><p><strong>Example (TensorFlow MirroredStrategy):</strong> TensorFlow’s MirroredStrategy automatically replicates the model on all GPUs and keeps weights synced. The framework splits input batches for you:</p><pre># TensorFlow: Data Parallel (MirroredStrategy on a single machine)<br>import tensorflow as tf<br><br>strategy = tf.distribute.MirroredStrategy()<br>with strategy.scope():<br>    model = tf.keras.Sequential([<br>        tf.keras.layers.Dense(1000, activation=&#39;relu&#39;),<br>        tf.keras.layers.Dense(10)<br>    ])<br>    model.compile(loss=&#39;sparse_categorical_crossentropy&#39;, optimizer=&#39;adam&#39;)<br><br>x_train = tf.random.normal((1000, 1000))<br>y_train = tf.random.uniform((1000,), maxval=10, dtype=tf.int32)<br>model.fit(x_train, y_train, epochs=5, batch_size=64)</pre><p><strong>Pros and Cons (Data Parallelism):</strong></p><ul><li><strong>Pros:</strong> Very simple to implement with modern libraries. Scales nearly linearly with GPUs (especially on a single machine with NVLink). Reduces training time by processing more data in parallel. Compatible with most models unchanged.</li><li><strong>Cons:</strong> Limited by memory of one GPU (model must fit). Communication (all-reduce) can dominate time for very large models or many GPUs. It also replicates optimizer states on each GPU, so memory use is higher.</li></ul><blockquote><strong><em>Warning:</em></strong><em> All GPUs must stay in sync. Any imbalance (e.g. uneven batch splits, straggler machines) can cause slowdowns or even deadlock. For example, PyTorch DDP processes must launch and complete backward passes in the same order on all ranks. If some GPUs finish early, they may idle waiting for others. Always ensure roughly equal workloads per GPU or increase communication timeout.</em></blockquote><h3>Model Parallelism</h3><p><strong>Model Parallelism</strong> splits <em>the model itself</em> across multiple GPUs. Instead of replicating the entire model on each GPU, different GPUs hold different layers or parts of the model. In the simplest form, you assign successive layers of a network to different devices. For example, GPU 0 computes layers 1–3, GPU 1 computes layers 4–6, etc. The forward pass sends activations from one GPU to the next; the backward pass sends gradients in reverse order.</p><ul><li><strong>How it works:</strong> Partition the model into “chunks” of layers. During forward, GPU 0 processes input through its chunk, then passes the output tensor to GPU 1 for the next chunk. Repeat until the final output. Backprop is analogous in reverse.</li><li><strong>Pros:</strong> Allows very large models (that exceed one GPU’s memory) to be trained. Only part of the model is stored on each GPU, so you overcome single-GPU memory limits.</li><li><strong>Cons:</strong> Typically <strong>slower</strong> than data parallelism. Only one GPU can work at a time on each data sample (until pipeline parallelism is introduced). This leads to low utilization unless carefully pipelined or batched. Also, every layer transfer incurs communication overhead. HuggingFace notes that model-parallel training “only one GPU is active at a time” and inter-GPU transfers can be a bottleneck.</li></ul><p><strong>Example (PyTorch Model Parallel):</strong> You can manually place layers on different GPUs. Here, we put layer1 on cuda:0 and layer2 on cuda:1:</p><pre># PyTorch: Simple Model Parallel (manual layer placement)<br>import torch, torch.nn as nn<br><br>class ModelParallel(nn.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.layer1 = nn.Linear(1000, 1000).to(&#39;cuda:0&#39;)<br>        self.relu   = nn.ReLU()<br>        self.layer2 = nn.Linear(1000, 10).to(&#39;cuda:1&#39;)<br><br>    def forward(self, x):<br>        x = x.to(&#39;cuda:0&#39;)         # send input to GPU0<br>        x = self.relu(self.layer1(x))<br>        x = x.to(&#39;cuda:1&#39;)         # send intermediate to GPU1<br>        return self.layer2(x)      # compute on GPU1<br><br>model = ModelParallel()<br>out = model(torch.randn(64, 1000))  # Forward across GPUs<br>print(out.device)  # should be cuda:1</pre><p><strong>Example (TensorFlow Model Parallel):</strong> TensorFlow lets you specify devices with tf.device. We can place different layers on different GPUs:</p><pre># TensorFlow: Model Parallelism (manual device placement)<br>import tensorflow as tf<br><br>with tf.device(&#39;/GPU:0&#39;):<br>    dense1 = tf.keras.layers.Dense(1000, activation=&#39;relu&#39;)<br>with tf.device(&#39;/GPU:1&#39;):<br>    dense2 = tf.keras.layers.Dense(10)<br><br>inputs = tf.keras.Input(shape=(1000,))<br>x = dense1(inputs)               <br>with tf.device(&#39;/GPU:1&#39;):<br>    outputs = dense2(x)           # compute second layer on GPU:1<br><br>model = tf.keras.Model(inputs, outputs)<br>y_pred = model(tf.random.normal((64, 1000)))<br>print(y_pred.device)  # e.g. &#39;/device:GPU:1&#39;</pre><p><strong>Pros and Cons (Model Parallelism):</strong></p><ul><li><strong>Pros:</strong> Enables training of extremely large models that cannot fit on one GPU by splitting parameters across devices. Memory footprint on each GPU is reduced.</li><li><strong>Cons:</strong> As described by HuggingFace, GPU utilization is <strong>unbalanced</strong> because work happens sequentially: “only one GPU is active at a time”. This leads to lower throughput than data parallel training. Inter-GPU communication (passing activations) also adds overhead. Implementation can be complex since you must manage tensor transfers and ensure correct ordering.</li></ul><blockquote><strong><em>Tip:</em></strong><em> Only use pure model parallelism if necessary (e.g. a single layer is too big). Often, model parallelism is combined with pipeline parallelism (to improve utilization) or with data parallelism (to have multiple data streams). PyTorch also offers higher-level APIs like </em><em>torch.distributed.rpc or </em><em>torch.distributed.pipeline.sync for more complex cases, and libraries like DeepSpeed/Megatron automate model parallel splits.</em></blockquote><h3>Pipeline Parallelism</h3><p><strong>Pipeline Parallelism</strong> is a form of model parallelism that batches the model layers into <em>stages</em> and pipelines the data through them in micro-batches. Think of a CPU pipeline: while GPU1 processes micro-batch 1 on layers 1–3, GPU2 can simultaneously process micro-batch 0 on layers 4–6. This overlaps computation across GPUs and reduces idle time.</p><ul><li><strong>How it works:</strong> Split the model into stages (contiguous blocks of layers). Split each mini-batch of data into several <strong>micro-batches</strong>. Push micro-batches through the pipeline: as soon as stage 1 finishes a micro-batch, it sends it to stage 2 and begins the next micro-batch. Backprop similarly goes stage-by-stage in reverse. This “staggered” execution improves utilization.</li><li><strong>Pros:</strong> Improves GPU utilization compared to simple model parallelism. Multiple GPUs work concurrently on different parts of the same mini-batch. No replica of full model per GPU, so can still fit larger models.</li><li><strong>Cons:</strong> More complex to implement. There will always be some <em>pipeline bubbles</em> (e.g. the first and last stages are partially idle when filling and draining the pipeline). Scheduling and micro-batch sizing become important. Models may need to be restructured (for example into nn.Sequential blocks) to fit the pipeline API.</li></ul><blockquote><strong><em>Note:</em></strong><em> HuggingFace emphasizes that pipeline parallelism “shares the same advantages as model parallelism” (i.e., memory savings for big models) but “optimizes GPU utilization and reduces idle time”. It can still be complex, and complete elimination of idle time isn’t possible (the last forward pass must wait for the backward pass to finish).</em></blockquote><p><strong>Example (PyTorch Pipeline):</strong> PyTorch offers torch.distributed.pipeline.sync.Pipe, but here’s a simple manual illustration with two stages and micro-batching:</p><pre># PyTorch: Pipeline Parallelism (manual 2-stage pipeline)<br>import torch, torch.nn as nn<br><br>class Stage1(nn.Module):<br>    def __init__(self): <br>        super().__init__(); <br>        self.layer = nn.Linear(1000, 1000).to(&#39;cuda:0&#39;)<br>    def forward(self, x):<br>        return torch.relu(self.layer(x))<br><br>class Stage2(nn.Module):<br>    def __init__(self): <br>        super().__init__();<br>        self.layer = nn.Linear(1000, 10).to(&#39;cuda:1&#39;)<br>    def forward(self, x):<br>        return self.layer(x)<br><br>stage1 = Stage1()<br>stage2 = Stage2()<br><br># Simulate micro-batching<br>batch = torch.randn(64, 1000)<br>micro_batches = torch.chunk(batch, 4)  # e.g. 4 micro-batches<br><br>outputs = []<br>for micro in micro_batches:<br>    out1 = stage1(micro.to(&#39;cuda:0&#39;))   # GPU0 processes micro-batch<br>    out2 = stage2(out1.to(&#39;cuda:1&#39;))    # GPU1 processes it<br>    outputs.append(out2)<br><br>output = torch.cat(outputs, dim=0)<br>print(output.shape)  # (64, 10)</pre><p><strong>Example (TensorFlow “Pipeline”):</strong> TensorFlow has no built-in pipeline strategy, but you can mimic it by manually splitting batches and placing parts of the model on different devices:</p><pre># TensorFlow: Pipeline-like Manual Split<br>import tensorflow as tf<br><br>with tf.device(&#39;/GPU:0&#39;):<br>    part1 = tf.keras.layers.Dense(1000, activation=&#39;relu&#39;)<br>with tf.device(&#39;/GPU:1&#39;):<br>    part2 = tf.keras.layers.Dense(10)<br><br>batch = tf.random.normal((64, 1000))<br>micro_batches = tf.split(batch, 4, axis=0)<br>outputs = []<br>for micro in micro_batches:<br>    x1 = part1(micro)   # runs on GPU0<br>    x2 = part2(x1)      # runs on GPU1 (if x1 moved to GPU1 after first op)<br>    outputs.append(x2)<br>output = tf.concat(outputs, axis=0)<br>print(output.shape)   # (64, 10)</pre><p><strong>Pros and Cons (Pipeline Parallelism):</strong></p><ul><li><strong>Pros:</strong> Significantly reduces idle GPU time compared to naive model parallel. Multiple GPUs can work simultaneously on different micro-batches, improving throughput. You can fit large models (split by layers) and still keep all GPUs busy.</li><li><strong>Cons:</strong> Complex scheduling. You must choose the number of micro-batches, balance stage workloads, and manage pipeline flush and fill. There are “pipeline bubbles” (some idle time at start/end) that can’t be fully eliminated. Implementation often requires rethinking the model structure (e.g. splitting into sequential blocks or using pipeline APIs). Debugging distributed pipeline code is trickier.</li></ul><blockquote><strong><em>Tip:</em></strong><em> If you use PyTorch, consider </em><em>torch.distributed.pipeline.sync.Pipe or libraries like FairScale/DeepSpeed’s pipeline module. They handle much of the micro-batching and scheduling for you. Also, combining pipeline parallelism with data parallelism (replicating the pipeline across multiple nodes) is common in very large training jobs.</em></blockquote><h3>Tensor Parallelism</h3><p><strong>Tensor Parallelism</strong> (also called intra-layer or sharding parallelism) splits <em>individual tensors</em> (e.g. weight matrices of a layer) across GPUs. For example, a large weight matrix might be partitioned by columns: GPU 0 holds columns 1–k, GPU 1 holds columns k+1–n. During a forward pass, each GPU computes its partial result (e.g. the product of inputs with its weight slice), and then the pieces are combined (all-gather) to form the full output. This is often used for very wide/deep layers (common in large Transformers).</p><ul><li><strong>How it works:</strong> Within each layer, split one or more large tensors into shards across devices. Each GPU computes on its shard. After computing, GPUs communicate partial results (e.g. using all-gather or reduce-scatter) to assemble the complete output. This happens for forward and backward passes. No GPU holds the entire tensor in memory at once.</li><li><strong>Pros:</strong> Enables training models with layers too large for one GPU by distributing the weights. It can significantly reduce memory per GPU. Also, multiple GPUs work on the <strong>same</strong> sample in parallel (unlike data parallelism, which splits samples), which can reduce batch-related overhead. It is often faster for large tensors because the compute can be evenly split.</li><li><strong>Cons:</strong> Communication cost: after computing partial outputs, GPUs must exchange data to combine results. This all-gather step can be expensive. Implementation is more complex than data parallelism. Tensor parallelism is also limited by having to pick a dimension to split (not all layers can be efficiently sharded). As HuggingFace notes, “tensor parallelism adds communication overhead” between GPUs.</li></ul><blockquote><strong><em>Tip:</em></strong><em> Tensor parallelism is widely used in large-scale transformer training (e.g. Megatron-LM’s 1D/2D/3D parallelism). Libraries like Megatron, DeepSpeed, or Colossal-AI automate these splits. If coding manually, you can use PyTorch’s distributed collectives or TensorFlow’s strategy to shard variables.</em></blockquote><p><strong>Example (PyTorch Tensor Parallel):</strong> Here’s a toy example splitting a linear layer’s weight matrix across 2 GPUs. We split the weight matrix in half and multiply inputs accordingly:</p><pre># PyTorch: Tensor Parallelism (split a linear layer across 2 GPUs)<br>import torch, torch.nn as nn<br><br>class TensorParallelLinear(nn.Module):<br>    def __init__(self, in_features, out_features):<br>        super().__init__()<br>        # Split the weight matrix into two parts<br>        self.w1 = nn.Parameter(torch.randn(out_features, in_features//2)).to(&#39;cuda:0&#39;)<br>        self.w2 = nn.Parameter(torch.randn(out_features, in_features//2)).to(&#39;cuda:1&#39;)<br>        self.bias = nn.Parameter(torch.randn(out_features)).to(&#39;cuda:0&#39;)  # keep bias on GPU0 for simplicity<br><br>    def forward(self, x):<br>        x = x.to(&#39;cuda:0&#39;)<br>        x1, x2 = torch.split(x, x.size(-1)//2, dim=-1)<br>        # Compute partial outputs on each GPU<br>        out1 = torch.matmul(x1, self.w1.t())                   # on GPU0<br>        out2 = torch.matmul(x2.to(&#39;cuda:1&#39;), self.w2.t())     # on GPU1<br>        out2 = out2.to(&#39;cuda:0&#39;)<br>        # Combine and add bias (on GPU0)<br>        return out1 + out2 + self.bias<br><br>layer = TensorParallelLinear(1000, 10)<br>x = torch.randn(64, 1000)<br>y = layer(x)<br>print(y.shape)  # (64, 10)</pre><p><strong>Example (TensorFlow Tensor Parallel):</strong> TensorFlow doesn’t have built-in tensor parallelism, but you can simulate it. For instance, manually split a Dense layer’s computation across two parts:</p><pre># TensorFlow: Tensor Parallelism (manually splitting a Dense computation)<br>import tensorflow as tf<br><br>dense = tf.keras.layers.Dense(10)<br>_ = dense(tf.random.normal((1, 1000)))   # build layer to create weights<br>w = dense.weights[0]    # shape (1000, 10)<br>b = dense.weights[1]    # shape (10,)<br><br># Split weights and input<br>w1, w2 = tf.split(w, 2, axis=0)  # split rows<br>x = tf.random.normal((64, 1000))<br>x1, x2 = tf.split(x, 2, axis=1)  # split columns<br><br># Partial matmuls<br>y1 = tf.matmul(x1, w1)<br>y2 = tf.matmul(x2, w2)<br>y = y1 + y2 + b   # combine<br>print(y.shape)    # (64, 10)</pre><p><strong>Pros and Cons (Tensor Parallelism):</strong></p><ul><li><strong>Pros:</strong> Allows splitting very large layers across GPUs, overcoming single-GPU memory limits for those layers. Multiple GPUs compute on the <em>same</em> batch, which can improve utilization for wide models. It can be <strong>combined</strong> with other parallelism (e.g. tensor+pipeline) for 3D parallelism.</li><li><strong>Cons:</strong> Adds an all-gather communication for each tensor, which can slow down training if not done carefully. Only certain layer types (like linear layers or embeddings) can be partitioned easily. More tedious to implement manually. Also, each GPU still holds a full input tensor and partial weights, so memory savings are limited to the weight.</li></ul><blockquote><strong><em>Note:</em></strong><em> Tensor parallelism is often layered on top of other strategies in practice. For instance, NVIDIA’s Megatron-LM does tensor parallelism across GPUs, combined with pipeline parallelism across machines, to train billion-parameter models efficiently.</em></blockquote><h3>Combining Strategies</h3><p>The above methods aren’t mutually exclusive. In fact, production-scale training often <strong>combines</strong> them (sometimes called “hybrid” or “3D parallelism”). For example:</p><ul><li><strong>Data + Model/Pipeline Parallelism:</strong> You can replicate a pipeline-parallel model across multiple data-parallel replicas. Each replica handles a different portion of the data, and within each replica the model is pipelined across GPUs. This lets you scale to more data <em>and</em> larger models.</li><li><strong>Data + Tensor Parallelism:</strong> Similarly, you can use tensor parallelism to split layers, while also running multiple data-parallel replicas. Each replica sees a different batch, and each batch’s computation is split across GPUs by tensor.</li><li><strong>3D Parallelism (Model+Pipeline+Data):</strong> HuggingFace describes combining data, pipeline, and model (or tensor) parallelism as “3D parallelism”. In practice, frameworks like DeepSpeed combine ZeRO (data parallelism stage 3) with pipeline and tensor parallelism to maximize memory and compute efficiency. For instance, one design is: split the model’s layers into stages (pipeline), within each stage use tensor parallelism to shard weights, and replicate the whole pipeline across multiple workers (data parallel).</li></ul><p><strong>When to combine:</strong></p><ul><li>If <strong>model size</strong> alone is the constraint, combine model/pipeline/tensor methods.</li><li>If <strong>dataset size</strong> or throughput is the target, use data parallel on top of that.</li><li>The cluster/network configuration matters: on multi-node setups with fast interconnect (NVLink or InfiniBand), using data parallelism (with NCCL) and pipeline together (so-called 3D parallelism) is effective. On slower interconnects, prefer reducing communication (e.g. larger micro-batches, ZeRO stage 3, etc.).</li></ul><blockquote><strong><em>Tip:</em></strong><em> Plan your strategy based on memory vs. compute needs. Start with data parallel if possible (simplest). If your model doesn’t fit one GPU, add tensor/model parallelism. If GPUs are idle, add pipeline. Use profiling (PyTorch’s nvprof, TensorBoard, etc.) to identify bottlenecks (comm vs compute).</em></blockquote><h3>Tooling and Framework Support</h3><p>Modern libraries make it much easier to implement these strategies:</p><p><strong>PyTorch Distributed:</strong></p><ul><li><strong>DDP:</strong> torch.nn.parallel.DistributedDataParallel for data parallel. It uses NCCL (on GPUs) or Gloo (on CPUs) by default. PyTorch docs recommend “Use the NCCL backend for distributed GPU training”.</li><li><strong>Model/TP/PP:</strong> PyTorch 2.x provides torch.distributed.pipeline.sync.Pipe for pipelines, and torch.distributed.tensor APIs (and third-party libs like FairScale/DeepSpeed) for tensor parallelism. There’s also <strong>Fully Sharded Data Parallel</strong> (FSDP) which shards layers’ parameters &amp; gradients (similar to tensor parallelism at a layer level) combined with data parallelism.</li><li><strong>NCCL:</strong> NVIDIA’s NCCL library underlies GPU collectives. PyTorch’s DDP and TensorFlow’s MirroredStrategy use NCCL for fast all-reduce by default. NCCL is optimized for GPU-to-GPU communication (including NVLink, InfiniBand). Use it when you have multiple GPUs.</li></ul><p><strong>TensorFlow :</strong></p><ul><li><strong>MirroredStrategy:</strong> Synchronous data-parallel on one machine (all GPUs). It “creates one replica per GPU”, keeping variables in sync via all-reduce. By default it uses NCCL for cross-GPU communication.</li><li><strong>MultiWorkerMirroredStrategy:</strong> Synchronous data-parallel across <strong>multiple machines</strong>. It’s “very similar to MirroredStrategy” but requires a cluster spec (via TF_CONFIG). You can choose an NCCL or RING communication implementation for cross-device communication. For example:</li></ul><pre>strategy = tf.distribute.MultiWorkerMirroredStrategy()</pre><p>will let you train on multiple nodes with GPUs, synchronizing updates across all of them.</p><ul><li><strong>ParameterServerStrategy:</strong> Asynchronous (or sync) data parallel using parameter servers (less common nowadays with small clusters).</li></ul><p><strong>Other frameworks:</strong></p><ul><li><strong>Horovod:</strong> A popular library (by Uber) that uses MPI/NCCL under the hood to do data-parallel all-reduce for both TensorFlow and PyTorch with minimal code changes.</li><li><strong>DeepSpeed / FairScale / Megatron / Colossal-AI:</strong> These libraries implement combinations of data/model/pipeline/tensor parallelism. For example, DeepSpeed’s ZeRO partitions optimizer state (data parallelism), and supports pipeline + tensor parallelism. Megatron-LM focuses on tensor+pipeline parallelism for transformers.</li><li><strong>NVIDIA tools:</strong> NCCL (communication), NVLink/NVSwitch (hardware interconnect) significantly impact performance. Ensure your GPUs are connected by fast links when doing heavy inter-GPU communication.</li></ul><p><strong>Important Tips:</strong></p><ul><li><strong>Communication:</strong> Whether using DDP or tensor parallelism, communication is key. NCCL is generally best for GPUs; Gloo or MPI are alternatives. Always set up your network (even memory for NCCL) correctly.</li><li><strong>Batch sizes:</strong> When using pipeline or data parallel, you may need to reduce batch size per GPU to fit memory. Adjust learning rate accordingly.</li><li><strong>Debugging:</strong> Start with a small example (few GPUs, small model) and verify correctness. Gradually scale. Mismatch in forward/backward (e.g. forgetting to move tensors between GPUs) can cause silent errors.</li><li><strong>Profiling:</strong> Use tools (PyTorch Profiler, TensorFlow Profiler, NVIDIA Nsight) to identify idle times or communication stalls.</li></ul><h3>Conclusion</h3><p>Multi-GPU training offers huge benefits but comes with complexity. <strong>Data parallelism</strong> is easy and often your first tool — it lets you leverage more GPUs just by splitting the data. <strong>Model, tensor, and pipeline parallelism</strong> let you train bigger models but require careful implementation. Each has trade-offs: memory vs. communication vs. GPU utilization. In practice, large-scale training often uses <strong>hybrid strategies</strong> (e.g. combining data parallel with pipeline parallelism and tensor splits).</p><p>Choose a strategy based on your constraints: model size, GPU memory, interconnect speed, and code complexity. For many, starting with PyTorch’s DDP or TensorFlow’s MirroredStrategy is sufficient. For very large models, consider frameworks like DeepSpeed or Megatron that automate much of the parallelism.</p><p>By understanding and experimenting with these methods, you’ll be able to scale up your training effectively. Happy scaling!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9fdf2e8176d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Could Generative AI Make Data Science Obsolete?]]></title>
            <link>https://medium.com/@samanch70/could-generative-ai-make-data-science-obsolete-c06475e98c6c?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/c06475e98c6c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-lear]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Thu, 14 Nov 2024 13:51:48 GMT</pubDate>
            <atom:updated>2024-11-14T13:51:48.789Z</atom:updated>
            <content:encoded><![CDATA[<p>The rise of generative AI has sparked numerous debates on its potential to disrupt industries, and one of the most intriguing questions that have emerged is: <strong>Could generative AI make data science obsolete?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x6i9Aaxlu-ycrOp8wAVK0w.png" /></figure><p>In their book <em>All Hands on Tech: The AI-Powered Citizen Revolution</em>, Thomas Davenport, Professor at Babson College, and Ian Barkin, a venture capitalist, delve into this very topic. They explore how generative AI is transforming the landscape of software development and data science, making these domains more accessible to non-experts. But does this mean the end of data science as we know it?</p><h3>The Democratization of Software Development</h3><p>With the advancement of generative AI, the pathway to becoming a software developer is now open to everyone. Low-code and no-code tools, robotic process automation, and now, AI technologies, particularly generative AI, are making it easier than ever for anyone — regardless of technical background — to design software, analyze data, and even write code. This shift is monumental, especially for industries traditionally dominated by specialized experts.</p><p>Davenport and Barkin emphasize in their book that <strong>technology is no longer the exclusive domain of specific departments or expert roles.</strong> With these emerging tools, <strong>data and analytics are no longer confined to PhD holders or number-crunching specialists.</strong> In fact, they predict a future where <strong>every employee</strong> will have the ability to design systems, perform data analysis, write code, and even innovate in ways that were previously the realm of data scientists and software engineers alone.</p><p><em>“Technology no longer belongs to a specific department. Data analysis is no longer just for PhDs. Soon, every employee will be able to design, analyze, and create in ways that were once reserved for experts,”</em> they write.</p><h3>Generative AI: A Game-Changer for Citizen Development</h3><p>Generative AI models, like ChatGPT, are already at the forefront of this transformation. As Davenport and Barkin point out, tools like ChatGPT represent a first step in a new era where AI can assist in generating code, automating processes, and solving complex problems — all at the speed of thought. With such advancements, it’s no wonder that the potential for non-experts to engage in technical tasks has skyrocketed.</p><p>However, while AI tools can automate routine tasks and create complex systems, they won’t completely replace the role of human workers in these fields. Instead, <strong>AI will serve as the ultimate assistant, enabling humans to work faster, more efficiently, and with greater flexibility</strong>. The true power of AI lies in its ability to assist humans, not replace them.</p><h3>A New Era for Data Science</h3><p>So, does generative AI spell the end for data science? <strong>Not entirely.</strong> While generative AI is democratizing access to software development and data analysis, the role of skilled data scientists is still crucial. <strong>The demand for human creativity, domain expertise, and critical thinking remains high</strong>, even as AI accelerates certain processes.</p><p>Davenport and Barkin believe that <strong>AI’s role will not be to replace workers but to augment their capabilities.</strong> Rather than obviating the need for data scientists, AI will enhance their productivity, allowing them to focus on more complex problems while automating routine tasks. AI tools will provide rapid prototyping, iterative feedback, and personalized solutions, freeing up humans to focus on what they do best: creativity, strategy, and innovation.</p><p>As Davenport and Barkin note, <strong>generative AI is the ultimate research or programming assistant.</strong> It can quickly produce code for analysis, and assist with defining key features of applications or models. The result? An empowered workforce that can continuously adapt, experiment, and innovate at an accelerated pace.</p><h3>Conclusion</h3><p>In response to the question of whether generative AI could make data science obsolete, the answer is clear: <strong>not entirely.</strong> While AI will undoubtedly reshape the field, creating more intense competition and redefining traditional roles, <strong>it will not eliminate the need for human expertise.</strong></p><p>Rather, the future of data science is one of collaboration between human ingenuity and AI-powered tools, where humans focus on complex, creative tasks while AI handles the repetitive and time-consuming ones. This symbiotic relationship will redefine how we approach technology, and unlock new potential for businesses, industries, and individuals.</p><p>Generative AI isn’t here to replace data scientists — it’s here to make them more powerful.</p><p><strong>Key Takeaways:</strong></p><ul><li>Generative AI is democratizing software development and data analysis.</li><li>AI tools like low-code/no-code platforms and ChatGPT are opening doors for non-experts.</li><li>While AI will automate many tasks, human creativity, domain expertise, and strategic thinking remain essential.</li><li>The future of data science lies in a partnership between AI and human experts, where AI enhances human productivity.</li></ul><p>Sourse: <a href="https://www.zdnet.com/article/could-ai-make-data-science-obsolete/">ZDnet</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c06475e98c6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Beauty of ADAM Algorithm: Fueling Neural Networks for Fast and Efficient Learning]]></title>
            <link>https://medium.com/@samanch70/the-beauty-of-adam-algorithm-fueling-neural-networks-for-fast-and-efficient-learning-f2252e21365a?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2252e21365a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Sat, 09 Nov 2024 22:11:57 GMT</pubDate>
            <atom:updated>2024-11-09T22:11:57.751Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tMJG7g5_Vbd1mwzNmFc0-A.png" /></figure><p>In the world of deep learning, the ADAM optimizer stands out as a reliable choice, celebrated for its efficiency and effectiveness in training complex models. But what makes ADAM so powerful? In this article, we’ll unpack ADAM’s foundations, from the basics of optimization to advanced techniques like exponentially weighted averages and RMSProp. By understanding these components, you’ll get a clearer picture of why ADAM is a top choice for training deep networks.</p><h4>Optimizers: The Backbone of Model Training</h4><p>Every machine learning model needs an optimizer to adjust its parameters, such as weights and biases, to minimize a loss function, which measures how well the model’s predictions align with actual outcomes. Different optimizers use unique algorithms to make these adjustments, aiming to find a balance between speed, accuracy, and computational efficiency. Optimizers like ADAM aim to achieve fast and stable convergence while effectively navigating complex data structures.</p><h4>Gradient Descent: The Core Algorithm</h4><p>The foundation of optimization in machine learning is <strong>Gradient Descent (GD)</strong>, a simple yet powerful method for minimizing loss functions. Imagine standing on a hill and wanting to get to the lowest point. Gradient descent takes small, calculated steps downhill by computing the gradient (slope) at each point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*ZA_CAdZEeqUxPWv_Hhon6g.jpeg" /><figcaption>Gradent Decent Algorithm Space</figcaption></figure><p>In gradient descent, each step updates the parameters in the direction of the steepest descent based on the gradient:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*_kD1sqByjd6dmSI3oG_9kQ.png" /></figure><p>where:</p><p><em>θ </em>represents the parameters (weights and biases) being updated,</p><p><em>α </em>is the learning rate, controlling step size, and</p><p>∇<em>θJ </em>(<em>θ</em>) is the gradient of the loss function with respect to <em>θ</em>.</p><p>While gradient descent is effective, calculating the gradient for each point in large datasets can be time-consuming and computationally expensive. <strong>Stochastic Gradient Descent (SGD) </strong>offers a faster, approximate approach.</p><h4>Stochastic Gradient Descent: A Faster Alternative</h4><p><strong>Stochastic Gradient Descent (SGD) </strong>updates the model parameters by calculating the gradient for a single data point or a small batch at each step, rather than for the entire dataset. This approach introduces randomness in each update, adding noise to the optimization path. While this may seem like a drawback, the noise can help the optimizer escape from local minima, making SGD particularly useful for large, complex datasets.</p><p>However, the noise in SGD can cause unstable updates, slowing down convergence. To counter this, techniques like <strong>Exponentially Weighted Averages </strong>and <strong>Momentum </strong>can smooth the updates, enhancing stability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*WKXBqm28g56YGhikI6zLAw.jpeg" /><figcaption>SGD</figcaption></figure><h4>Exponentially Weighted Averages: Smoothing the Updates</h4><p><strong>Exponentially Weighted Averages </strong>reduce noise by calculating a smoothed version of the gradients. In this technique, each gradient update contributes to a moving average, with recent gradients carrying more weight than older ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4KmEU25t_G6im2q3OtCBPw.png" /><figcaption>Exponentially Weighted Averages</figcaption></figure><p>The formula for an exponentially weighted average at time <em>t </em>is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*Y_cBHIqqp9DqhBXNvuborw.png" /></figure><p>where:</p><p><em>gt </em>is the gradient at time <em>t</em>,</p><p><em>vt </em>is the exponentially weighted average at <em>t</em>, and</p><p><em>β </em>(between 0 and 1) controls the rate of decay.</p><p><strong>Example</strong>: Let’s assume we have gradient values over four steps as follows: 3, 4, −2, 1. If we set <em>β </em>= 0.9, the weighted averages become:</p><p>1. Step 1: <em>v</em>1 = 0.9 × 0 + 0.1 × 3 = 0.3</p><p>2. Step 2: <em>v</em>2 = 0.9 × 0.3 + 0.1 × 4 = 0.67</p><p>3. Step 3: <em>v</em>3 = 0.9 × 0.67 + 0.1 × (−2) = 0.403</p><p>4. Step 4: <em>v</em>4 = 0.9 × 0.403 + 0.1 × 1 = 0.4627</p><p>This smoothed gradient sequence stabilizes the updates, which reduces oscillations and helps in converging more efficiently.</p><h4>Momentum: Adding Velocity to Gradient Descent</h4><p><strong>Momentum </strong>builds on exponentially weighted averages by adding a fraction of the previous gradient to the current update, creating a “momentum” effect that propels the model faster in the optimal direction. This is similar to how a ball rolling down a hill gains speed as it moves. The formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*oKOcyLsqgUWubw-dGa3BPA.png" /></figure><p>This approach makes gradient descent more effective at escaping local minima and moving faster in regions where gradients are flatter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y2FJQL4gxW_TJywyOLty8Q.jpeg" /><figcaption>Momoentum in SGD</figcaption></figure><h4>RMSProp: Adapting the Learning Rate for Stability</h4><p><strong>RMSProp (Root Mean Square Propagation) </strong>is another technique designed to improve gradient descent by adapting the learning rate for each parameter.</p><p>RMSProp uses <strong>Exponentially Weighted Averages</strong> to adapt the learning rate for each parameter dynamically, ensuring stable convergence. Although RMSProp itself doesn’t directly include a <strong>momentum</strong> term, its functionality is similar to momentum because it effectively smooths and stabilizes parameter updates by scaling them according to the history of past squared gradients.</p><p>RMSProp maintains a moving average of squared gradients, which helps balance the magnitude of updates across parameters. This is especially useful for optimizing complex models with many layers.</p><p>The RMSProp formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*pCLSNnkwAgFmoMmzqrVvhw.png" /></figure><p>where:</p><p><em>st </em>is the exponentially weighted average of squared gradients,</p><p><em>β </em>controls decay,</p><p><em>α </em>is the learning rate</p><p><em>ϵ </em>is a small constant added for numerical stability</p><p><strong>Example</strong>: Suppose we have squared gradient values over four steps: 9, 16, 4, 1, and we set <em>β </em>= 0.9 and <em>ϵ </em>= 10−8.</p><p>1. Step 1: <em>s</em>1 = 0.9 × 0 + 0.1 × 9 = 0.9</p><p>2. Step 2: <em>s</em>2 = 0.9 × 0.9 + 0.1 × 16 = 2.41</p><p>3. Step 3: <em>s</em>3 = 0.9 × 2.41 + 0.1 × 4 = 2.57</p><p>4. Step 4: <em>s</em>4 = 0.9 × 2.57 + 0.1 × 1 = 2.413</p><p>This gradually changing update size stabilizes the optimization process, especially for noisy datasets.</p><h4>The ADAM Optimizer: Combining Momentum and Adaptive Learning</h4><p>The <strong>ADAM optimizer</strong> (short for Adaptive Moment Estimation) combines the advantages of both momentum and RMSProp, offering adaptive learning rates with added stability. Here’s how ADAM works:</p><ul><li><strong>Momentum component</strong>: ADAM keeps an exponentially weighted average of past gradients, helping retain the direction of optimal updates.</li><li><strong>Adaptive learning rates</strong>: ADAM also calculates an exponentially weighted average of squared gradients (similar to RMSProp), enabling it to adapt the learning rate for each parameter.</li></ul><p>The parameter update rule for ADAM is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*5IzVSOGjtNW4dtDo5DN8sQ.png" /></figure><p>where:</p><p><em>mt </em>and <em>vt </em>are the moving averages of gradients and squared gradients, respectively,</p><p><em>β</em>1 and <em>β</em>2 control the decay rates,</p><p><em>m</em>^<em>t </em>and <em>v</em>^<em>t </em>are bias-corrected estimates, and</p><p><em>α </em>is the learning rate.</p><h4>Why ADAM?</h4><p>ADAM’s ability to adapt learning rates for each parameter combined with momentum allows for a faster and more stable convergence, especially for deep networks on large datasets. ADAM’s reliable convergence has made it popular among researchers and developers for optimizing deep neural networks, from simple image classifiers to cutting-edge language models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*fsPTdXl9WeuNnnueuXv_CA.png" /><figcaption>ADAM in action</figcaption></figure><h4>Conclusion</h4><p>The ADAM optimizer, as we’ve explored, is more than just an enhancement of gradient descent — it’s a sophisticated tool that incorporates momentum and adaptive learning rates to make deep learning models more effective. Understanding these components offers insight into why ADAM has become such a powerful tool for researchers and practitioners alike. Whether you’re a beginner or an experienced practitioner, mastering ADAM and its underlying principles is essential for effective and efficient model training.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2252e21365a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Curve: The Magic of Sigmoid + Code]]></title>
            <link>https://medium.com/@samanch70/beyond-the-curve-the-magic-of-sigmoid-code-44775f6e334f?source=rss-ad7ad221e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/44775f6e334f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Saman Chitsazian]]></dc:creator>
            <pubDate>Sat, 09 Nov 2024 14:50:58 GMT</pubDate>
            <atom:updated>2024-11-09T14:50:58.123Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Unlocking Activation Functions in Neural Networks</strong></p><p>Let us begin with a quick example, assume we have a AND gate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/136/0*j3W2QB8wzTM6NgKn.png" /><figcaption>AND gate</figcaption></figure><p>The AND gate is a basic digital logic gate that implements logical conjunction (∧) from mathematical logic — AND gate behaves according to the truth table. A HIGH output (1) results only if all the inputs to the AND gate are HIGH (1). If not all of the inputs to the AND gate are HIGH, a LOW output results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/0*7QwVIQUMl7vK0_DN.png" /><figcaption>AND truth table</figcaption></figure><p>If you plot the truth table of the gate you would get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*LGv4xDYTmHdrY4t3.png" /><figcaption>AND gate plot</figcaption></figure><p>As you can see we have 4 data points and two categorical stand point in the plot, (0,0) , (0,1) , (1,0) and finally (1,1). As the AND gate logic goes, every input combination will result in a “zero” or a False output until the two inputs were “ones” and then the output changed to “one” or a True.</p><p>Let’s say we want to create a computational model to do the work of an AND gate by creating a decision boundary in our plot.</p><p>It seems so easy as the data points are separable linearly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/0*ZPUalZNh6LSXo7XK.png" /><figcaption>AND gate classifier</figcaption></figure><p>The plot will show the data points (0,0), (0,1), (1,0), and (1,1) with the decision boundary learned by the model. Points are color-coded based on their true class, and the decision boundary separates the 0 class from the 1 class.</p><p>But what if we have more complex data in a more complex space? How the model will learn then?</p><p>We can’t just separate the data points with a linear line! That’s where the “<strong>Activation Functions</strong>” come into play.</p><p>For example, lets plot an XOR gate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*TRNigPeRXb9oH9M0.png" /><figcaption>XOR gate plot</figcaption></figure><p>Here we need some non-linear function to correctly map data and separate the classes.</p><p>The activation function in an artificial neural network (ANN) plays a critical role in determining the output of a neuron. It introduces non-linearity into the model, which is essential for the network to learn and model complex patterns.</p><p>Here’s why the activation function is crucial:</p><p><strong>1. Introducing Non-Linearity</strong></p><p>Without an activation function, a neural network would be simply a linear combination of inputs and weights. No matter how many layers the network has, the entire network would act as a single linear transformation. This means the model could only learn linear relationships between inputs and outputs.</p><p>Activation functions like ReLU, Sigmoid, and Tanh introduce non-linearities, enabling the network to learn complex patterns, interactions, and representations in the data.</p><p><strong>2. Enabling Multi-Layer Networks</strong></p><p>In multi-layer networks (deep networks), activation functions allow each layer to learn different levels of abstraction. For example, early layers might learn simple features like edges in an image, while deeper layers learn more complex patterns like shapes or even object parts.</p><p>Without non-linear activation functions, these layers would collapse into a single layer with no added benefit from having multiple layers.</p><p><strong>3. Thresholding and Decision-Making</strong></p><p>Some activation functions, like the sigmoid, squash the output to a range (0 to 1), which can be interpreted as probabilities. This is particularly useful in classification tasks.</p><p>Activation functions like ReLU set negative values to zero, allowing networks to decide which neurons should be “active” or “inactive,” effectively creating a sparse representation and improving computational efficiency.</p><p><strong>4. Gradient-Based Learning</strong></p><p>Activation functions like Sigmoid and Tanh are differentiable, meaning that their derivatives can be used in backpropagation to update the network’s weights during training.</p><p>While ReLU has a non-linear derivative, it is still simple and effective for gradient-based optimization. The choice of activation function affects how gradients flow through the network and how efficiently the network can be trained.</p><p><strong>5. Enabling Complex Decision Boundaries</strong></p><p>Activation functions allow neural networks to form complex, non-linear decision boundaries. This is essential for solving real-world problems like image recognition, natural language processing, and other tasks where simple linear models are insufficient.</p><p><strong>Sigmoid</strong></p><p>Among these functions, the ”Sigmoid Function“ is one of the most well-known and widely used for classification tasks. In this post, we’ll dive into the magic of the sigmoid function, why it’s used, its strengths and weaknesses, and when to consider alternatives.</p><p>The sigmoid function is especially valuable in binary classification tasks, where we predict between two classes (e.g., yes/no, 0/1). By mapping the output of a model to a value between 0 and 1, the sigmoid function allows us to interpret that output as a probability of belonging to a particular class. Values closer to 1 indicate a higher probability of being in class 1, while values closer to 0 suggest class 0.</p><p>Sigmoid:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*OukqopXW67enybsK.png" /><figcaption>Sigmoid function plot</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*Gvt3axiHJ4cAgmFE.png" /></figure><p>Here, ( e ) (Euler’s Number) is approximately 2.718 and is a constant in mathematics, essential for modeling exponential growth and decay. This exponential nature helps create a smooth curve, which is crucial for gradient-based optimization.</p><p>The sigmoid function as we can see is an s-shaped curve. For any value of x the sigmoid function will output a value between 0 and 1.</p><p>As said before, sigmoid function which is denoted by the Greek letter Sigma is given as 1 over 1 plus the exponential of minus X.</p><p>Here the exponential of minus X is the inverse of the well-known exponential function how does the plot of the inverse of the exponential look like let us plot both exponential(green) and it’s inverse(red) on the same graph</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/0*plcEmTjhC_PhICQB.png" /></figure><p>the inverse of the exponential has the same profile as the exponential function the only difference is the inverse exponential is flipped with respect to the y-axis if we study the limits of the inverse of the exponential function, we can see that when X takes large negative values toward minus infinity the infests exponential diverges to plus infinity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/0*2BmFmgRP40IbuCeM.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/0*frJVqLcTSiLFCbs8.png" /></figure><p><strong>Characteristics of the Sigmoid Function:</strong></p><ul><li>Output Range: The sigmoid function maps any input x to an output value between 0 and 1. This makes it useful for models that need to predict probabilities, as the output can be interpreted as a probability.</li><li>Smooth Gradient: The sigmoid function has a smooth gradient, which helps in the learning process during backpropagation. The function’s derivative is continuous and non-zero, ensuring that small changes in input result in small changes in output.</li><li>Non-Linear: The sigmoid function introduces non-linearity into the network, enabling it to learn and model complex patterns. Without non-linearity, the network would only be able to represent linear functions, regardless of how many layers it has.</li></ul><p>One interesting property of the sigmoid function is that it effectively compresses inputs within a narrow range, roughly between -5 and 5. Beyond this range, the function approaches either 0 or 1 so closely that any further changes in x make a negligible difference. For example, any input less than -5 will produce an output very close to 0, and any input greater than 5 will be near 1.</p><p>When we look at the sigmoid function graph, it has a characteristic S-shape. This curve quickly rises from 0 to 1 as x increases, but only within a specific range. Outside this range, the curve flattens out. Here’s how it works:</p><p><strong>Middle Range Sensitivity (Approximately -5 to 5)</strong>:</p><ul><li>Between approximately -5 and 5, the sigmoid function is highly sensitive to changes in x.</li><li>Small changes in x within this range produce noticeable changes in the output.</li></ul><p><strong>Saturation Beyond -5 and 5</strong>:</p><ul><li>When x is less than -5, the exponential term grows very large (since x is negative and the exponent flips the sign), causing the denominator to also grow. This results in a very small output close to 0, no matter how much further x decreases.</li><li>Conversely, when x is greater than 5, the exponential term becomes very small because x is positive, making the denominator close to 1. As a result, the output is close to 1, regardless of further increases in x.</li></ul><p>This compression property makes sigmoid useful for models that require output in a [0, 1] range, especially for <strong>binary classification</strong>. For each input, the sigmoid function provides an interpretable probability-like output, indicating the likelihood of a sample belonging to a specific class.</p><p>While sigmoid can work well as an output activation in binary classification, it’s generally not recommended for hidden layers in deep networks. The main reason is the vanishing gradient problem. In regions where the function outputs values close to 0 or 1, the gradient (or slope) is very small. During backpropagation, this tiny gradient slows down or even stops the learning process, especially in deep networks, which need more robust gradients for efficient learning.</p><p>(The vanishing gradient problem is the main reason sigmoid isn’t widely used in hidden layers. The small gradients in the function’s “flat” regions (i.e., where it approaches 0 or 1) can slow down or even stop the training process. This issue is especially problematic for deep networks, which rely on strong gradients to propagate error signals effectively. Additionally, the sigmoid function tends to saturate for large positive or negative inputs, further slowing the learning process.)</p><h3>Sigmoid as a “Biased Average” Function</h3><p>Another interesting interpretation of the sigmoid function is as a type of <strong>biased average</strong>. When combining multiple inputs, the sigmoid function acts like a smoothing filter, effectively reducing extreme values. This makes it particularly useful when a network requires a smoothed or averaged output, though this same smoothing can be detrimental in hidden layers.</p><p>For instance:</p><pre>import numpy as np<br>def sig(x):<br> return 1/(1 + np.exp(-x))<br><br>x = 1.0<br>print(&#39;Applying Sigmoid Activation on (%.1f) gives %.1f&#39; % (x, sig(x)))<br><br>x = -10.0<br>print(&#39;Applying Sigmoid Activation on (%.1f) gives %.1f&#39; % (x, sig(x)))<br><br>x = 0.0<br>print(&#39;Applying Sigmoid Activation on (%.1f) gives %.1f&#39; % (x, sig(x)))<br><br>x = 15.0<br>print(&#39;Applying Sigmoid Activation on (%.1f) gives %.1f&#39; % (x, sig(x)))<br><br>x = -2.0<br>print(&#39;Applying Sigmoid Activation on (%.1f) gives %.1f&#39; % (x, sig(x)))</pre><p>output:</p><blockquote>Applying Sigmoid Activation on (1.0) gives 0.7<br>Applying Sigmoid Activation on (-10.0) gives 0.0<br>Applying Sigmoid Activation on (0.0) gives 0.5<br>Applying Sigmoid Activation on (15.0) gives 1.0<br>Applying Sigmoid Activation on (-2.0) gives 0.1</blockquote><p><strong>Sigmoid Derivative and Its Role in Machine Learning</strong></p><p>The derivative of the sigmoid function as an activation function is essential for backpropagation in neural networks, as it helps compute the gradient for updating weights. The sigmoid derivative’s shape ensures that the gradient remains between 0 and 0.25, which can make training slower. It’s part of why alternative activation functions like ReLU are often preferred for hidden layers in deeper networks.</p><p><strong>Sigmoid derivatives:</strong></p><p><strong>σ′(x)=σ(x)⋅(1−σ(x))</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*ISBntOH1Xw1ayprs.png" /></figure><p>This formula indicates that the derivative (or slope) of the sigmoid function depends directly on the output of the sigmoid function itself.</p><p><strong>Gradient Magnitude and the Vanishing Gradient Problem</strong>:</p><ul><li>In backpropagation, gradients are calculated based on the derivative of the activation function at each neuron. When using the sigmoid activation function, the gradients are always between 0 and 0.25.</li><li>This narrow range of gradients means that weight updates are inherently small. When the output of a neuron is close to 0 or 1 (the “saturated” regions of the sigmoid curve), the derivative becomes very close to 0, leading to extremely small gradients.</li><li>This results in the <strong>vanishing gradient problem</strong>: as gradients become very small, weight updates diminish, and training becomes slower. In deep networks, where many layers are stacked, this problem compounds as gradients are multiplied through each layer, often causing them to shrink to near-zero values for the earlier layers.</li></ul><p><strong>Effect on Learning Speed</strong>:</p><ul><li>Because the gradient is limited to 0.25 at most, the weight updates are constrained, making the learning process slower, especially when dealing with deep networks.</li><li>This is particularly problematic in complex tasks that require a network to learn a large number of parameters. The network may take many more epochs (iterations over the training data) to converge to an optimal solution or may fail to converge effectively at all.</li></ul><h3>Solution to the vanishing gradient problem: A Hint at the Hyperbolic Tangent (Tanh) Function</h3><p>For hidden layers, the <strong>hyperbolic tangent (tanh) function</strong> is often preferred over sigmoid. Tanh is similar to sigmoid but ranges between -1 and 1, allowing for a broader range of values and helping with gradient propagation. In many cases, tanh can be a better choice as it doesn’t suffer from the same level of vanishing gradient issues, making it a powerful alternative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/0*9wqfD5pJqtHFUwaK.png" /></figure><p><strong>Code:</strong></p><p><a href="https://github.com/SamanCh/pymastery/blob/main/Blog.pymasterycourse.com/The%20magic%20of%20Sigmoid.ipynb">pymastery/Blog.pymasterycourse.com/The magic of Sigmoid.ipynb at main · SamanCh/pymastery</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44775f6e334f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>