<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Syeda Ameena on Medium]]></title>
        <description><![CDATA[Stories by Syeda Ameena on Medium]]></description>
        <link>https://medium.com/@syeda9118?source=rss-bc7fd0472dd0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IS23CecY3D-E6PT0T4fibQ@2x.jpeg</url>
            <title>Stories by Syeda Ameena on Medium</title>
            <link>https://medium.com/@syeda9118?source=rss-bc7fd0472dd0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:52:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@syeda9118/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Engineering the Harness, Part II: Teaching Bob to Remember, Specialize, and Scale]]></title>
            <link>https://medium.com/@syeda9118/engineering-the-harness-part-ii-teaching-bob-to-remember-specialize-and-scale-d7f47566c55e?source=rss-bc7fd0472dd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7f47566c55e</guid>
            <category><![CDATA[ibm-bob]]></category>
            <category><![CDATA[spec-driven-development]]></category>
            <category><![CDATA[agentic-coding]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <category><![CDATA[context-window-management]]></category>
            <dc:creator><![CDATA[Syeda Ameena]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 08:17:34 GMT</pubDate>
            <atom:updated>2026-07-13T13:35:20.507Z</atom:updated>
            <content:encoded><![CDATA[<h3>Engineering the Harness, Part II: Teaching IBM Bob to Remember, Specialize, and Scale</h3><p><em>A continuation of </em><a href="https://medium.com/@syeda9118/moving-past-the-raw-model-engineering-an-agentic-harness-with-ibm-bob-8b6f5be33cec"><em>Moving Past the Raw Model: Engineering an Agentic Harness with IBM Bob</em></a></p><p>In the first part of this series, we established a core idea: when an AI coding agent behaves differently than expected, the root cause is rarely the model. It is usually the harness around it.</p><p>We explored the five pillars that make up Bob’s harness: Guidance, Memory, Capabilities, Feedback, and Scale — and showed how each one shapes what Bob can do before, during, and after it acts.</p><p><strong>But we left some important questions open</strong></p><p>How exactly do you build the capabilities layer? What does it mean in practice to create a Skill or a Subagent? How do you make AGENTS.md work over time, not just at project setup? And how do you stop Bob’s context window from becoming a liability as your project grows?</p><p>This post answers those questions. And it closes with something that has been missing from most conversations about AI coding agents: <strong>the discipline of spec-driven development</strong>, and why it matters more, not less, in an agentic world.</p><h3>Part I: Customization — Making Bob Yours</h3><p>Before we dive into Skills and Subagents, let’s first look at Bob’s customization options and the role each one plays.</p><p>Think of it this way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16XS5ezyoHbP-jbMJeJ_kw.png" /></figure><blockquote>These are not interchangeable. They are solving different layers of the same problem.</blockquote><p>A <strong>Custom Mode</strong> is the container. It defines Bob’s entire operating environment for a given context — what tools it can use, what files it can touch, what its role is. A <strong>Skill</strong> is expertise Bob pulls in when a task demands it. A <strong>Subagent</strong> is a child instance of Bob that does focused work in isolation and hands back a summary. <strong>Rules</strong> are always-on guardrails — team conventions and standards that apply regardless of which mode or task is active. And <strong>MCP Servers</strong> are how Bob reaches beyond the codebase entirely, connecting to external platforms, databases, and services through a standard protocol.</p><p>They are designed to stack.</p><h3>Part II: Skills — Reusable Expertise on Demand</h3><h3>What a Skill is</h3><p>A skill is a folder of task-specific instructions that Bob loads into context only when the task matches. The key word is <strong><em>only</em></strong>.</p><p>Bob does not load every skill into every conversation. The base prompt stays lean. Deep expertise arrives on demand.</p><p><strong>Directory structure:</strong></p><pre>.bob/skills/your-skill-name/<br>├── SKILL.md # Required - the main skill file (YAML frontmatter + body)<br>├── scripts/ # Optional - executable code (Python, Bash, etc.)<br>├── references/ # Optional - documentation loaded as needed<br>└── assets/ # Optional - templates, fonts, icons used in output</pre><p>The SKILL.md is the entry point. Everything else is supporting material Bob navigates to only when the task specifically requires it.</p><h4>The anatomy: Progressive Disclosure</h4><p>The design principle behind skills is worth understanding explicitly, because it shapes how you should write them.</p><p><strong>Bob should never load more than it needs.</strong></p><p>Skills implement this through a three-level loading system:</p><p><strong>Level 1 — YAML frontmatter</strong> <em>(always loaded)</em> This is the skill’s identity card. It contains just enough for Bob to know when to activate the skill — without loading any of the actual content yet. Think of it as the skill’s description visible in a catalogue. Keep this under 1,024 characters. It is always in Bob’s context, so every word costs something.</p><p><strong>Level 2 — SKILL.md body</strong> <em>(loaded when relevant)</em> The full instructions and guidance. Bob loads this when it determines the skill matches the current task. This is where your checklists, workflows, output formats, and decision criteria live. Keep this under 5,000 words. It is loaded per task, not per conversation — so it can be detailed, but not sprawling.</p><p><strong>Level 3 — Linked files</strong> <em>(loaded only as needed)</em> <strong>References, scripts, assets. </strong>Bob navigates to these only when the task specifically requires them. Move everything non-essential here. An API guide that Bob might need to consult. Example outputs. A template file. If Bob does not need it for most invocations of the skill, it belongs at Level 3.</p><h4>Scripts: the efficiency multiplier you should not overlook</h4><p>One detail that deserves more attention than it usually gets: the scripts/ folder is not just a convenience. It is a significant efficiency lever.</p><p>When Bob performs a complex task without scripts — reading files, extracting data, applying transformations, generating structured output — it is doing all of that work through reasoning, which burns tokens at every step. Scripts offload that computational work to executable code. Bob invokes the script, gets back a result, and moves on. The reasoning budget is spent on judgment, not mechanics.</p><p>In practice, <strong>the</strong> <strong>scripts accompanying a </strong><strong>SKILL.md can reduce Bob coin usage dramatically</strong> — what would otherwise cost tens or hundreds of coins without scripts becomes a fraction of that with them. For skills that run frequently, or that involve heavy data processing, this is not a marginal saving. It compounds across every invocation.</p><h4>When to reach for a Skill</h4><p>Three signals that a skill is the right tool:</p><ol><li><strong>You have task-specific expertise that should apply consistently:</strong> Code review standards, Security review checklists, Naming conventions, API design principles, testing practices, documentation guidelines, and other instructions you’d otherwise have to repeat in every prompt.</li><li><strong>You have reusable procedures:</strong> A process for converting regulatory documents to executable logic. A workflow for evaluating agent output before deployment. A template for generating API documentation. Package these once and Bob applies them reliably.</li><li><strong>You have knowledge Bob should combine with other features automatically:</strong> Skills work alongside custom modes and subagents. A subagent can use a skill, and a custom mode can define which skills are available or relevant to its workflow.</li></ol><blockquote><strong>A real example: Tax Regulations → Executable Logic</strong></blockquote><blockquote><em>Full skill on </em><a href="https://github.com/ibm-self-serve-assets/Bob-customization-ww-se/blob/master/skills/tax-doc-to-logic/Readme.md"><em>GitHub</em></a><em>.</em></blockquote><p>Hand Bob a tax bulletin a real <a href="https://dam.ldr.la.gov/lawspolicies/RIB%2025-007%20-%20State%20Sales%20Tax%20Rate.pdf">Louisiana Department of Revenue notice</a> about a sales tax rate change and this skill converts it into a decision table, pseudocode, and audit summary. Every rule is extracted with its source clause so engineers and auditors can trace it back to the original document. Cross-references Bob cannot resolve get flagged, not assumed. Output is always stamped as draft for engineer and tax counsel review.</p><p>The same skill works on any jurisdiction. Louisiana, GST from India, a VAT circular from the EU — the rigour is consistent because the skill encodes it, not the prompt.</p><h4>How to set it up</h4><p>Clone the repo and drop the skill folder into your project:</p><pre>git clone https://github.com/ibm-self-serve-assets/Bob-customization-ww-se.git<br>cp -r Bob-customization-ww-se/skills/tax-doc-to-logic .bob/skills/</pre><p>The skill structure that lands in your project:</p><pre>.bob/skills/tax-doc-to-logic/<br>├── SKILL.md                          # Skill definition and workflow<br>├── assets/<br>│   └── decision_table_template.csv   # Column template for decision tables<br>└── references/<br>    └── pseudocode_patterns.md        # Pseudocode conventions and examples</pre><h4>How to run it</h4><p>In a chat with Bob, just upload the PDF and ask naturally:</p><pre>Help me extract all tax rules from the attached @RIB 25-007 - State Sales Tax Rate.pdf<br>and convert them into implementable logic</pre><p>That is it. Bob recognises the task, activates the skill, and does the rest.</p><p>Bob will produce three files every time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ViwKg5_JxEoAsjLPgGHhKA.png" /></figure><p><strong>One important note:</strong> review audit_summary.md first. Resolve all needs_review items with your tax counsel before using any other output. The skill flags ambiguity it never resolves it silently.</p><h3>Part III: Subagents — Letting Bob Delegate</h3><h3>What a Subagent is</h3><p>A subagent is a child instance of Bob with its own isolated context window. The parent Bob spins it up, hands it a focused task, and gets back a summary. The parent conversation never sees all the intermediate work — just the result.</p><pre>~/.bob/agents/<br>├── your-agent-name.md             Scope: Global<br><br>                            # One markdown file per agent<br>.bob/agents/<br>├── your-agent-name.md    	     Scope: Project</pre><p>Subagents are defined as markdown files with YAML frontmatter <strong>that tells Bob two things:</strong> <strong>when to use the subagent, and how the subagent should behave.</strong></p><p>The frontmatter carries the metadata — name, description, and tool access. The body carries the specialized instructions.</p><pre>- - <br>name: your-agent-name<br>description: When and why Bob should invoke this agent. This controls<br> both when the subagent is launched and what context it receives.<br>tools: Read, Glob, Search<br> - -<br>You are a specialist in [domain]. Your job is to [focused task].<br>## Instructions<br>…<br>## Output format<br>Return a concise summary to the parent agent covering: …</pre><h4>Benefits:</h4><p><strong>Focused work.</strong> Each subagent concentrates on one thing without distraction from the broader task. It does not have to hold the full project context in mind while doing deep work on a specific problem.</p><p><strong>A clean context window.</strong> All the messy exploration happens in isolation. The main conversation does not get polluted with intermediate steps, failed attempts, or file contents that are only relevant during investigation.</p><p><strong>Concise results.</strong> The parent gets back only what it needs — a summary, a structured output, a verdict.</p><h4>What makes an effective subagent</h4><p>Not all subagents are equally useful. Effective ones share four characteristics:</p><p><strong>1. A single, bounded responsibility.</strong> A subagent that tries to do too much loses the isolation benefit. The dockerfile generator generates Dockerfiles. The security scanner scans for security issues. They do not overlap. The more specific the responsibility, the more reliably the subagent performs it.</p><p><strong>2. A specific, well-written description.</strong> The description is arguably the most important part of the subagent definition — it controls when the agent is launched and what instructions it receives. Write it to steer both.</p><p><strong>3. Structured output.</strong> Define an explicit output format in the system prompt so the subagent knows when it is done and returns information the main thread can actually use.</p><p><strong>4. Limited tool access.</strong> Only give a subagent the tools it actually needs. Read-only access for research and investigation agents. Bash access for agents that need to run checks. Write and edit access only for agents that should change code.</p><blockquote><strong>A real example: Dockerfile Generator</strong></blockquote><blockquote>Here is a subagent that demonstrates these properties well:</blockquote><pre>---<br>name: dockerfile-generator<br>description: Use this agent when you need to generate, review, or optimize<br>  a Dockerfile for any language or framework. Invoke proactively when the<br>  user mentions Docker, containerization, or building an image.<br>tools: Read, Write, Glob, Bash<br>---<br><br>You are a Docker specialist focused on generating production-ready Dockerfiles.<br><br>When given a project directory or tech stack description:<br>1. Read key project files (package.json, requirements.txt, go.mod, etc.)<br>   to detect the stack automatically<br>2. Choose official slim/alpine base images where possible<br>3. Use multi-stage builds for compiled languages<br>4. Minimize layers — combine RUN commands, clean caches in the same layer<br>5. Always run as a non-root user<br>6. Copy dependency files before source for layer cache efficiency<br>7. Ask clarifying questions only if runtime, entry point, or port are<br>   genuinely ambiguous<br><br>Return: the Dockerfile written to disk, a short decision summary,<br>and a recommended .dockerignore block.</pre><blockquote><strong>Notice what this subagent does: </strong>it reads package.json, requirements.txt, go.mod— whatever is in the project — figures out the stack without being told, makes production decisions (slim base image, multi-stage build, non-root user, layer caching), writes the Dockerfile to disk, and hands back a short summary of the decisions it made. The parent conversation gets the artifact. It never sees the file scanning, the stack detection, or the intermediate reasoning.</blockquote><p>This is the pattern subagents are built for: bounded, file-touching work that produces a clear deliverable and needs to be isolated from the main thread.</p><h4>How to set it up</h4><p>Create the agents folder if it does not exist and add the markdown file:</p><pre>mkdir -p .bob/agents</pre><p>Then create .bob/agents/dockerfile-generator.md with the content above — or clone it directly from the <a href="https://github.com/ibm-self-serve-assets/Bob-customization-ww-se/blob/master/agents/dockerfile-generator.md">repo</a>:</p><pre>cp Bob-customization-ww-se/agents/dockerfile-generator.md .bob/agents/</pre><p>That is the entire setup. No registration, no configuration file to update. Bob discovers agents by reading the .bob/agents/ directory.</p><h4>How to run it</h4><p>Bob invokes the subagent automatically when the description matches — you do not call it explicitly. Just work naturally:</p><pre>I need to containerize this FastAPI service. Can you generate a Dockerfile?</pre><p>Bob reads the description, decides the dockerfile-generator matches, spins it up, and hands back the result. What you get in the main conversation:</p><ul><li>The Dockerfile written to the project root</li><li>A short summary: base image chosen and why, whether multi-stage was used, port, user setup, any caveats</li><li>A recommended .dockerignore block</li></ul><p>You can also invoke it explicitly if you prefer:</p><pre>Use the dockerfile-generator agent to create a Dockerfile for this project.</pre><h4>Subagents versus Skills — when to use which</h4><blockquote>A skill gives Bob knowledge and procedures. A subagent goes off and <em>does the work</em> in its own isolated context. They are not competing — they complement each other. A subagent can invoke a skill as part of its work. The dockerfile generator, for example, could reference a Docker best-practices skill to inform its decisions.</blockquote><h3>Part IV: Agents That Remember — AGENTS.md and /init</h3><h4>The statelessness problem</h4><p>Large language models are stateless. Each new conversation starts with no memory of previous interactions. Without explicit project context, Bob has to rediscover the codebase every time — which is inefficient and impractical at scale.</p><p>This is the problem AGENTS.md solves.</p><h4>Picking up where we left off</h4><p>In the <a href="https://medium.com/@syeda9118/moving-past-the-raw-model-engineering-an-agentic-harness-with-ibm-bob-8b6f5be33cec">first post</a>, we introduced AGENTS.md as Bob’s project memory — the persistent context file that travels across conversations so Bob does not have to rediscover the codebase every time. We also established the principle that every line should be earned: traced back to a real failure, a code review finding, or a hard constraint.</p><p>This section goes further. Not how to create AGENTS.md, but how to make it actually improve over time.</p><h4>A quick recap: /init generates it, you evolve it</h4><p>If you have not set it up yet — run /init in Agent mode. Bob scans the project and generates two things:</p><p><strong>A main AGENTS.md</strong> at the project root with a project overview, directory structure, technology stack, architectural patterns, and development workflows.</p><p><strong>Mode-specific AGENTS.md files</strong> inside .bob/ that extend the main file with context relevant to each mode:</p><blockquote>.bob/rules-agent/AGENTS-agent.md — Agent mode</blockquote><blockquote>.bob/rules-plan/AGENTS-plan.md — Plan mode</blockquote><blockquote>.bob/rules-ask/AGENTS-ask.md — Ask mode</blockquote><h4>Improving AGENTS.md over time</h4><p>The most important thing about AGENTS.md is that it should evolve. It is not a one-time setup artifact. It is a living document that gets sharper as your team and Bob learn from real-world work.</p><p>As we established in the first post, <strong>every line should be earned</strong>. If a rule can be traced back to a production issue, a code review comment, a recurring mistake, or a hard business constraint, it probably belongs. If not, it is likely noise.</p><p><strong>A practical rule of thumb:</strong> when Bob makes the same mistake twice, that’s a signal. The first time is the model. The second time is your workflow failing to learn. Capture the lesson in AGENTS.md so it doesn’t happen again.</p><p>The best additions are specific and grounded in experience. They aren’t style preferences — they’re constraints backed by real incidents, business requirements, or engineering decisions.</p><blockquote>Re-run /init after major project changes, such as adding new modules, migrating frameworks, or onboarding new team members. You can also edit AGENTS.md manually to add context that automated scanning cannot capture, such as business rules, deployment conventions, or architectural decisions that never made it into the codebase.</blockquote><h3>Part V: Managing Context Efficiently</h3><p>As your project and your sessions grow, context management becomes an active discipline rather than a passive concern. Here is what matters in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AKZzx2daknsKYtvW6GEKOA.png" /><figcaption>The mental model that ties these together: <strong>every token Bob spends on noise is a token it is not spending on your actual problem.</strong> Good context management is making sure Bob’s attention is always on what matters.</figcaption></figure><p><strong>Start fresh for new tasks.</strong> This is one of the most common mistakes people make. After spending an hour debugging an authentication issue, don’t ask Bob to switch gears and review code or plan a feature in the same conversation. The previous context can bleed into the new task and reduce response quality.</p><p>Open a new conversation instead. It takes two seconds and gives Bob a clean slate — just like opening a new terminal tab for a different task.</p><p><strong>Be specific about what Bob needs to see.</strong> Bob does not need the whole codebase to fix one function. Tell it exactly which files are relevant. “Look at auth/endpoints.py and config/settings.py&quot; is far more effective than &quot;look at the project.&quot; The more you narrow the scope, the sharper the output.</p><p><strong>Let Skills carry the deep expertise.</strong> This is what skills are designed for. Instead of pasting your coding standards document into every conversation, package it as a skill. Bob loads it only when the task is relevant. The base prompt stays lean. Deep knowledge arrives on demand.</p><p><strong>Delegate heavy work to Subagents.</strong> When a task involves significant exploration — scanning files, running checks, investigating patterns that intermediate work eats context fast. Delegate it to a subagent. The heavy lifting happens in isolation and the parent conversation receives a clean summary.</p><p><strong>Summarize and reset on long sessions.</strong> If you are deep into a long session and Bob starts drifting — repeating itself, losing track of earlier decisions the context is getting crowded. Ask Bob to summarize the key decisions and current state, then start a new conversation with that summary as the opening message.</p><p><strong>Treat .bobignore as context hygiene, not just security.</strong> Blocking node_modules/, build artifacts, and log files with .bobignore is not only about protecting sensitive data. It is about keeping Bob&#39;s attention on what actually matters. Every file Bob can see is a file Bob might pull into context when it is completely irrelevant to the task at hand.</p><p><strong>Custom Modes set the boundaries upfront.</strong> A well-configured mode is doing context management before the conversation even starts. You have already defined Bob’s role, its tools, its file scope. You are not spending the first ten messages re-establishing ground rules. The mode handles that so you can start working immediately.</p><h3>Part VI: The Discipline That Makes Everything Else Work — Spec-Driven Development</h3><p>Everything we have covered — custom modes, skills, subagents, AGENTS.md, context management makes Bob more effective at executing. But there is a question that precedes execution that most teams do not take seriously enough.</p><p><strong>What exactly should Bob be building?</strong></p><p>Markus Eisele put it clearly in his O’Reilly Radar piece, <a href="https://www.oreilly.com/radar/why-ai-coding-agents-still-need-clear-specs/">Why AI Coding Agents Still Need Clear Specs</a>:</p><blockquote>“The real challenge in software engineering has always been specification. Not typing. Not syntax. Not even architecture in the abstract. The hard part was agreeing what should exist, what should never happen, which trade-offs matter, what the system is allowed to forget, and what ‘done’ means when the world is messier than the ticket. Agents don’t remove that problem. They make it more visible.”</blockquote><p>This is not a peripheral concern. It is the central one.</p><p>If agents make the spec problem more visible, Bob gives you concrete places to address it. Here is where that shows up in practice:</p><p><strong>Use Plan mode before Agent mode.</strong> It is not overhead. It is the harness enforcing the discipline of agreeing on what you are building before building it. When Bob asks a clarifying question during planning, that is a gap in the spec surfacing early — exactly where you want it.</p><p><strong>For subagents, the spec is the handoff contract.</strong> When Agent A’s output becomes Agent B’s input, interpretive drift compounds. A loose brief to one subagent becomes a confident wrong answer from the next. Define what each subagent should return before you wire them together.</p><p><strong>AGENTS.md is a living spec.</strong> Every constraint you add — service layer conventions, architectural boundaries, lessons from past mistakes is specification that shapes every future session. The same discipline applies: earn every line, make it traceable, and treat Bob’s recurring mistakes as gaps in the spec, not failures of the model.</p><p>For further depth, <strong>Bill</strong> <strong>Higgins and Gabe Goodhart’s</strong> talks on <a href="https://www.linkedin.com/posts/billhiggins1_building-building-activity-7452468672901832704-Z2D4?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAABlxeyQBmxF7plsCES1WddjWf9IR7VM1-Ig">spec-driven development</a> with AI agents are worth your time alongside Markus’s piece.</p><h3>What Comes Next</h3><p>The harness we have described in this post is not a finished product. It is a starting point that gets sharper with use.</p><p>Every time Bob makes a mistake you capture in AGENTS.md, the harness gets stronger. Every skill you build from a repeated task is time you never spend on that task again. Every subagent you define is complexity you remove from your main conversation. Every mode you configure is a context you never have to re-establish.</p><p>The teams that pull ahead with AI coding agents are not the ones with the best prompts. They are the ones treating the harness as a first-class engineering artifact — something you build, maintain, and improve alongside the codebase it serves.</p><p>Start small. Pick the one thing Bob gets wrong most often in your project and put it in AGENTS.md today. That single line, earned by a real failure, is worth more than a hundred tokens of prompt engineering.</p><p>The harness compounds. Build it deliberately.</p><p><em>Happy building.</em></p><p><em>Syeda</em></p><h4>Further Reading</h4><ul><li><strong>Markus Eisele</strong>, <a href="https://www.oreilly.com/radar/why-ai-coding-agents-still-need-clear-specs/">Why AI Coding Agents Still Need Clear Specs</a> — essential reading on the cost structure of specification in agentic work</li><li>IBM Bob Documentation, <a href="https://bob.ibm.com/docs/ide/getting-started/tutorials/start-a-project">Start a project with /init and AGENTS.md</a> — the official guide to persistent project context</li><li><strong>Bill</strong> <strong>Higgins and Gabe Goodhart’s</strong> talks on <a href="https://www.linkedin.com/posts/billhiggins1_building-building-activity-7452468672901832704-Z2D4?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAABlxeyQBmxF7plsCES1WddjWf9IR7VM1-Ig">spec-driven development</a> with AI agents are worth your time</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7f47566c55e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving Past the Raw Model: Engineering an Agentic Harness with IBM Bob]]></title>
            <link>https://medium.com/@syeda9118/moving-past-the-raw-model-engineering-an-agentic-harness-with-ibm-bob-8b6f5be33cec?source=rss-bc7fd0472dd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b6f5be33cec</guid>
            <category><![CDATA[ibm-bob]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <category><![CDATA[agentic-coding]]></category>
            <category><![CDATA[ai-coding-agent]]></category>
            <dc:creator><![CDATA[Syeda Ameena]]></dc:creator>
            <pubDate>Tue, 02 Jun 2026 01:58:18 GMT</pubDate>
            <atom:updated>2026-06-03T16:19:52.757Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://bob.ibm.com/"><strong>IBM Bob</strong></a> is an agentic development partner designed to help developers work across the software development lifecycle, from understanding large codebases and implementing new features to testing, modernization, and deployment.</p><p>As teams begin using Bob for increasingly complex engineering tasks, they quickly discover something compelling. Bob can navigate massive codebases, accelerate modernization efforts, and assist with complex engineering workflows that would otherwise require significant manual effort. Yet, that same agent can sometimes produce dramatically different outcomes when given seemingly similar tasks.</p><p>And what is the usual reaction?</p><p><strong>We blame the model.</strong> And with Bob, that reaction can feel even stronger because the model is not always something the user directly selects. When the system uses dynamic model allocation, users do not always see or choose the underlying model directly, which can make it tempting to focus on the model as the primary explanation for an agent’s behavior.</p><p><strong>But that is often the wrong conclusion.</strong></p><p>Instead of asking, “Why did Bob use that model?”, a more useful question is:</p><p><strong>“What harness was Bob working inside?”</strong></p><p>Because when an agent behaves differently than expected, the root cause is often not the model itself.</p><blockquote><strong>The real issue is usually the harness.</strong></blockquote><p><strong>“Harness”</strong> is one of those terms that suddenly seems to be everywhere in AI conversations.</p><p>The reason is simple: teams are discovering that better results often come from improving the system around the model rather than changing the model itself.</p><h4><strong>So before we talk about how to get more out of IBM Bob, let’s first understand what a harness is.</strong></h4><p>At its core, a Large Language Model (LLM) provides the raw reasoning intelligence. But it is still just a text-in, text-out engine.</p><p>On its own, a model cannot open a file, run a test, access enterprise tools, or correct its own code.</p><p>That’s where the harness comes in.</p><p><strong>A harness is the software infrastructure built around the model that gives it a body, tools, memory, safety boundaries, and feedback loops so it can behave like an autonomous coding engineer rather than a chatbot.</strong></p><p>The industry often summarizes this idea with a simple equation:</p><p><strong>Agent = Model + Harness</strong></p><p>The model provides the intelligence. The harness provides everything else.</p><p>And that distinction is becoming increasingly important.</p><p>As AI models continue to improve, many teams are discovering that the difference between a frustrating AI experience and a productive one often has less to do with the model itself and more to do with the quality of the harness around it.</p><p>Or put another way:</p><blockquote><strong><em>The model is the engine. The harness is the car.</em></strong></blockquote><p>Without the steering wheel, tires, brakes, and dashboard provided by the harness, even the most powerful engine cannot take you very far.</p><h4>What Makes a Good Harness?</h4><p>Not all harnesses are created equal.</p><p>A good harness doesn’t just give an agent access to tools. It actively shapes how the agent behaves.</p><p>Software pioneer Martin Fowler describes this through two complementary mechanisms: <strong>Feedforward Controls</strong> and <strong>Feedback Controls</strong>.</p><p><strong>Feedforward controls</strong> provide guidance before the agent takes action. Think of coding standards, architectural boundaries, project conventions, and custom instructions that help the agent start in the right direction.</p><p><strong>Feedback controls</strong> provide signals after the agent takes action. Think of test failures, linting errors, code reviews, security checks, and findings that help the agent identify mistakes and improve its output.</p><p>In simple terms, a good harness helps an agent avoid mistakes before they happen and recover from mistakes when they do.</p><p>The quality of a coding agent is therefore determined not just by the model it uses, but by the harness surrounding it.</p><p>And this is where IBM Bob’s design becomes interesting.</p><h3>How Bob Helps You Engineer a Harness</h3><blockquote><em>“The model matters. But the harness often matters more.”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OgG5YKhnT4KM8r5LRUChkg.png" /></figure><p>The easiest way to think about Bob is that the foundation models sit at the center, but they are not what most users interact with directly.</p><p>Bob wraps those models with an agent layer and a configurable harness that provides guidance, memory, capabilities, feedback, and scale.</p><p>Together, these components transform a raw model into a productive coding agent capable of operating within your team’s workflows and standards.</p><p>So what do these harness components actually do, and how can they help you get better results from Bob?</p><h4><strong>Let’s look at the pillars that make up Bob’s agent harness.</strong></h4><p>You’ll notice something interesting about these pillars.</p><p><strong><em>None of them involve changing the underlying model.</em></strong></p><p>That’s intentional.</p><p>When an AI coding agent struggles, most teams immediately focus on the model. They ask whether they need a larger context window, a newer foundation model, or the next major release.</p><p>But many of the biggest improvements in agent performance come from somewhere else entirely.</p><p>They come from the harness.</p><p>A better harness gives the agent better guidance. It gives the agent access to the right tools. It helps the agent remember important project context. It creates feedback loops that catch mistakes early and encourage self-correction.</p><p>In other words, it helps the agent behave more like a software engineer and less like a chatbot.</p><blockquote><em>The six pillars below are examples of how IBM Bob allows teams to build that harness. Some guide Bob before it acts. Others help Bob learn from its actions. Together, they shape how Bob operates inside your development environment.</em></blockquote><ol><li><strong>Guidance: Custom Modes, Rules, and Premium Modes</strong></li></ol><p>Most agent mistakes happen before the first line of code is written.</p><p>The first job of a harness is to guide the agent before it starts generating code.</p><p>Think of it as giving Bob context before it acts.</p><p>In Bob, this guidance can come from multiple places. <strong>Custom Modes</strong> define specialized personas and workflows. <strong>Rules</strong> capture repository-specific conventions, architectural constraints, and team standards. <strong>Bob Premium Modes</strong> provide institutional and domain-specific knowledge with curated guidance for environments such as IBM Z, IBM i, and Java.</p><p>Together, these mechanisms shape how Bob approaches a task before it generates a single line of code.</p><p><strong>For example,</strong> in the screenshot below, I created a custom <strong>watsonx Orchestrate</strong> (<strong>wxO) Agent Architect</strong> mode. Instead of relying on a generic coding assistant, this mode tells Bob exactly how to behave when working with watsonx Orchestrate.</p><p>If you look closely, the mode isn’t simply changing Bob’s persona. It is defining:</p><ul><li>Which MCP servers Bob should use</li><li>Which documentation sources Bob should consult first</li><li>Which architectural constraints must be followed</li><li>Which evaluation loops must run before deployment</li><li>Which commands are blocked until validation succeeds</li></ul><p><strong>That’s not prompt engineering — That’s harness engineering.</strong></p><blockquote><em>Notice something important: I didn’t change the model. I changed the harness.</em></blockquote><p>Now every time Bob works on a watsonx Orchestrate agent, it automatically follows those instructions without me repeating them in every prompt.</p><p>This is a classic example of a <strong>feedforward control</strong>: guiding the agent before mistakes happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GvOzyfmRd5anHKCIZePaag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7ch1B5k5XL7jJuaki8gVw.png" /></figure><p><strong>2. Memory: AGENTS.md</strong></p><p>If Custom Modes and Rules help Bob start in the right direction, AGENTS.md helps Bob carry project knowledge forward across conversations.</p><blockquote><em>One of the ideas I borrowed from Addy Osmani’s writing on </em><a href="https://addyosmani.com/blog/agent-harness-engineering/"><em>Harness Engineering</em></a><em> is that </em><strong><em>agent mistakes should be treated as signals, not accidents.</em></strong></blockquote><p>When Bob makes the same mistake twice, the problem is no longer the model. The problem is that the harness failed to learn from the first occurrence.</p><p>AGENTS.md acts as the project’s living memory. It captures lessons learned, team conventions, and hard-earned engineering knowledge so Bob can apply them in future sessions.</p><blockquote><em>There is one important caveat. AGENTS.md works best when it stays focused. As Addy Osmani points out in his writing on Harness Engineering, every instruction competes for the agent’s attention.</em></blockquote><blockquote><em>A good </em><a href="https://bob.ibm.com/docs/ide/getting-started/tutorials/start-a-project#explore-agentsmd-files"><em>AGENTS.md</em></a><em> is not a style guide or a dumping ground for every team preference. It should contain the conventions, constraints, and lessons that genuinely matter.</em></blockquote><blockquote><em>A useful rule of thumb is to earn every line. </em><strong><em>If a rule can be traced back to a production issue, a code review comment, a recurring mistake, or a hard business constraint, it probably belongs. If not, it is likely noise.</em></strong></blockquote><p>The best AGENTS.md files are not written all at once. They evolve as your team and Bob learn from real-world failures.</p><p>For example, if Bob repeatedly suggests direct database access instead of using your team’s service layer, you could add:</p><pre>Always use CustomerLookupService.<br>Never access CustomerDB directly.</pre><p>The goal isn’t to create a massive document. The goal is to capture lessons that matter.</p><ul><li><strong>Keep AGENTS.md short</strong></li><li><strong>Every line should be earned by a real failure or constraint</strong></li></ul><p>Over time, AGENTS.md becomes a reflection of your team’s experience, ensuring that every mistake makes the harness a little stronger.</p><p><strong>Good memory prevents the same mistake from happening twice.</strong></p><h4>3. Capabilities: Skills and MCP Servers</h4><p>A model can only reason about what it can see. Enterprise work often requires information that lives outside the codebase.</p><p>This is where Bob’s Skills and MCP Servers come in.</p><p>MCP Servers extend Bob’s reach beyond the local repository, allowing it to interact with documentation, internal systems, APIs, and enterprise tools. Skills, on the other hand, help package repeatable workflows and expertise into reusable capabilities.</p><p>For example, suppose your team frequently builds watsonx Orchestrate agents and must run a <strong>Quick Evaluation of Agents and Tools</strong> before deployment.</p><p>Rather than relying on Bob to rediscover the process every time.</p><blockquote><em>You could create a skill that teaches Bob how to:</em></blockquote><ul><li>Run the wxO evaluation workflow</li><li>Interpret evaluation metrics</li><li>Identify schema mismatches and hallucinations</li><li>Review evaluation output files</li><li>Recommend fixes before deployment</li></ul><p>Now Bob is not just generating code. It is applying a workflow that your team has already validated.</p><p>Similarly, in the <strong>wxO Agent Architect example</strong>, Bob is instructed to consult documentation through the <strong>wxO-docs MCP server</strong> before authoring agents. Instead of relying solely on model knowledge, Bob can retrieve information directly from the source.</p><blockquote>The result is simple: less guessing and fewer hallucinations.</blockquote><p>The more relevant context and capabilities you give the agent, the less it needs to make assumptions.</p><p><strong>4. Feedback: Bob Findings</strong></p><p>Guidance helps Bob start correctly. Feedback helps Bob recover when things go wrong.</p><p>Martin Fowler describes this as the second half of a good harness: <strong>feedback controls</strong>.</p><p>If feedforward controls guide the agent before it acts, feedback controls evaluate the results after it acts.</p><p>One example is <strong>Bob Findings</strong>.</p><p>In the screenshot below, Bob automatically surfaced findings from the active Python file, including:</p><ul><li>Security concerns such as exposed secrets</li><li>Refactoring opportunities</li><li>Code quality suggestions</li></ul><p><strong>Nobody explicitly asked Bob to review the code — The harness did it automatically.</strong></p><p>Another example is evaluation loops. In the wxO Agent Architect mode, Bob is instructed to run evaluations, analyze failures, fix issues, and rerun the evaluation before deployment.</p><p>This creates a self-correction loop.</p><p>Instead of generating code and hoping for the best, the harness provides signals that help Bob improve its own work.</p><blockquote>Better feedback enables self-correction.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltBSfXw67YXdtWj-q4A7Aw.png" /></figure><p><strong>5. Scale: Subagents</strong></p><p>As tasks become larger, context becomes harder to manage.</p><p>A single agent can quickly become overwhelmed when working across dozens of files, multiple services, or long-running workflows.</p><p>This is where Subagents become useful.</p><p>One of the emerging patterns in agent engineering is to delegate specialized work to focused agents instead of forcing a single agent to keep everything in its context window.</p><blockquote><em>A good example is </em><strong><em>Bob’s Java Modernization workflow</em></strong><em>.</em></blockquote><p>During a modernization or refactoring effort, Bob may identify prerequisite issues that need to be resolved before the main transformation can continue. Rather than abandoning the task or trying to manage everything in a single conversation, Bob can spawn specialized subagents to investigate and resolve those issues in parallel.</p><p>For example, a subagent might:</p><ul><li>Analyze missing prerequisites</li><li>Fix dependency or configuration issues</li><li>Validate migration requirements</li><li>Resolve code quality findings</li><li>Prepare the codebase for the next modernization step</li></ul><p>Meanwhile, the primary agent can continue focusing on the larger modernization workflow. The result is faster execution, better task separation, and less context overload.</p><p>While this example comes from Java modernization, the pattern applies much more broadly. As agents take on increasingly complex engineering tasks, they need ways to break work into smaller, focused units that can execute independently.</p><p><strong>Subagents provide exactly that mechanism:</strong> They help agents scale beyond a single context window, keep reasoning focused on the task at hand, and enable parallel execution of work that would otherwise become a bottleneck.</p><h3>Conclusion</h3><p>Let’s come back to where we started.</p><p>When Bob hits a wall, the instinct is often to blame the model.</p><p>Sometimes the model is the limitation. But more often, <strong>the issue is the harness around it</strong>.</p><blockquote><em>A raw model is just a text-in, text-out engine. What turns it into an effective coding agent is everything wrapped around it: guidance, memory, capabilities, feedback, and scale.</em></blockquote><p>That is why harness engineering has become such an important conversation in the AI community.</p><p>The future of enterprise AI will not be determined solely by who has access to the best model. It will increasingly be shaped by the systems built around those models — the mechanisms that provide context, memory, governance, capabilities, and feedback.</p><p><strong>Models provide intelligence. Harnesses determine how that intelligence is applied.</strong></p><p>The most successful teams are not simply waiting for the next model release. They are actively shaping how their agents behave by engineering better harnesses.</p><p><strong>IBM Bob</strong> already provides many of these building blocks through Custom Modes, Rules, AGENTS.md, Skills, MCP Servers, Bob Findings, and increasingly, Subagent-based workflows.</p><p>As Bob continues to evolve, additional harness capabilities such as user-configurable Hooks will further strengthen the enforcement and governance layers available to teams.</p><p>The opportunity is not just to use the coding agent, but to continuously engineer the harness around it.</p><p>Because the gap between what today’s models can do and what we often see them doing in production is frequently <strong>not a model gap — It’s a harness gap.</strong></p><h3>Acknowledgements &amp; Further Reading</h3><p>This article was inspired by several excellent resources that helped shape my understanding of agent harnesses and harness engineering:</p><ol><li>Addy Osmani’s article on <a href="https://addyosmani.com/blog/agent-harness-engineering/">Harness Engineering</a>, which influenced many of the ideas around treating agent failures as signals, building feedback loops, and improving the system around the model rather than waiting for the next model release.</li><li><a href="https://huggingface.co/blog/agent-glossary#:~:text=The%20model%20is%20the%20LLM,answers%20one%20prompt%20and%20stops.">Hugging Face’s Agent Glossary</a>, which provides a practical introduction to agents, tools, memory, and the broader agent ecosystem.</li><li>And I would highly recommend watching — <a href="https://youtu.be/C_GG5g38vLU?si=IbSrVkTwZygt6ku-">Harnesses in AI: A Deep Dive — Tejas Kumar, IBM</a></li></ol><p>While this article focuses specifically on IBM Bob, these resources are valuable regardless of which coding agent or AI platform you use.</p><p>Happy Learning!</p><p>Syeda.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b6f5be33cec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[InstructLab  –  “Ever imagined the ease of tuning pre-trained LLMs?]]></title>
            <link>https://medium.com/@syeda9118/instructlab-ever-imagined-the-ease-of-tuning-pre-trained-llms-3331ccea8d88?source=rss-bc7fd0472dd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3331ccea8d88</guid>
            <dc:creator><![CDATA[Syeda Ameena]]></dc:creator>
            <pubDate>Sat, 18 May 2024 02:58:17 GMT</pubDate>
            <atom:updated>2024-05-28T15:42:04.956Z</atom:updated>
            <content:encoded><![CDATA[<h3>InstructLab – “Ever imagined the ease of tuning pre-trained LLMs? InstructLab makes it a reality. Let’s delve into how it sets itself apart from other model tuning methods.”</h3><h3><strong>What is InstructLab ?</strong></h3><p>Before we delve into what InstructLab is and what it does, let’s take a moment to understand why it’s needed ?</p><blockquote><em>Large Language Models (LLMs) have the ability to fuel a diverse array of practical applications, like having a virtual assistant that can chat with you, help you with coding, or even generate content for your next marketing email. That’s the power of Large Language Models (LLMs).</em></blockquote><p>Sure, it’s impressive to see these models effortlessly generate coherent text or provide insightful responses and offer insightful solutions to business challenges.</p><p><strong>💡 But here’s the catch</strong>: while LLMs offer a plethora of possibilities, there are boundaries to what you can do with them. The real complexity lies beneath the surface, in the intricate web of computations required to bring these models to life for specific business needs.</p><blockquote><strong><em>AI practitioners often find themselves needing to tweak pre-trained LLMs to fit specific business needs, but there are limits to how much you can modify these models.</em></strong></blockquote><p>For instance, let’s take a closer look at the Llama2–70B model — a large language model released by <a href="http://meta.ai/">Meta.ai</a>.</p><p><strong>The real computational challenge arises when attempting to adjust these parameters. Given that LlAMA2–70B is an open-source model, we know quite a bit about how it was trained because Meta released that information in paper. </strong>So these are some of the numbers of what’s <strong>involved.</strong> You basically take a chunk of the internet that is roughly, you should be <strong>thinking, 10 terabytes of text.</strong> This typically comes from like a crawl of the internet. So just imagine just collecting tons of text from all kinds of different websites and collecting it together. So you take a large chunk of internet, then you procure a GPU cluster. And these are very specialised computers intended for very heavy computational workloads like training of neural networks.</p><p>You need about <strong>6,000 GPUs and you would run this for about 12 days to get a Llama2 — 70B. And this would cost you about $2 million</strong>.</p><blockquote><strong>We’re familiar with the fact that fine-tuning pre-trained LLMs on downstream datasets leads to significant performance improvements compared to using them out-of-the-box (for example, in zero-shot inference).</strong></blockquote><blockquote><strong>However, we’re also aware that this process presents challenges such as being time-consuming and computationally super expensive.</strong></blockquote><ul><li>As you could see, training LlAMA2–70B, for instance, comes with a hefty price tag of $2 million and requires a staggering 6,000 GPUs.</li><li>As LLMs continue to grow in size, the feasibility of fully fine-tuning them on consumer hardware becomes increasingly difficult. Additionally, the cost of storing and deploying fine-tuned models independently for each downstream task becomes super expensive, as these models maintain the same size as the original retrained model.</li><li>And LLM refinements typically required large amounts of human-generated data, which can be time-consuming and again gets expensive.</li></ul><blockquote><em>This is where </em><strong><em>InstructLab</em></strong><em> steps in, providing a solution to the challenges faced by AI practitioners in fine-tuning and deploying LLMs for specific business needs.</em></blockquote><p><strong>InstructLab takes a different approach to overcome these limitations.</strong> It can improve a Large Language Model (LLM) using much less human-generated information and fewer computing resources than what’s typically needed to retrain a model. Moreover, it allows ongoing contributions from users to continually enhance the model. Additionally, it’s worth noting that InstructLab is an open-source project, meaning its code and development are accessible to the public for collaboration and improvement of the models.</p><blockquote>💡 <strong>InstructLab is</strong> <em>Named after </em><strong><em>IBM Research’s work on Large-scale Alignment for chatBots (LAB)</em></strong><em>, InstructLab is built upon this methodology. Described in a 2024 research paper by members of the MIT-IBM Watson AI Lab and IBM Research, the LAB method serves as the foundation for InstructLab.</em></blockquote><h3>Now, let’s explore what exactly InstructLab entails.</h3><p>InstructLab is an open-source AI project<strong> developed by IBM and Red Hat, </strong>designed to enhance Large Language Models (LLMs) through community contributions. It streamlines the process of training LLMs by facilitating the submission of skills and knowledge from the community.</p><h3>How does it work ?</h3><p>It uses a combination of <strong>Taxonomy-driven data curation approach + Large-scale synthetic data generation + Multi-phased instruction-tuning method.</strong></p><p>Let’s examine this process carefully, breaking it down step by step.</p><blockquote><strong>1 .Taxonomy-driven data curation approach:</strong> In simple terms, a taxonomy is like a tree structure that organizes things into categories and subcategories. In our context, taxonomy classifies data samples into smaller groups based on different tasks.</blockquote><p>It has three main branches: <strong>knowledge, foundational skills, and compositional skills.</strong></p><p>Each branch is further divided into more specific levels where tasks are defined.</p><p>The tasks are represented as leaves on the branches and are illustrated with manually written instruction-response pairs. This structuring makes it easy to identify missing tasks and add them to the training data. New tasks can be added to the taxonomy by creating a new category under the appropriate branch.</p><ol><li><strong>Knowledge</strong>: This is like a big branch in our tree of information. We divide it into different types of documents, like textbooks or technical manuals, and then into specific topics like finance or statistics. Each topic has its own set of documents and sample questions and answers. This helps us make sure we’re using the right documents for our training, and we only choose ones that we have permission to use.</li><li><strong>Foundational skills</strong>: These are the basic skills that our model needs to learn first, like math, coding, language skills, and reasoning. To teach these skills, we use datasets that are available to the public. This helps our model get ready to learn more complex skills later on.</li><li><strong>Compositional skills:</strong> These are skills that combine knowledge and foundational skills to answer complex questions. For example, if our model needs to write an email about a company’s performance, it needs to understand financial terms, do math, and know how to write a formal email. These skills work together to help our model tackle tough tasks.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPyoPmh611ldx9bNprFw0A.png" /></figure><blockquote><strong>2. Large-scale synthetic data generation:</strong></blockquote><p><strong>For example</strong>, if the task is to remove personally identifiable information (PII) from a text, we would provide a few examples of how to do that. Then, we use a model to generate more examples based on the initial samples we provided about removing PII. However, it’s important to note that synthetic data can vary in quality. To address this, the LAB method includes an automated step to refine the generated examples, ensuring they are accurate and safe to use.</p><blockquote>Discover How Instruction Generation and Evaluation Work in InstructLab</blockquote><p><strong>a. Instruction Generation:</strong></p><p>In the first stage, the teacher model serves as a question generator, using a specialized prompt to leverage its extensive knowledge and create diverse questions. By iterating through each leaf node of a taxonomy, the teacher generates queries that adhere to specific principles. This method thoroughly explores the targeted domain, ensuring the comprehensiveness of the generated content.</p><p><strong>b. Evaluating Synthetic Instructions:</strong></p><p>Next, the teacher model acts as an instruction evaluator. Using targeted prompts, it filters out questions that do not meet predefined criteria, such as relevance to the domain, potential harm, or feasibility within a language model’s answering capabilities. This stage ensures that only high-quality, contextually appropriate questions progress further in the process.</p><p><strong>c. Generating Responses:</strong></p><p>In this stage, the teacher model functions as a response generator, adopting dual personas for precision and creativity, guided by distinct prompts. This tailored approach allows the generation of both creative responses for domains like writing and role-play, and precise answers for STEM and data extraction. The response style is aligned to human expectations using principles and seed examples from the leaf nodes.</p><p><strong>d. Evaluating the Synthetic Instruction-Response Pair:</strong></p><p>The final stage involves a meticulous process to filter and select high-quality instruction and response pairs. The teacher model evaluates each sample using a 3-point rating system. It filters out responses that are incorrect, irrelevant, or deviate from the provided principles. This ensures the quality and relevance of the training dataset are enhanced for the student model.</p><p>This helps maintain the integrity and reliability of the synthetic data generated for training purposes.</p><blockquote><strong>3. Multi-phased instruct-tuning framework:</strong> Finally, the model undergoes retraining using the synthetic data.</blockquote><p>InstructLab employs a multi-phase tuning framework that serves two vital purposes:</p><p>(i) <strong>Ensuring Diversity and Quality in Data Generation:</strong></p><p>This phase concentrates on creating a dataset for instruction-tuning that is both diverse and high in quality. By utilizing multiple phases, the method captures a broad spectrum of scenarios, ensuring that the generated data accurately represents real-world situations. This diversity is crucial for effectively training the model, enabling it to tackle various tasks with precision and dependability.</p><p>(ii) <strong>Ensuring Training Stability and Preventing Catastrophic Forgetting:</strong></p><p>Another crucial aspect of the method is its focus on maintaining training stability and preventing catastrophic forgetting. <strong>Catastrophic forgetting occurs when a model loses previously learned information upon learning new data</strong>. <strong><em>To counter this, the method integrates replay buffers</em></strong><em>,</em> acting as a memory bank for the model. By revisiting and retraining on earlier data periodically, the model retains its knowledge while acquiring new skills, ensuring stable and consistent performance over time.</p><p><strong>Why Does Catastrophic Forgetting Occur?</strong></p><p>Neural networks are typically optimized for specific tasks by adjusting their weights through back-propagation. When trained on a new task, the model’s weights are readjusted to fit the new task’s data. Improper management of these adjustments can overwrite information from previous tasks, leading to catastrophic forgetting.</p><p><strong>Illustrative Example of Catastrophic Forgetting</strong></p><p>Consider training a model to recognize fruits from images. Initially, it learns to identify apples and performs well. Next, you train the same model to recognize oranges. During this new training, adjustments to improve orange recognition may degrade the model’s ability to recognize apples, despite its prior proficiency.</p><blockquote>Here’s a simple pictorial representation to illustrate the scenario where a model is trained to recognize fruits and experiences catastrophic forgetting when trained on a new fruit:</blockquote><p><strong>1 .Initial Training: Recognizing Apples</strong></p><ul><li><strong>Before Training:</strong> No knowledge of fruits.</li><li><strong>After Training:</strong> High proficiency in recognizing apples.</li></ul><p><strong>2 .Subsequent Training: Recognizing Oranges</strong></p><ul><li><strong>Before Training:</strong> High proficiency in recognizing apples.</li><li><strong>After Training:</strong> High proficiency in recognizing oranges but degraded proficiency in recognizing apples.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6XSEwaFgcODz9cvpisdwaA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iM4GtYLFKMSXzX4bB1SdbQ.png" /></figure><p><strong>Replay Buffer Mechanism</strong></p><p>InstructLab employs a <strong>replay buffer</strong> mechanism during model training to address catastrophic forgetting, particularly in scenarios with sequential training phases. Here’s how it works:</p><p><strong>What is a Replay Buffer?</strong></p><p>A replay buffer acts as a storage system, retaining a subset of data samples from earlier training phases. As the model progresses to new training phases, it periodically revisits and trains on this stored data alongside new data. This helps the model maintain knowledge from earlier phases while acquiring new information.</p><p><strong>How Does It Work?</strong></p><ol><li><strong>Storing Data:</strong> Data samples from initial training phases are saved into the replay buffer.</li><li><strong>Replaying Data:</strong> As the model advances to subsequent phases, it trains on new data while periodically revisiting and retraining on data stored in the replay buffer. This process reinforces earlier knowledge while incorporating new skills.</li></ol><h4>Now that we’ve gained insight into the challenges and understand why InstructLab employs multi-phase tuning, let’s delve deeper into InstructLab’s Multi-phase Tuning Framework.</h4><p><strong>Phase 1: Knowledge Tuning</strong></p><p>The knowledge-tuning phase is the initial step in the LAB training process, essential for enhancing the model’s foundational understanding. It consists of two steps:</p><p><strong>Step 1: Short Response Training</strong></p><ul><li><strong>Data:</strong> Training on samples with short responses from the knowledge and foundational skills branches.</li><li><strong>Replay Buffer:</strong> Not applicable in this step.</li></ul><p><strong>Step 2: Long Response Training</strong></p><ul><li><strong>Data:</strong> Training on samples with longer responses, ensuring the model can handle complex information.</li><li><strong>Replay Buffer:</strong> Data from Step 1 is included to prevent forgetting earlier learning.</li></ul><p><strong>Phase 2: Skill Tuning</strong></p><p>After completing knowledge tuning, the model proceeds to skill tuning:</p><ul><li><strong>Data:</strong> Training on samples from the compositional skills branch, integrating multiple foundational skills.</li><li><strong>Replay Buffer:</strong> Includes data from both knowledge-tuning steps to mitigate catastrophic forgetting.</li></ul><blockquote><em>During intermediate phases, the selection of the optimal model checkpoint is based on benchmark assessments. Specifically, it utilizes the MMLU benchmark (Hendrycks et al., 2020) during the knowledge-tuning phase and the MT-bench (Zheng et al., 2024) during the skills-tuning phase. - Source: </em><a href="https://arxiv.org/pdf/2403.01081">https://arxiv.org/pdf/2403.01081</a></blockquote><h3><strong>Well, what if contributing to LLM’s was within everyone’s reach?</strong></h3><p>To contribute to a model using InstructLab, The easiest approach resembles a pull request in software development, and it entails crafting a “skill.” This process requires two essential components: a <strong>qna.YAML file</strong> and a <strong>separate text file</strong> providing attribution for the content, refer the below example. The YAML file contains structured data in a format called YAML, which helps organize the information for the model. Think of it as a straightforward text file with structured formatting.</p><p>InstructLab utilizes a selection of these skills to generate a more extensive set of synthetic data related to the provided examples.</p><blockquote>However, it’s important to note a constraint: there’s a limit to the amount of content that the model can process effectively. <strong>As a result, contributors should ensure that the question and answer pairs in the qna.yaml file don’t exceed approximately 2300 words</strong>. By adhering to this limit, contributors can help maintain the quality and efficiency of the model’s training process.</blockquote><h4><strong>For instance, let’s consider an example.</strong></h4><p>Let’s start with a simple example. We’ll enhance the model to provide meeting summaries by training it with InstructLab. Using InstructLab, we’ll generate, train, deploy, and evaluate the chat model using the provided qna.yaml file for generating meeting transcript summary data.</p><p>For the initial setup process, please follow the step-by-step instructions outlined in the following link: <a href="https://github.com/instructlab/instructlab/blob/main/README.md">https://github.com/instructlab/instructlab/blob/main/README.md</a></p><ol><li><strong>Contribute Knowledge or Skills:</strong></li></ol><ul><li>Create a qna.yaml file containing the meeting transcript summary data, including the minutes of the meeting, attendees, agenda, discussions, and action items.</li></ul><pre>created_by: IBM Research<br>seed_examples:<br>- answer: &#39;**Minutes of Meeting**<br><br><br>    **Date:** [Insert Date]<br><br><br>    **Time:** [Insert Time]<br><br><br>    **Location:** [Insert Location]<br><br><br>    **Present:**<br><br>    - Ms. Patel<br><br>    - Mr. Thompson<br><br>    - Ms. Rodriguez<br><br>    - Mr. Kim<br><br><br>    **Agenda: Employee Engagement Initiatives**<br><br><br>    **Opening Remarks:**<br><br>    Ms. Patel welcomed everyone to the meeting, emphasizing the importance of enhancing<br>    employee engagement.<br><br><br>    **Recognition of Employee Achievements:**<br><br>    Mr. Thompson suggested implementing regular recognition of employee achievements<br>    through a monthly shout-out in the company newsletter or a quarterly awards ceremony.<br><br><br>    **Personal Development:**<br><br>    Ms. Rodriguez proposed offering workshops, training sessions, and mentorship programs<br>    to help employees feel valued and grow professionally.<br><br><br>    **Cross-Departmental Collaboration:**<br><br>    Mr. Kim suggested creating opportunities for cross-departmental collaboration<br>    to foster innovation and help employees understand the bigger picture of the organization.<br><br><br>    **Suggestion Box System:**<br><br>    Ms. Patel proposed implementing a suggestion box system for anonymous sharing<br>    of ideas or concerns, promoting open communication.<br><br><br>    **Team-Building Activities:**<br><br>    Ms. Rodriguez suggested organizing team-building activities outside of work hours,<br>    such as potlucks, game nights, or volunteer events, to strengthen relationships<br>    and foster a sense of community.<br><br><br>    **Work-Life Balance:**<br><br>    Mr. Kim emphasized the importance of work-life balance, encouraging flexible work<br>    hours and remote work options.<br><br><br>    **Wellness Program:**<br><br>    Mr. Thompson proposed creating a wellness program that includes fitness challenges,<br>    meditation sessions, or stress management workshops to promote employee health<br>    and happiness.<br><br><br>    **Employee Resource Group:**<br><br>    Mr. Kim suggested establishing an Employee Resource Group to support diversity<br>    and inclusion initiatives, helping employees feel more connected and included.<br><br><br>    **Action Plan:**<br><br>    The group agreed to evaluate the suggestions and create an action plan, reconvening<br>    the following week to discuss the next steps.<br><br><br>    **Closing Remarks:**<br><br>    Ms. Patel thanked everyone for their valuable input, expressing excitement about<br>    implementing the new employee engagement initiatives.<br><br><br>    **Next Meeting:**<br><br>    [Insert Date and Time for the next meeting]<br><br><br>    **Prepared by:** [Insert Name]<br><br>    **Approved by:** [Insert Name]&#39;<br>  context: &#39;*Ms. Patel:* Good morning, everyone. Today, we&#39;&#39;re here to discuss ways<br>    to enhance employee engagement. I&#39;&#39;d love to hear your ideas.<br><br><br>    *Mr. Thompson:* I think regular recognition of employee achievements could be<br>    a great start. A monthly shout-out in our company newsletter or a quarterly awards<br>    ceremony could work wonders.<br><br><br>    *Ms. Rodriguez:* I agree, but let&#39;&#39;s not forget about personal development. Offering<br>    workshops, training sessions, and mentorship programs can help employees feel<br>    valued and grow professionally.<br><br><br>    *Mr. Kim:* Absolutely. And we should consider creating opportunities for cross-departmental<br>    collaboration. This not only fosters innovation but also helps employees understand<br>    the bigger picture of our organization.<br><br><br>    *Ms. Patel:* Great suggestions! How about implementing a suggestion box system<br>    where employees can anonymously share their ideas or concerns?<br><br><br>    *Mr. Thompson:* That&#39;&#39;s a fantastic idea, Rita. It promotes open communication<br>    and gives everyone a voice.<br><br><br>    *Ms. Rodriguez:* We could also organize team-building activities outside of work<br>    hours. Potlucks, game nights, or volunteer events can strengthen relationships<br>    and foster a sense of community.<br><br><br>    *Mr. Kim:* And let&#39;&#39;s not forget about work-life balance. Encouraging flexible<br>    work hours and remote work options can help employees manage their personal lives<br>    while staying committed to their jobs.<br><br><br>    *Ms. Patel:* Yes, absolutely. We should ensure that our policies support a healthy<br>    work-life balance. Any other ideas?<br><br><br>    *Mr. Thompson:* How about creating a wellness program that includes fitness challenges,<br>    meditation sessions, or stress management workshops?<br><br><br>    *Ms. Rodriguez:* That&#39;&#39;s a wonderful idea, Tom. A healthy employee is a happy<br>    employee.<br><br><br>    *Mr. Kim:* And we could establish an Employee Resource Group to support diversity<br>    and inclusion initiatives. This can help employees feel more connected and included.<br><br><br>    *Ms. Patel:* Wonderful ideas, everyone! Let&#39;&#39;s take some time to evaluate these<br>    suggestions and create an action plan. We&#39;&#39;ll reconvene next week to discuss the<br>    next steps.&#39;<br>  question: Could you generate the minutes of the meeting from the given transcript?<br>task_description: &#39;&#39;</pre><ul><li>Provide a second text file detailing attribution of the content, such as who created it and where it came from.</li></ul><p><strong>2. List and Validate Your New Data:</strong></p><ul><li>Run the command ilab diff to list your new data and ensure it’s registered correctly.</li></ul><pre>ilab diff</pre><p><strong>3. Generate a Synthetic Dataset:</strong></p><p>Run the command <strong>ilab generate</strong> to generate a synthetic dataset based on.</p><pre>ilab generate</pre><p>the newly added knowledge or skill set in the taxonomy repository.</p><ul><li>This step may take from 15 minutes to 1+ hours to complete, depending on your computing resources.</li></ul><p><strong>4. Train the Model:</strong></p><p>Train the model locally on Linux/Mac M Series, by running</p><pre>ilab train.</pre><ul><li>This step can potentially take several hours to complete depending on your computing resources.</li><li>The trained model will be outputted in the models directory.</li></ul><p><strong>5. Test the Newly Trained Model:</strong></p><p>Run the command</p><pre><br> ilab test </pre><p>to test the model and verify its performance.</p><p><strong>6. Serve the Newly Trained Model:</strong></p><ul><li>Stop any existing server by entering ctrl+c keys.</li><li>Convert the newly trained model using ilab convert.</li></ul><pre>#Serve the model locally via   <br>ilab serve --model-path &lt;New model name&gt;.</pre><p><strong>7. Chat with the New Model:</strong></p><ul><li>Try out the fine-tuned model live using the chat interface by running</li></ul><pre>ilab chat -m &lt;New model name&gt;</pre><p><strong>8. Submit Your Contribution:</strong></p><ul><li>If you’ve improved the model, open a pull request in the taxonomy repository to include the files with your improved data.</li></ul><p>Following these steps will allow you to contribute the meeting transcript summary data to the model and train a new model based on it, enhancing its capabilities in generating meeting minutes.</p><h3>How is InstructLab different from RAG and different Fine-Tuning techniques?</h3><h4><strong>How is it different from RAG ?</strong></h4><p>I came across this insightful analogy from a fellow IBMer <a href="https://medium.com/u/2c0a6bff798b">Suhas</a> that really resonated with me. It goes like this.</p><blockquote>💡 RAG functions like adding data or content while the program is running, simply means to making adjustments on the fly. Conversely, InstructLab operates more like adding data during the program’s compilation phase, ensuring that the model is equipped with the necessary information from the start.</blockquote><p><strong><em>However A fusion of InstructLab techniques with RAG yields superior outcomes.</em></strong></p><h4><strong>Now, let’s examine the distinctions between Fine-tuning and InstructLab.</strong></h4><blockquote>InstructLab is an end-to-end solution for generating data and training models. It encompasses the entire process, ensuring seamless integration from data creation to model training.</blockquote><blockquote>PEFT (Parameter-Efficient Fine-Tuning) , QLORA (Low Rank Adaption) is a specialized approach where instead of training the entire model, only an external adapter is trained. This method is efficient and can significantly reduce the computational resources required.</blockquote><blockquote>Fine-tuning, on the other hand, is a broader term that refers to training a language model with specific data to adapt it to particular tasks or domains. Our focus within this spectrum is on instruction tuning, where we optimize the model’s performance by training it with specific instructional data. This targeted approach allows us to enhance the model’s ability to follow instructions accurately and effectively.</blockquote><p>InstructLab introduces a distinct approach to address these challenges. By combining taxonomy-driven data curation, large-scale synthetic data generation, and a multi-phase tuning framework, InstructLab diverges from conventional fine-tuning methods. This unique methodology enables the improvement of Large Language Models (LLMs) using minimal human-generated data and computational resources. Additionally, InstructLab fosters ongoing model enhancements through user contributions. Notably, InstructLab stands out as an open-source project, promoting collaboration and community-driven development to refine these models.</p><h3><strong><em>Choosing the Right Approach for Model Adaptation: Prompt Tuning, Fine Tuning, or InstructLab?</em></strong></h3><p><strong>Let’s explore each method to determine the ideal choice for different scenarios.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rB8H0G1zZ3Yd3kXju69iQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AaTcYyKVw2HhX1_2VDvhdw.png" /></figure><p><strong>What’s Next ?</strong></p><p>To get started, join the <a href="https://github.com/instructlab">InstructLab</a> community in GitHub. You can also explore IBM foundation models from IBM <a href="http://watsonx.ai/">watsonx.ai</a> studio that are designed to support knowledge and skills contributed by the open source community.</p><p>The following foundation models support community contributions from InstructLab:</p><ul><li>granite-7b-lab</li><li>granite-13b-chat-v2</li><li>granite-20b-multilingual</li><li>merlinite-7b</li></ul><blockquote><strong>Take Action Now:</strong></blockquote><ol><li>Join <a href="http://watsonx.ai/">watsonx.ai</a> community -<a href="https://community.ibm.com/community/user/watsonx/communities/community-home?CommunityKey=81927b7e-9a92-4236-a0e0-018a27c4ad6e">https://community.ibm.com/community/user/watsonx/communities/community-home?CommunityKey=81927b7e-9a92-4236-a0e0-018a27c4ad6e</a></li></ol><p>2. Join regional watsonx user groups (attached pic)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*98nz8Xx3Fae1pWByg1u2Yg.png" /></figure><p>Happy Learning! 😊</p><p>Keep an eye out for further use case implementations with InstructLab.</p><p>— Syeda</p><p>A big shootout to Sumeet Kapoor, <a href="https://medium.com/u/2c0a6bff798b">Suhas</a> and <a href="https://medium.com/u/8af2088c8110">amold</a> for their valuable feedback and support.</p><p>Special Thanks to Oleg Silkin for clarifying all my doubts :) and heartfelt congratulations for your outstanding contributions to InstructLab 👏🏻.</p><p><strong>Sources:</strong></p><p><a href="https://youtu.be/zjkBMFhNj_g?si=Y-oxCVlD0w8xOjjj">https://youtu.be/zjkBMFhNj_g?si=Y-oxCVlD0w8xOjjj</a> — About LlAMA2–70B computational details.</p><p><a href="https://www.redhat.com/en/topics/ai/what-is-instructlab">https://www.redhat.com/en/topics/ai/what-is-instructlab</a></p><p><a href="https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-instructlab.html?utm_source=skills_network&amp;utm_content=in_lab_content_link&amp;utm_id=Lab-439&amp;context=wx&amp;pos=10&amp;locale=en&amp;audience=wdp">InstructLab-compatible foundation models</a></p><p>Please note that this views expressed in this blog are solely my own opinions about InstructLab.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3331ccea8d88" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>