What is an Agent Harness?

From LLMs to Agentic Loops to Managed Execution with Microsoft Copilot Harness in Foundry

9 min read5 days ago

The evolution from simple back-and-forth conversations with LLMs to reasoning-driven, task-oriented systems is what paved the way for AI agents.

Along the way, we came to an important realization: building powerful AI applications isn’t just about choosing the most powerful LLM. It’s also — an mainly — about what we build around it.

System instructions (aka the “job description”), context, knowledge retrieval (RAG), tools, and orchestration can transform a general-purpose model into a specialized system capable of reasoning about problems, accessing relevant information, interacting with external environments, and working toward specific goals.

And as AI systems evolve from answering questions to executing increasingly complex tasks, one component is becoming particularly important: the agent harness, the machinery that makes all of this work together.

To understand why, it helps to look at the evolution of AI agents through four stages:

LLM: The foundation that understands requests, reasons, and generates outputs — think of it as the “brain”.
RAG and tool calling: Extend the model’s capabilities by grounding its responses in external knowledge and enabling it to request actions from external systems.
Agentic loop: Introduces iterative decision-making, where actions and observations inform the model’s next steps.
Agent harness: The software that implements and manages the loop, bringing together context, tools, execution environments, and operational controls.

Think of this as a progressive enrichment of the LLM: from generating responses, to interacting with the world, to autonomously coordinating multiple steps toward an outcome.

Let’s unpack them through a practical example.

The LLM: Intelligence without execution

Imagine a sales manager asking an AI assistant:

“Analyze our Q3 sales performance, identify the main revenue drivers, and prepare a PowerPoint for the leadership team.”

A capable LLM can understand the request, outline an analytical methodology, and propose the presentation’s structure.

We could start with a simple prompt:

response = llm.generate(
    system="You are a senior sales analyst.",
    user="Analyze Q3 sales and prepare an executive presentation."
)

The model might suggest comparing Q3 with Q2, calculating revenue growth by region, and highlighting underperforming products.

However, it doesn’t automatically have access to the company’s sales data, real-time market trends, or any other type of information which was not part of the LLM’s training data.

RAG and tool calling: Extending the LLM beyond its internal knowledge

The next step is to connect the model to information and capabilities outside its training data.

Two complementary mechanisms make this possible.

Retrieval-Augmented Generation (RAG) provides relevant external knowledge to the model. Instead of relying exclusively on its learned knowledge, the application retrieves supporting information and includes it in the model’s context.

For our sales-analysis agent, this could mean retrieving the company’s sales definitions, regional reporting methodology, or quarterly business review guidelines from a knowledge base.

context = knowledge_base.search(
    "Q3 sales reporting methodology and KPI definitions"
)

response = llm.generate(
    system="You are a senior sales analyst.",
    context=context,
    user="Analyze Q3 sales and prepare an executive presentation."
)

The model now has access to company-specific information that can ground its analysis.

But knowledge retrieval alone doesn’t execute business operations.

Tool calling allows the model to request an action through an external function, API, or service.

Let’s extend our agent with three tools:

tools = [
    get_sales_data,       # Retrieve Q2 and Q3 figures
    run_python,           # Calculate trends and create charts
    create_presentation   # Generate the PowerPoint
]

response = llm.generate(
    user="Analyze Q3 sales and prepare an executive presentation.",
    tools=tools
)

The model might respond with a structured request:

{
  "tool": "get_sales_data",
  "arguments": {"period": "Q3-2026"}
}

An external runtime executes the requested tool and returns its result to the model.

The distinction is important: the LLM decides which action to request, but the surrounding software executes it.

Note: RAG and tool calling aren’t the same mechanism. RAG can happen before inference, without the model initiating a tool call. However, retrieval can itself be exposed as a tool, meaning that the agent will decide whether it needs additional knowledge before vectorizing the query and proceeding with the retrieval pipeline.

What they have in common is that both extend the capabilities of the underlying model.

Our agent can now access business knowledge and request operations. But to complete the original task, it must coordinate multiple dependent actions.

3. The agentic loop: From individual actions to multistep work

An agentic loop enables the system to repeatedly invoke the model, execute actions, observe their results, and determine the next step.

The fundamental pattern is:

Decide → Act → Observe → Repeat

For our sales-analysis agent, that could mean:

Retrieve Q3 and Q2 sales figures.
Execute Python calculations to identify growth trends.
Investigate an unexpected decline in one region.
Generate charts and create the PowerPoint.
Inspect the output and complete the task.

Rather than executing a completely predefined sequence, the model can adapt its next action based on the results of previous steps.

Get Valentina Alto’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Conceptually, the execution might look like this:

messages = [user_request]
while True:
    response = llm.generate(
        messages=messages,
        tools=tools
    )
    if response.is_final:
        break
    result = execute_tool(response.tool_call)
    messages.append(response.tool_call)
    messages.append(result)

This is deliberately simplified pseudocode. A real implementation needs to manage limits, errors, invalid tool requests, and other execution concerns.

However, we’ve introduced a new layer: the model is no longer being invoked just once, as a new “piece of software” (in our example, the while loop) is repeatedly calling it, executing its requested tools, and maintaining the information required for subsequent decisions.

That software is already a minimal agent harness.

The agent harness: The machinery around the model

An agent harness is the software system surrounding an LLM that enables it to execute multistep tasks.

The agentic loop is its central mechanism, but a sophisticated harness also manages context, tools, execution environments, state, and operational controls.

So far, we’ve built our sales-analysis agent progressively: starting from a model, enriching it with RAG and tools, and introducing a loop to coordinate multiple actions.

But who actually builds and operates this machinery? There are three main approaches:

Building a custom harness. At the most basic level, we can write our own loop in Python, as we did in the previous section. This gives us maximum flexibility, but we own the implementation of context management, tool orchestration, error handling, observability, and other runtime capabilities.
Using an agent framework. Rather than implementing every capability ourselves, we can use an agent framework such as Microsoft Agent Framework (MAF) or LangGraph. These provide reusable abstractions for building agents, including model interaction, tool execution, state management, and orchestration.
Using a platform with a managed harness. A third possibility is to use a platform where the harness is already provided as a managed capability.

Let’s enrich our sales-analysis example in the light of a more sophisticated agent harness.

Our minimal Python loop works for a basic prototype. But what happens if a sales API fails? If the analysis generates a large amount of intermediate data? Or if execution is interrupted after producing most of the presentation?

A more sophisticated harness might introduce functionality (for this scenario, I will use the MAF create_harness_agent() class):

# Illustrative harness configuration
agent = create_harness_agent(
    client=client,
    name="sales-analyst",
    agent_instructions=(
        "Analyze quarterly sales performance. "
        "Use verified data and explain your findings."
    ),
    tools=[get_sales_data],
    disable_web_search=True,
)

session = agent.create_session()

response = await agent.run(
    "Analyze Q3 sales and prepare an executive summary.",
    session=session,
)

print(response.text)

The same LLM and tools can now operate within a more capable runtime.

Now, as you migth rightfully think, there is an “art” behind designing such a capable runtime, which is now part of the same family originated with prompt engineering. This one is called harness engineering.

Why harness engineering matters

Imagine two teams building our sales-analysis agent using the same model.

One implements a minimal execution loop, repeatedly sends the entire conversation history, and has limited error handling.

The other manages context efficiently, provides reliable tool execution, tracks intermediate state, and implements appropriate controls.

Even if the LLM behind is the same, we will potentially see very different outcomes: Context management, orchestration, tool reliability, and execution strategy all affect the ability to complete useful work.

However, building and maintaining a sophisticated harness is no trivial, so organizations might ask themselves: Why should every development team have to build and maintain the same execution machinery?

From harness engineering to a managed harness: Microsoft Copilot in Foundry

On October 28, 2026, Microsoft announced the public preview of the Microsoft Copilot harness in Foundry Agent Service.

The idea is straightforward: rather than requiring every developer to build and operate the entire system around an agent, Microsoft provides much of that execution machinery as a managed capability.

The Copilot harness runs the agentic loop and sandbox, coordinates model and tool interactions, and manages working context. It also supports reusable skills, connected tools, code and file execution, and delegation to built-in subagents.

Developers bring the instructions, knowledge, procedures, tools, and supported model choice. Microsoft manages much of the underlying runtime.

NOTE: Bringing agent harnesses to managed platforms draws an interesting parallel with managed agentic RAG in Foundry IQ. Just as Foundry IQ abstracts much of the complexity of query planning, retrieval, and knowledge orchestration, the Microsoft Copilot harness brings a similar approach to agent execution, managing the agentic loop, context, and tool orchestration.
Both reflect a broader trend: shifting ownership of increasingly sophisticated agentic components from developers to managed platforms, allowing teams to focus less on infrastructure and more on business outcomes.

This doesn’t mean developers no longer need to design secure tools, configure permissions, evaluate outputs, or validate business results. It means they can invest less effort in generic runtime implementation and more in the specific capabilities that differentiate their application.

Copilot Harness in Foundry Agent is currently in preview and available in a subset of regions. You can test it already while building your agent with the following configuration (portal view):

More details here: Use the Microsoft Copilot harness with prompt agents — Microsoft Foundry | Microsoft Learn

Conclusion: Building systems, not just choosing models

We started with a simple request: analyze quarterly sales and prepare an executive presentation.

An LLM could understand the task. RAG provided business-specific knowledge, while tools enabled external actions. An agentic loop coordinated the sequence of decisions and observations. And the harness supplied the machinery that made the entire process operational.

Microsoft’s Copilot harness in Foundry represents a move toward making that execution layer a managed, reusable capability, reducing the need for each development team to engineer it from scratch.

The result is a shift in engineering effort: from building generic agent infrastructure toward designing specialized agents that deliver measurable business outcomes.