Today, LLM agents are everywhere and they are fundamentally changing the way we work. They are revolutionizing automation, decision support, and enterprise workflows. Companies are integrating agents into their ecosystems while more platforms are offering agents as a core offering. The buzz is warranted; agents bring a level of power that attracts tech and non-tech crowds alike. Building an agent and observing its capabilities creates excitement. You can see the agent take instructions, interact with its environment, and take actions to achieve goals. That’s great. But how do we know if the agent is doing a good job? Let’s say we make changes to our agent; do we know the effect of those changes? This is where evaluations come into play.

Evaluations exist throughout the entire agent development lifecycle. We focus on testing and validation during the development phase, then shift our attention to monitoring once the agent is in production. 2025 was dubbed the year of agent POCs. Now, companies are ready to take agents into production. Evaluation will allow teams to ship agents with confidence and build trust.

Understanding agent evaluation

What are agents?

Before diving into how to evaluate an agent, let’s define it.

Agents are autonomous systems generally powered by a large language model (LLM) and capable of taking actions required to achieve objectives across multiple steps. Unlike traditional software that generally maps inputs to outputs, agents have the liberty of reasoning and deciding what action to take. Powerful but more difficult to control.

An agent is generally made up of three key components:

• Reasoning engine: typically an LLM that analyzes information and makes decisions on what actions to take.
• Tools: external capabilities the agent can invoke like an API call or web search. The sequence of actions and tool calls is known as a trajectory and is a major focus of agent evaluations.
• Memory: how the agent stores and retrieves context such as conversation history and user preferences.

What is agent evaluation?

Agent evaluation is a structured process of measuring an agent’s behavior. We assess how well the AI system can autonomously perform tasks, leverage tools to take actions, and reason over multiple steps to achieve an objective.

For practitioners experienced with evaluating traditional machine learning systems, agent evaluation introduces a more comprehensive set of evaluations. Traditional ML evaluations focus primarily on a model’s performance on a fixed dataset; agent evaluations require us to assess an autonomous AI system that can reason, call tools, and has multiple ways of achieving its objectives.

We know agents pack power with a hint of unpredictability. With an agent, we aren’t working with a simple input/output combination; they operate over multiple turns. This makes agents useful, but it also makes them difficult to evaluate.

In traditional software development, unit and integration tests help ensure code correctness, reliability, and stability by catching bugs before they reach production. Agents introduce variability in their outputs that renders standard software testing inadequate. Holding everything constant and asking the same question twice doesn’t guarantee the same result, let alone the steps required to get to the result. Using evaluations is a good mechanism to measure and limit this variability.

Another challenge is silent failures and error propagation. Agents can fail a step, continue to run steps, and return a faulty result with complete confidence. This means some errors won’t even return an error message!

Agent evaluation isn’t a one-off activity, it actually spans the entire project lifecycle. During the development phase, we will validate and test whether our agent behaves correctly by grading it on a diverse set of examples where the expected outcome is known. In production, we will monitor our agent to ensure things are running smoothly and performing well when handling real-world requests.

What do we measure?

It’s time to figure out what we want to measure. Agent performance can’t be captured in a single score so we need to measure different aspects of an agent’s performance. We define two performance areas: Operational and Behavioral. Each performance area comes with its own set of evaluation dimensions and corresponding metrics.

Operational performance

Operational performance measures whether the agent executes tasks reliably and efficiently.

Key evaluation dimensions

• Latency
• Resource usage
• Cost efficiency
• Reliability

Example metrics

• Response latency: time taken to complete actions and produce the final output.
• Memory and processing usage: CPU, GPU, and memory consumed during task execution.
• Token consumption: the number of input and output tokens used by an LLM-based system, which helps estimate system costs.
• Tool call failure rate: ratio of failed tool calls to total tool calls.

These metrics don’t require comparing the agent’s performance to a reference; they capture how the system performs.

Behavioral performance

Behavioral performance measures whether the agent behaved correctly while completing a task. We evaluate both the final response and the trajectory.

While we are building the agent, we can establish ground truth, which consists of the expected outcome and, when relevant, the expected behavior or trajectory. We then compare the agent’s actual behavior against the ground truth. The appropriate metric depends on the task. High-stakes workflows may require an exact trajectory match, while more flexible tasks may only require certain tools to appear in the correct order.

Key evaluation dimensions

• Tool-use correctness
• Trajectory correctness
• Output correctness

Example metrics

• Single-tool use: verifies that a mandatory tool call was used.
• Exact/strict match trajectory: requires the actual actions to match the expected actions and be in the expected order.
• Precision of trajectory: measures how many actions taken were correct according to the expected trajectory.
• Semantic match: measures whether the agent’s final response conveys the same meaning as the expected response.

How do we measure it?

Once you know what to measure, we can choose how to measure it. A robust evaluation strategy combines deterministic checks, LLM judges, and human judgment.

Deterministic evaluations should be our first choice when success can be expressed objectively. We can execute these checks using code. They can verify the metrics we defined above such as single-tool use, trajectory match, and token consumption.

They can answer questions such as:

• Did the agent produce valid JSON?
• Did it call the correct tool with the correct arguments?
• Did it stay under the token budget?
• Did the workflow complete within the latency SLA?

These evaluations are fast, cheap, and reliable. When success relies on subjective qualities, this is when the limitations of deterministic evaluations kick in.

LLM judges or LLM-as-a-judge come into play when our checks aren’t deterministic. Instead of defining rigid rules, we use one LLM to evaluate the output of another LLM/agent. LLM judges can evaluate an agent based on a rubric or trait for things like helpfulness, tone, or relevance. LLM judges provide the benefit of evaluating subjective qualities and scaling better than human evaluations, but they come with their own set of complications. They rely on an LLM, which incurs a cost each time the metric runs, and their reliability depends on the quality of the evaluation criteria. For subjective or domain-specific traits, calibration against human judgment can help ensure the LLM judge differentiates good and bad output.

These are questions LLM judges would be perfect for answering:

• Was the response helpful?
• Was the explanation complete?
• Was the tone appropriate?
• Was the reasoning sound?
• Did the agent choose a reasonable strategy?

Human judgment remains the gold standard for evaluating agent performance. Humans act as the calibration layer for the entire evaluation pipeline. It is the responsibility of a human to define what a good output looks like and why it is a good output. When starting the agent evaluation process, a Subject Matter Expert (SME) is responsible for labeling responses in an evaluation dataset and writing an explanation. This is what we will eventually calibrate LLM judges against.

Each of these strategies answers a different question. Humans define ground truth and periodically validate evaluations. Deterministic evaluations provide fast, cheap, and reliable checks. LLM judges expand our evaluation capabilities by checking subjective and complex behaviors.

Evaluating agents across the lifecycle

We know what to measure and how to measure it; now, let’s figure out when to apply evaluations. There are two distinct phases of agent evaluation: pre-deployment validation and post-deployment monitoring. The data we have available, the goals, and the techniques we use across the two phases differ. Let’s look deeper into what makes these two phases different.

Pre-deployment validation

The primary objective before deploying an agent is to ensure it meets the quality requirements for real-world use.

This is our opportunity to run validation and testing in a controlled environment. We determine what tasks an agent can do, create a set of representative test cases with expected outcomes, and define what good behavior looks like. This makes the pre-deployment phase perfect for validating both what the agent produces and how it gets to those outcomes. We can also establish performance baselines, uncover failure modes, and iterate on the agent to prevent known failures from reaching users.

At the center of validation is what is known as the evaluation dataset or golden dataset. It contains representative examples of tasks we expect the agent to handle along with the expected outcomes and behaviors required to evaluate them. Building a good evaluation dataset often requires SMEs to provide their domain knowledge to properly define correct responses and behaviors. In the case of an HR agent, an HR expert will know whether an answer actually provides the level of explanation required to handle an employee’s question or if an HR request needs to be routed through a particular system.

Once we have an evaluation dataset, we can run the agent against those test cases and measure its performance.

During pre-deployment, behavioral evaluation is particularly important because we can compare the agent’s performance to expected outcomes and behaviors. As mentioned earlier, we evaluate the final response and trajectory using a combination of deterministic checks, LLM judges, and human review. For operational evaluation, we can establish performance thresholds around latency, resource usage, and token consumption we expect the production-ready agent to meet.

Putting it all together, we create a validation loop:

1. Build a representative evaluation dataset: include common tasks, edge cases, and important workflows.
2. Define expected outcomes and behaviors: specify the agent’s desired final response and how it should behave.
3. Run the agent multiple times: since agents are nondeterministic, measure variance in responses and identify inconsistent behavior.
4. Apply multiple evaluation methods: combine deterministic checks, LLM judge traits, and human review.
5. Analyze failures: inspect final response, individual tool calls, trajectory, and other relevant metadata to diagnose failures.
6. Iterate and rerun: fix failures, update the evaluation dataset, and compare the new version against the previous baseline.

There will always be room for improvement, but that shouldn’t hold us back from deploying the agent. The objective of the pre-deployment phase is to understand how the agent behaves, where it fails, and whether the agent’s performance shows it’s ready for production.

Example: Evaluating an HR agent

Let’s use an HR (Human Resources) agent to walk through the entire agent evaluation lifecycle. The goal of the HR agent is to help employees answer HR questions and automate basic HR workflows.

The agent has access to several tools that allow it to accomplish HR tasks:

• Company policy retrieval to find relevant HR policies
• PTO (Paid Time Off) balance API to retrieve an employee’s available PTO balance
• Payroll system to answer pay-related questions
• HR ticket system access to look up, submit, and route HR requests
• Employee directory lookup to identify the appropriate contact

Let’s see how the HR agent is validated before deployment and monitored after deployment.

Step 1: Build the evaluation dataset

The first step is to work with HR SMEs to create a representative set of test cases that cover edge cases and important workflows. The dataset could be as simple as this:

These examples define the answer we want to see and how the agent should get to that answer.

Step 2: Validate answers and behaviors

Next, we run the HR agent against the evaluation dataset and examine three things:

• What the agent answered with
• What the agent did
• How the system performed

For the PTO days test case, the final answer might be correct, but the actual trajectory could reveal that the agent never called the PTO API. A single-tool-use check could verify that a mandatory tool was used. For a parental-leave question that requires escalation to an HR specialist, the agent must first search the employee directory to identify the appropriate contact, then retrieve the contact’s details. This is perfect for an exact trajectory match since we require the actual trajectory to match the actions and order of the expected trajectory.

Operational metrics can reveal if the agent consistently takes too long to answer or if it consumes more tokens than we allow it to.

It’s also important to run the agent multiple times to identify any sort of inconsistent behavior across runs.

There is no single score that can represent agent performance. We must strategically choose metrics to evaluate different dimensions. This is why pre-deployment validation should combine:

Final response quality + tool use and trajectory + operational performance

Step 3: Use LLM judges and human judgment

Some HR interactions require more nuance than others. Determining if the PTO balance tool was called can be handled by a deterministic check. Determining if an explanation of a benefits policy is clear and helpful? That’s much harder to answer with deterministic checks. This is where an LLM judge can evaluate a defined trait or rubric. An LLM judge can answer questions like:

• Is the response consistent with the provided policy?
• Does the response provide enough context for the employee to understand what to do next?
• Does the response directly answer the employee’s question?
• Is the agent’s final outcome the same as the expected final outcome?

All four of these questions are individual traits. Each one has a prompt that specifies the criteria for the check. For example, the criteria for the final outcome check might direct the LLM judge to check if the agent’s answer and the reference answer are semantically equivalent with no contradictions.

SMEs should also review traits for the test cases in the evaluation dataset to validate whether the LLM judge aligns with their expert judgment. Disagreements between the SME and LLM judge help highlight weaknesses in the agent but also potential ambiguity in the evaluation criteria itself. The goal is to ultimately have your test cases pass both SME and LLM judge evaluation.

Post-deployment monitoring

Now that the agent is deployed, our evaluation focus changes.

In development, we operate in a controlled environment where we decide the test cases and define the ground truth. In production, we face unexpected requests, changing business processes, and model updates.

This is why we can’t treat production monitoring as pre-deployment validation at a larger scale. They have two completely different objectives.

The goal of the post-deployment phase is to determine whether the deployed agent remains healthy and behaves correctly when tasked with real-world requests. We need to detect operational issues, performance degradation, unexpected behaviors, and new failure modes.

A key part of monitoring is capturing the agent’s traces. A trace is a raw record of what happened during a run. It includes the user’s request, actions taken by the agent, the tools called, intermediate results, guardrails triggered, timespans, the final output, and other metadata. The final output isn’t enough; we need a detailed log of every decision and action the agent took to confirm the agent produced the correct answer without triggering guardrails, making excessive tool calls, or burning unnecessary tokens.

Operational monitoring provides the baseline. Evaluation dimensions like latency, resource consumption, token consumption, and tool-call failures allow us to establish alerts, catch cost or latency increases, and identify infrastructure problems.

Behavioral monitoring also has its role in the post-deployment phase. In production, we generally don’t have a defined reference answer or expected behavior. Instead, we extract the actual response and trajectory from the agent trace and use different evaluation methods to understand what’s going on. Deterministic checks can monitor objective characteristics. LLM judges can check for relevance, helpfulness, and other subjective traits. Even human feedback can be incorporated into the mix through thumbs up/down, escalations, support tickets, and periodic SME reviews.

Example: Evaluating an HR agent continued

Step 4: Monitor the agent in production

Once the agent meets the required validation criteria, we can deploy it into production with monitoring in place.

Monitoring our agent starts with capturing traces. We can then decompose the trace to inspect the agent’s behavior. For each interaction, we capture a trace that contains information like tool calls, final responses, technical metadata, and more. We start with tracking operational dimensions like latency, token consumption, tool failures, and resource usage to monitor the health of our system. We can also use behavioral evaluations to observe what is happening and identify interactions that require further inspection.

Step 5: Turn production failures into new test cases

Let’s say an employee tells the HR agent, “I need to change my direct deposit.” The agent then incorrectly attempts to modify the employee’s bank details instead of routing the employee to the Payroll workflow as expected.

How do we know this happened? From the agent’s trace, we know there is an unexpected tool call. An LLM judge could be used to assess improper behavior. A user could have submitted a thumbs down because the agent made a mistake with the bank details.

Once an interaction is flagged, an SME can review it to confirm that it is a genuine failure. They can confirm the root cause and we can turn the interaction into a new test case in our evaluation dataset with expected outcome and behavior documented. We could even add a guardrail that prevents the agent from performing similar actions again.

The next version of the agent can be tested against the updated evaluation dataset before the next deployment. This creates the evaluation flywheel:

The agent evaluation strategy evolves as the agent encounters more real-life interactions. Production failures become new evaluation cases which make the evaluation dataset more comprehensive and help future versions of the agent improve.

Conclusion

Agents are powerful systems that can reason, take action, and autonomously achieve goals. That same autonomy makes agents harder to trust. Evaluations give us a way to comprehensively assess the agent’s responses and behaviors.

A single score isn’t an agent evaluation. We need a comprehensive strategy to evaluate what the agent produces, how it gets there, and how reliably the system operates using a combination of methods like deterministic checks, LLM judges, and human feedback. Before deployment, we validate the agent’s performance against a representative evaluation dataset composed of test cases with known answers and behaviors. After deployment, we monitor real-world interactions, find new failure modes, and feed those back into the evaluation dataset for future validation.

Ultimately, evaluation is what separates an impressive demo from a reliable agentic system. As agents move into more critical and complex workflows, a strong evaluation strategy will be key to building systems you can confidently deploy, continuously improve, and users can trust.

Agent evaluation: What to measure, how, and when was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background on the Surf Trip Planner Agent

The project was born out of a personal pain point: the need to streamline remote work logistics — specifically, coordinating surfing trips to southwest France. Traditionally, this requires a recurring manual workflow whenever favorable conditions align:

• Environmental Monitoring: Analyzing surf forecasts and wave conditions, primarily for upcoming weekends.
• Schedule Synchronization: Reviewing professional calendars to identify mandatory in-person meetings or events in Paris.
• Travel Procurement: Searching for available round-trip train tickets and managing booking logistics.

The goal of this project was to determine how an agentic system could automate these low-value administrative tasks, allowing for more efficient decision-making and eliminating manual oversight.

The advantages of code-based agents

Developing the surf trip planner required a well-defined workflow, including specific data requirements, clear decision-making logic, and procedural milestones.

Code-based agents in Dataiku offer the flexibility of custom development alongside seamless platform integration. This ensures that custom agents can leverage the full suite of features available in Dataiku, operating with the same compatibility as native visual agents.

Specifically, code-based agents provide:

• Graph Customization: The ability to design tailored logic graphs by blending standard workflows with ReAct loops.
• Context Management: High-precision control over the information and state provided to the agent.
• Operational Granularity: Fine-tuned oversight of the agent’s execution and reasoning processes.

Note: With the introduction of structured agents, users can now build sophisticated workflows using a no-code visual interface, offering an alternative for complex agent design.

Framework selection

The agent was developed using LangChain and LangGraph, both of which are industry-standard libraries for building agentic workflows. These frameworks offer seamless integration with Dataiku, providing a robust foundation for development.

A significant advantage of this stack within the Dataiku ecosystem is the ease of observability. For instance, the Traces Explorer web app — a dedicated tool for trace analysis — can be activated with a single line of code.

While alternative frameworks such as CrewAI or LlamaIndex remain viable options, the LangChain ecosystem was selected for its versatility and the efficiency it offers when operating within Dataiku.

Requirements

The agent’s primary objective is to coordinate round-trip logistics for a surfing excursion. To ensure successful execution, the system must perform a comprehensive assessment of two critical variables:

• Environmental Validation: The agent must verify that forecasted wave conditions align with predefined quality criteria.
• Operational Feasibility: The itinerary must be cross-referenced with the user’s schedule to ensure the trip is compatible with existing professional and personal commitments.

User Scenario

The agent’s workflow is designed to automate the decision-making process through a structured sequence of operations:

• Input Collection: The agent prompts the user for core journey parameters, including the destination and the desired timeframe.
• Condition Assessment: The system analyzes surf forecasts for the specified dates.
• Conditional Logistics Processing: If the environmental conditions meet the predefined quality threshold, the agent executes the following:
  - Availability Audit: It cross-references the user’s professional calendar to verify availability.
  - Travel Sourcing: It identifies the optimal round-trip train itineraries.
• Final Recommendation: The process concludes with the agent presenting one or two concise, actionable travel options for final selection.

Tooling and System Integrations

For an agent to transition from reasoning to execution, it requires access to tools — modular functions that allow it to interact with external data sources and services. To support the defined workflow, the following integrations were implemented:

• Transportation Management: Utilizing the Navitia API to query real-time train schedules and availability.
• Schedule Coordination: Leveraging the Google Calendar API to audit user availability and cross-reference professional commitments.
• Environmental Data: Integrating the OpenMeteo API to retrieve precise surf conditions and wave forecasts.

Agent Architecture

With the requirements defined, we can outline the agent’s structural architecture. The system is designed around a multi-stage workflow to ensure precise execution and error handling:

• Intent Classification — route_intent: The initial node identifies the user’s intent. This serves as a primary filter to ensure the agent responds accurately, particularly when handling out-of-scope queries or non-surf-related requests.
• Information Parsing — update_trip_details: This node processes the user’s input to extract relevant parameters and determines the subsequent logic path:
  - Incomplete Data: If essential parameters are missing, the workflow triggers the request_missing_detailsnode to prompt the user for the necessary information.
  - Complete Data: Once all parameters are validated, the system moves to the execution phase.
• Execution and Orchestration: The planning phase consists of two primary components:
  - Environmental Validation: A deterministic Python function verifies surf forecasts against user preferences.
  - Travel Planning: An autonomous agent, using a ReAct (Reasoning and Acting) loop, orchestrates the logistics.
• Response Generation: The workflow concludes by providing a concise summary of the proposed itinerary and travel options.

The following state diagram illustrates the agent’s workflow. To distinguish between execution types, nodes are color-coded as follows:

• Blue Nodes: Represent deterministic logic (standard code or rule-based processing).
• Pink Nodes: Represent LLM-powered decision points or generative tasks.

Conditional edges are named after the output of the decision function.

Note: While request_missing_detailsand handle_errorare currently driven by the LLM, these could also be transitioned to deterministic functions.

Developing Agents With LangGraph

Core Components

The architecture of a LangGraph agent is built upon several core components that manage the orchestration of logic and data flow:

• The Graph: This serves as the primary orchestration layer. It defines the overall workflow and the specific sequence of operations by connecting nodes via edges.
• Nodes: These are the functional units of the graph. In practice, nodes are Python functions designed to execute discrete tasks. They ingest the current state, process information, and return updates to be merged back into that state.
• Edges: These define the transition logic between nodes. Edges can be static, following a predetermined path, or conditional, allowing for dynamic routing based on the agent’s reasoning or external data inputs.
• State: Functioning as the agent’s persistent memory, the state is a shared schema maintained throughout the execution lifecycle. It enables seamless data persistence and communication between different nodes as the workflow progresses.

To ensure long-term readability and maintainability, it is highly recommended to adopt a modular structure by decoupling component definitions into distinct files or packages. The following directory structure illustrates the architectural organization implemented for this code-based agent:

surf_planner/
└── agent/
    ├── tools/
    ├── __init__.py
    ├── build.py
    ├── edges.py
    ├── graph.py
    ├── helpers.py
    ├── nodes.py
    ├── prompts.py
    └── state.py
└── apis/
 config.py

graph.py

This file defines the agent’s graph architecture. It serves as the orchestration layer, utilizing nodes and edges to formalize the structural logic and execution flow of the agent’s workflow.

workflow = StateGraph(AgentState)

# --- 1. ADD NODES ---
workflow.add_node("route_intent", partial(node_route_intent, ...))
.../...

# --- 2. ADD EDGES & ROUTING ---
workflow.set_entry_point("route_intent")
workflow.add_edge("request_missing_details", END)
.../...

# --- 3. BUILD AGENT ---
agent = workflow.compile()

nodes.py

This file houses the functional definitions of the agent’s nodes. In this architecture, nodes represent discrete tasks implemented as Python functions. A recommended best practice is to prefix these functions with node_ to maintain a clear distinction within the codebase. The internal logic of these nodes typically follows a consistent execution pattern: accessing the current state, performing the designated operation, and returning a state update.

def node_route_intent(state: AgentState, model):
    # 1. Read inputs from state
    msgs = "\n".join([f"{msg.content}" for msg in state["messages"]])
    .../...

    # 2. Do something
    classifier_chain = route_intent_prompt_template | model
    .../...

    #3. Return the update of the state
    return {"current_intent": intent}

edges.py

This file manages the transitions between nodes. While many transitions are deterministic and follow a fixed path, conditional edges facilitate dynamic routing. These are defined as functions that analyze the current State to determine the most appropriate subsequent node based on the agent’s internal logic.

def edge_after_update(state: AgentState) -> str:
    # 1. Read inputs from state
    details = state.get("trip_details", {})
    .../...

    # 2. Apply some logic
    if all(details.get(key) for key in mandatory_keys):
        # 3. Return the next node
        return "details_complete"
    else:
        # 3b. Return the next node
        return "details_incomplete"

state.py

The State serves as the agent’s persistent memory. It is fundamentally a schema — typically implemented as a TypedDict — that allows for the storage and retrieval of attributes throughout the execution lifecycle. In this implementation, the state includes a dedicated object to track trip details, ensuring that contextual data is seamlessly maintained and accessible across all nodes in the workflow.

class TripDetails(TypedDict, total=False): ...

class AgentState(TypedDict):
    messages: Annotated[List[BaseMessage], add_messages]
    trip_details: TripDetails
    .../...

To finalize the agent’s construction, we must define and implement the specific tools required for execution and external interaction.

Tools Implementation

Tools are modular Python functions (though alternative structures, such as class-based implementations, are also supported) that augment the agent’s capabilities, enabling it to act autonomously. They should be viewed as specialized skills — for instance, a function dedicated to retrieving real-time surf forecasts.

To ensure the agent can utilize these functions effectively, it requires a clear definition of the tool’s purpose and its required parameters. In the implementation below, a tool is declared using the @tool decorator. The function’s docstring serves as the primary description interpreted by the LLM, while the Pydantic model provides a structured schema for the arguments.

Key Considerations for Tool Design

• Type Hinting: Explicitly defining the types for both inputs and outputs is a best practice that ensures code clarity and facilitates easier debugging.
• Dependency Injection: In the provided example, the tool requires an external API client (e.g., a geolocator). This is managed via a wrapper function, allowing the client to be instantiated once and reused. While it is possible to instantiate clients directly within a tool, using a wrapper is more efficient; agents are not intended to manage the lifecycle of complex dependencies.

class SurfForecastArgs(BaseModel):
    spot: str = Field(description="The name of the surf spot or town.")
    .../...

# Tool
def create_surf_forecast_tool(geolocator: GeolocatorAPIClient, openmeteo: OpenMeteoAPIClient):
@tool(args_schema=SurfForecastArgs)
def get_surf_forecast(spot, from_date, to_date) -> list[DailySurfForecast]:
    """ Retrieve the surf forecast for a specific spot between two dates."""
    location = geolocator.get_coordinates(spot)
    forecasts_data = openmeteo.get_forecasts(...)
    return [DailySurfForecast(...) for f in forecasts_data]

return get_surf_forecast

The following tools were implemented for this project:

• Surf Forecast Tool: Retrieves real-time wave conditions and meteorological data via the Open-Meteo API.
• Rail Logistics Tool: Interfaces with the Navitia API (SNCF) to identify and source available train itineraries.
• Calendar Integration Tool: Audits schedule availability by querying the Google Calendar API to ensure trip feasibility.

Strategic Advantages of Code-Based Implementation

Developing an agent programmatically offers several distinct advantages for complex automation:

• Design Flexibility: Code-based development allows for highly specialized and tailored workflows. This enables the creation of intricate logic structures — such as the graph-based architecture discussed earlier — that define exact operational behaviors.
• Granular Control Over LLM Logic: Developers gain direct oversight of context management and prompt engineering for specific “intelligent” components. This ensures precise control over how individual nodes and tools interact with the underlying model.
• Operational Optimization: Implementation at the code level facilitates bespoke optimizations, such as managing API session persistence and optimizing data retrieval patterns to improve performance and reduce latency.

Furthermore, leveraging the LangChain and LangGraph ecosystem streamlines development through:

• Asynchronous Execution: Simplified implementation of streaming and asynchronous calls for better responsiveness.
• Standardized Syntax: Utilization of LCEL (LangChain Expression Language) for cleaner, more readable code.
• Built-In Observability: Integrated features like automated tracing provide comprehensive monitoring of node inputs and outputs by default.

Note: While a code-first approach offers maximum control, it also introduces significant complexity. This level of customization should be reserved for advanced requirements; in many scenarios, native no-code agentic features provide sufficient functionality to meet business objectives.

Implementation Insights

Development Environment

To accommodate the extensive coding requirements of this project, I opted for a local development workflow using PyCharm. While integrated options like Dataiku’s Code Studio are available, a local IDE provides the necessary flexibility and robust tooling required for this specific build.

The objective was to establish a seamless pipeline: developing and testing the agent locally before deploying it within the Dataiku environment. A primary focus of this setup was maintaining a portable codebase — minimizing the need for environment-specific configurations — to ensure the agent operates consistently whether running in a local terminal or within the Dataiku platform.

Project Libraries

The implementation leverages Project Libraries, a centralized code repository within Dataiku designed for developing reusable modules. By centralizing the agent’s logic within these libraries, the code becomes highly portable and can be seamlessly imported across different platform components — most notably within the Code Agents.

This approach ensures a clean separation between the core logic and the platform’s orchestration layer, facilitating better version control and modularity.

Local Integration

To optimize the local development cycle, I leveraged the Dataiku extension for PyCharm. This integration facilitates the seamless synchronization of local source files with Dataiku Project Libraries, ensuring that code authored in the IDE is immediately available within the Dataiku instance (It is also possible to use Git).

Remote Connectivity via the Dataiku API Client

To facilitate local execution, it was necessary to mirror the Dataiku runtime environment. This was achieved by installing and configuring the Dataiku Python API client on the local workstation.

This setup ensures parity between the local development environment and the Dataiku platform. Consequently, API calls to instance resources — such as datasets, managed folders, or global variables — are executed seamlessly from the local machine, providing a robust and integrated debugging experience.

Deployment and Execution in Dataiku

To deploy the agent within Dataiku, the initial configuration begins by creating a “code agent” through the project’s GenAI menu. Following this, a class must be defined to implement a specific interface, which enables Dataiku to orchestrate and execute the custom code.

Specifically, it is mandatory to implement a processmethod (note that asynchronous and streaming variants are also supported). This method serves as the entry point where the agent logic — previously defined in the Project Libraries — is imported. Dataiku invokes this function whenever the agent is triggered. Once the class and its processing logic are established, the agent is fully operational within the Dataiku ecosystem.

To minimize environment-specific boilerplate and enhance portability, the implementation utilizes an AgentBuilderclass. This component is responsible for assembling the agent by injecting all necessary dependencies, such as API clients and project-level parameters, ensuring a clean and modular initialization process.

.../...
from surf_planner.agent.build import AgentBuilder # from Project Libraries

class MyLLM(BaseLLM):
    
    def __init__(self):
        settings = ProjectSettings()
        client = dataiku.api_client()
        builder = AgentBuilder(settings)
        self.agent = builder.build()
    
    def process(self, query, settings, trace):
        messages = []
  # ... <format input messages> ...
        inputs = {"messages": messages}
        final_state = self.agent.invoke(inputs)
        final_answer_message = final_state["messages"][-1]
        return {"text": final_answer_message.content}

Once deployed, the agent can be leveraged across multiple channels within the Dataiku ecosystem:

• Batch Processing: The agent can be integrated into a Prompt Recipe to process large-scale datasets efficiently.
• Programmatic Access: It can be invoked via API, enabling integration into external applications or automated enterprise workflows.
• Conversational Interfaces: The agent can be exposed through integrated interfaces such as Agent Hub, providing a natural-language front-end for users to interact with the underlying logic.

LangSmith Studio

The ability to run the agent locally makes the use of LangSmith Studio seamless for advanced debugging. This tool transforms agent execution into an interactive process, providing a visual representation of the agent’s logic.

Key benefits of this local debugging workflow include:

• Real-Time Visualization: Monitor node transitions and state updates dynamically as the agent executes.
• State Inspection: Gain immediate visibility into the agent’s memory (the “State”) to ensure data persistence and accuracy across the workflow.
• Advanced Control: Utilize breakpoints and the ability to re-run execution from a specific node using a pre-defined state.

Once LangSmith is configured locally, initiating the development server is straightforward: simply run the langgraph devcommand to access the interactive visual interface and begin auditing the agent’s reasoning process.

Chat With the Agent

As previously highlighted, the code-based agent is integrated within the Dataiku ecosystem. For this project, we leveraged Agent Hub — a plug-and-play web application — to provide a streamlined conversational interface for end-users.

The agent is integrated into the “enterprise agents” catalog, where we configure curated sample queries to guide user interaction. With this final configuration in place, the system is fully prepared for functional testing and deployment.

The agent prompts the user for specific journey parameters and, upon confirming that all environmental and logistical criteria are met, executes the planning workflow. The following output illustrates the agent’s performance for a sample weekend:

Conclusion

This project demonstrated how to build a sophisticated code-based agent by addressing a practical logistics challenge: automating the planning of surfing trips.

By leveraging LangGraph, we developed a robust solution that coordinates complex workflows, manages state persistence, and utilizes specialized tools to interact with external APIs. This architectural approach provides the flexibility and control necessary for handling non-linear tasks that go beyond the capabilities of standard automation.

Furthermore, we explored how a local development workflow — combining a traditional IDE with the Dataiku API client — provides an excellent developer experience.

Ultimately, this implementation serves as a successful proof of concept, showing how autonomous agents can effectively eliminate repetitive administrative tasks and streamline complex decision-making processes.

Building a LangGraph Agent in Dataiku was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

Harness engineering has become a hot topic and seems to be ever evolving. What framework or architecture should it use? What capabilities should it include? How do I know if it is good enough? How do I make it safe and reliable? What even is an agent harness?

At Dataiku, we worked with 60+ enterprise customers across multiple industry verticals to help them deliver agents in production from which we’ve been able to identify AI agent building patterns.

We distilled all these agent building learnings into this article, splitting the agent building process into three layers: the model, the agent harness, and the agent harness configuration.

Layer 1: The Model

The first thing to remind ourselves is that an agent is a powerful system and, at its core, it boils down to a smart orchestration of LLM calls. The system that handles and orchestrates LLM calls within an agent is what we will call the agent harness.

LLM output generations can be structured to enable tool calling, the ability of an LLM to request for an action to be performed, which has become the backbone of all agentic systems.

The challenge is that a single LLM call is only able to “request” for a tool to be called, it doesn’t directly have the ability to execute it. That means that we still have to call the requested tool on behalf of the model for the action to be performed. This motivates the need for the second layer: the agent harness.

Layer 2: The Agent Harness

Historically, agentic systems were limited by their underlying model’s performance. Recently, the limiting factor has shifted to the agent harness. More real-world agent harness benchmarks like Terminal-Bench showcases how significant performance improvement can be achieved and attributed solely to agent harness optimizations.

That being said, with new agent harness features and capabilities being released on an almost daily basis, it can get complicated to focus on the ones that will survive long term versus the brittle features destined to be eaten by the next big model release feature, commonly known as “agent scaffolding.”

Agent Harness Core Model Loop

To give more agency to a model, we want to build a mechanism to inform the model of their previous action’s result so it can decide to perform another action OR give its final answer. This looping mechanism typically follows the ReAct Framework and is a foundation feature of any agent harness.

We found that the agent harness’s core loop can be reduced to a repeating cycle of five steps:

1. Input Request: This can be a new user request, a user clarification, or the result of the previous core model loop iteration.
2. Context Engineering: This step focused on assembling all the context the model needs for its next generation including general LLM guidelines saved in the system prompt, user preferences, historical conversation & steps taken, current plan, expertise instructions, and retrieved data.
3. Equip Relevant Tools: Select and attach the tools available to the model for this turn. These are generally kept the same but could be updated based on the task requirements.
4. Run the Model: Choose the right model to run based on task complexity/cost requirements and execute it with the assembled context and tools that handle model errors and retries.
5. Harness Housekeeping (where most of the complexity lies): Also known as hooks or middleware, this step parses the model outputs; executes any requested tool or sub-agent call while handling errors and retries; applies guardrails; provides observability and logging; manages memory; tests the agent’s work and calls for user intervention when needed (human in the loop).

At the end of each cycle, a decision is made to return a final response OR feeds the result of the current cycle into the start of the next one and kicks it off.

Most of the core agent harness features we will cover are built on top of this core model loop and apply to multiple steps of the harness cycle. They can shape context in step 2, provide tools in step 3, and enforce rules in step 5.

Simple Agent Harness Limitations

We started implementing agents with the simplest harness possible: a core loop. In theory, an agent harness equipped with a core model loop, with the right tools to interact with its environment and a capable model should be enough to complete any task.

In practice, we found that it was not the case. As tasks get longer and more complex, a bare core model loop harness runs into problems like:

• Context Rot when information in the context gets out of date or irrelevant.
• Context Anxiety when the model ends early when it approaches its context window.
• Tool Pollution when tool descriptions drown the context.
• Infinite loops when getting stuck in a local minima.
• Lack of visibility and trust as harness reasoning can become an unconstrained blackbox.

Agent Harness Design Principles

This motivated the need to add more features to our harness to remediate the issues listed above. Through trial and error, we noticed that to avoid building unnecessary “agent scaffolding,” the following design principles have reliably helped produce versatile and robust agent harness features.

• Progressive Disclosure — As agent working environments start getting more and more complex, agent harnesses should be designed to dynamically allow discovery, navigation and deep dive into knowledge about that environment based on the assigned task.
• Design for Reliability not Capability — You can have all the capabilities in the world, but if you can’t trust the agent, its value and usability drops to zero.
• Build for the Models of Tomorrow — Build features that work for the model today and that will allow future better model versions to grow into the harness and not be restricted by it.

Agent Harness Feature Implementation Patterns

When implementing a harness feature, we found that we needed to choose the level of autonomy which ended up falling into one of four implementation patterns. Let’s take the agent memory feature ordered by increasing level of model agency as an example:

1. Fully Handled by the Harness: The harness manages the feature end to end, invisible to the model. Ex: At the start of each turn, relevant user preferences & memory fragments are injected automatically into the context by the harness.
2. Specialized Tool-Based Capability: The model is given access to the capability via hyper-specific tools, with some light agent harness housekeeping. Ex: The agent is given specific memorizing and remembering tools that include deduplication of memory fragments.
3. High-Level Instruction: The model is instructed at a high level how to implement and run the feature via core multi-purpose generic tools. Ex: The harness is instructed to save and load any relevant user information in a user/memory.md file, segmenting the memory by type.
4. Full Autonomy: The model just knows that the feature is required, decides on how to implement it, and how to use it. Ex: The model decides to build a memory system and leverages the generic tools available to spin up a database or file systems structure as appropriate.

As models get better, harness builders naturally start to favor implementing harness features in the 3–4 buckets. This naturally increases agent entropy, making controlling, standardizing, and monitoring agent behaviors more challenging. Working mainly with enterprise agents, all the harness features we implemented landed in the 1–3 range for control and auditability reasons, with any feature related to access control and security kept in the first bucket.

Agent Harness Core Features

It is finally time to dive into the features we ended up equipping our enterprise agents.

We can divide these harness features into two categories. Let’s use a car as an analogy to motivate both categories. The car’s purpose is to get you from A to B, and the agent’s purpose is to execute (hopefully) valuable tasks delegated to it. The car, same as the agent harness, has two conflicting missions to achieve:

1. Performance: Get from A to B in the fastest way possible by converting as much raw motor power into speed, with the motor being the model and the car being the harness capabilities.
2. Safety: Keep the car from crashing or breaking down by providing reliable brakes, well maintained parts, etc. Here, the brakes would be the safety, reliability, and observability features within the agent harness.

Mission 1: Performance

A performant agent harness does two things well:

1. Gives the model access to the right context at the right time
2. Gives the model powerful and versatile tools to explore and interact with its environment.

Intuition tells us that if we want an agent to be as versatile as a human when assigned tasks, we need to equip it with the same tools that a human would use. Nowadays, most knowledge work can be done remotely via a computer and an internet connection. The idea therefore is, what if we give the agent access to a computer and an internet connection?

For the last 80+ years, the terminal has been the main text-based entrypoint into any computer, a medium that LLMs are known to be very good at. Terminal access would then allow agents to:

1. Progressively disclose information via the file system: This is done by leveraging commands like cat, grep, find, etc.
2. Persist information and work over time by writing/editing files to the filesystem: This allows the agent to progressively save data and decompose and perform complex tasks step by step.
3. Execute code via runtime environments: This allows the agent to run previously written or pre-existing scripts to execute complex predefined logic or deploy complete applications.
4. Call third-party applications: This allows the agent to call any existing tool, software, or application, provided they have a command line interface (CLI) like curl, ssh, run cron jobs, etc. to further interact with their environment to perform their test.
5. Leverage built-in Operating System (OS) features: This allows the use of the underlying hardware, access control & security, and networking native features.

Terminal access is necessary but not sufficient, as the models of today still need a bit more structure and guidance from the agent harness to efficiently use these low-level, powerful primitives provided by the terminal to reliably perform requested tasks. This is why we still equipped our agent harnesses with higher-level functionalities like:

• General system prompt providing guidance agnostic to the task at hand.
• Planning to decompose any task into manageable subtasks and track their progress.
• Skills to provide structured instruction & tribal knowledge discoverability to perform specific tasks.
• RAG to dynamically enrich the context with relevant data.
• Context management/reduction, a subsection of context engineering that allows the agent to keep their context as relevant as possible as the task horizon increases.
• Memory allows the agent to simulate remembering preference, facts, and procedures between turns and sessions.
• Self evaluation/correction loops via systems like the ralph loop that systematically reviews the work of the main agent to validate that the job is completed.

Mission 2: Safety

The faster and longer agents can do things, the faster mistakes can happen and compound. The more freedom and access the agents are given, the more dangerous those mistakes can become, which is why equipping the right agent “brakes” is equally, if not more important.

When it came time to build our agent harnesses, we made sure to provide the right level of security, reliability, and observability.

Security involves access control and prevention from malicious intent:

• User-level access control should be applied for every action performed by the model on behalf of the calling user, respecting that user’s permissions.
• A secure sandbox for code executions provides a safe environment to run agent-triggered code, separated from the existing agent environment following least privilege principles.
• Prompt injection prevention is needed for ingestion of uncontrolled or unverified data sources such as from web scraping or from external emails that should be vetted for any potential malicious intent, especially when in the lethal trifecta situation.

Reliability involves giving confidence in the agent abilities and maintainability via versioning and rollback of work capabilities and resilience in runtime execution failures:

• Versioning & rollbacks allows the user to see the history of changes and actions taken and rollback to an older state if needed, erasing all actions performed in between.
• Core harness cycle error management is the capability that allows the harness to handle errors within its cycle like LLM call errors, LLM output parsing error, tool calling errors, infinite loop detection, or context limit management.

Observability provides transparency and auditability to the overall system, all while delivering the right level of agency to the agent user:

• Agent interaction logging has two components, the agent execution logs captured in the form of a trace that contains all the steps taken by the agent and the agent runtime logs emitted by the agent’s runtime environment throughout a run.
• Live async user feedback informs the user on what is currently being worked on, what is the current plan and where human sign off is required. The right level of feedback loop can make or break your harness.

Each of these categories and sub topics could warrant their own article but it is important to know that they should be considered when designing the agent harness.

In practice, building a harness from scratch can be technically challenging, time consuming, and with the AI space rapidly changing, it is easy to quickly accumulate technical debt. Agent harnesses will also most likely end up getting standardized and commoditized in the near- to medium-term, making it an overall low return on investment to build one from scratch if it is not core to the business.

It is much more worthwhile to focus time and effort on the real agent-building value generator: the third layer of the agent building process, the configuration of the agent harnesses.

Layer 3: The Agent Harness Configuration

The agent harness provides a fast car with adequate brakes, but we are still missing the wheels, so to speak. Only when the agent is given access to the right use case/environment-specific context and tools will it actually be able to hit the road and showcase that valuable performance.

To configure an agent harness, you first need to identify and define the use case that would benefit from an agent. This exercise can end up being more challenging than it seems, so we put together a simple and intuitive framework to help answer the question, “How can you identify an agentic use case?”

Once the agent’s core five characteristics (from the framework above) have been identified and documented, the agent harness can be configured following the next six steps.

1. Giving the Agent Access to Context

Context is king. Provide the right context and the rest will follow. This can be done by giving general guidance via the system prompt (ex: AGENT.md) or providing procedural knowledge via skills, ranging from tribal knowledge to internal tool documentation to company-approved formats for building slide decks or reports.

2. Giving the Agent Access to Use Case/Environment-Specialized Tools

On top of the built-in tools to the harness, the agent needs to be given access to the right read and write tools to properly interact with its environment. READ tools are needed to query CRM data, news data, shared emails, semantic data models or unstructured shared drive folders. WRITE (action) tools allow the agent to not only inform but do things for the user like submit draft email, generate pptx/doc, build data visualizations, submit a transaction, write/run code scripts, etc.

Tools implementations can range from MCP servers to CLIs. As a rule of thumb, the more generic tools, the better. Tools can be made more or less specific based on model performance and requirements for strict guardrails or auditability. As an example, a supply chain agent could have full generic access to a CRM to “create” a PO for the user OR could be required to explicitly call a “submit PO” tool which automatically triggers the proof-reading workflow before submission.

3. Set Up the Right Level of Permissioning

The moment the agent is user facing and is connected to internal/external systems, role-based permissioning should become top of mind. For each of the previously defined tool calls, the agent should act on behalf of the calling user and respect the Role Based Access Control (RBAC) policies set on the underlying system via Oauth or at the agent harness level via pre-tool call hooks to avoid any permission leaks.

4. Human-in-the-Loop Requirements

Once you have your tools defined and set up with the right permissions, deciding on the right “intervention threshold” is crucial. Too many human interventions can cause decision fatigue and kill the calling user’s productivity, but too little can risk irreversible actions being made, non-compliance with company and legal regulations, and lost confidence in the system as a whole.

5. Equipping Sub-Agents

An agent can be powerful, but give it a team to delegate to, allowing it to handle much more complex tasks and run for longer. Equipped sub-agents can either be generalist sub-agents to perform a specific delegatable subtask or hyper specific sub-agents for specific tasks like running deep research or a full review of the main agent’s work via a curated test suite following a testing framework like Test-Driven Development (TDD).

6. Integrating the Agent Into Existing Processes

Your agent might have all the functionalities in the world, but if it does not fit into core processes and doesn’t have an appropriate interface, it won’t have the expected value add. Chat interfaces, albeit intuitive interfaces for agents, are only one way of embedding an agent and rarely the most efficient to interface with an agent. Agents can also be kept running in the background as batch automation or live triggered by events like email receipts, alerts, or other notifications.

A key takeaway is that once our agent harness was built, we didn’t really need to touch it anymore. Value starts being unlocked once the agent harness actually gets configured to fit the use case. This means, giving the agent access to the relevant context, integrations and systems required for the tasks you plan to delegate to it.

Conclusion

We rely on the agent harness to bridge the gap between the model and its environment allowing it to perform meaningful and valuable tasks the same way that we rely on the computer’s OS and programming languages to bridge the gap between high level business applications and the underlying CPU and RAM.

It is looking like LLMs will soon become a commodity (if not already) thanks to the open-source community publishing open weights of very powerful models and fierce competition between the model vendors bringing the cost of models down. Agent harnesses are the next in line as agent harness architectures will start converging, driven again by the open-source community projects like OpenClaw or Langchain’s Deep Agents, industry-leading agent harnesses like Anthropic’s Claude Code, Google’s Antigravity, OpenAI’s Codex and incentives for labs to post-train models to work well with specific harness architectures.

One thing that only time will tell is, with models starting to support more and more modalities and possibly transition over to world models, would the core harness primitives we’ve built still hold or would a new harness paradigm emerge?

Either way, at least in an enterprise setting, the best use of time and effort remains on the last mile configuration of the agent harnesses as it allows a model to effectively interact and understand your one-of-a-kind environment, your data, and your processes; all in a reliable and safe way. This is possible via solutions and platforms like Dataiku that offer prebuilt agent-harnesses-as-a-service, considerably reducing technical debt and empowering developers to focus on the last mile agent configuration to translate their business and industry knowledge into an agentic system in a centralized agent control plane.

The 3 Layers of Agent Building was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

Blog post co-written with François Phe

Building generative AI applications is no longer just about using one powerful language model. Today, the real challenge is designing systems made of several specialized LLM agents (or AI agents) that work together to solve complex problems. The hard part is not only deciding what to build, but also figuring out how to build it in practice.

You are facing a complex problem, and you know that using several LLM agents together is the right approach. But a key question quickly comes up: How should these agents be organized? Should they follow a clear hierarchy, operate as a flexible group with more dynamic, non-deterministic interactions, or adopt a hybrid approach with a portion of deterministic task flows? And which framework should you use to build this system, i.e., LangGraph, AutoGen, CrewAI, or something else?

This represents a new form of choice overload in the field. The way you organize your agents acts like a playbook: It can either help your system run smoothly or lead to confusion and poor results.

What Are Multi-Agent Systems?

A multi-agent system is a setup where several LLM agents interact with each other to solve a problem. Instead of relying on one single, all-purpose agent, you build a small team of specialists, such as a researcher, a writer, or a reviewer, who work together on the same task.

The architecture is the rulebook for this team, including:

• Who can talk to whom
• Who makes decisions
• How tasks move from one agent to another

Choosing the right architecture is critical. It directly affects how much control you have over the system and how well it can scale. In this blog post, you’ll find practical advice to help you understand these options and pick the right design and associated framework for your own application.

Defining the Landscape

Before comparing solutions, let’s clarify two key ideas:

• Architectures: How agents are organized
• Frameworks: The software libraries you use to build them

You can think of architectures as blueprints, and frameworks as toolkits.

Common Multi-Agent Architectures (The “Blueprints”)

We will focus on the most widely used MAS architectures.

Hierarchical (or Supervisor)

This is a top-down model, similar to a manager leading a team. One supervisor agent is in charge: It evaluates the request, selects the appropriate next actions, and routes well-defined tasks to worker agents.

Each agent runs, produces an output, and hands the result back to the supervisor, which then determines what to do next.

Several common forms of this hierarchical pattern include:

• Simple Supervisor (Tool Calling): In the simplest version, the supervisor treats other agents as callable tools. Using the LLM’s tool-calling feature, it selects which agent to invoke and provides the corresponding input.
• Complex Hierarchy: For larger or more complex systems, the hierarchy can be extended. Instead of one supervisor managing all agents, you may have multiple layers of supervision. In this setup, a top-level supervisor delegates work to sub-supervisors, each of which manages its own small group of agents. These “pods” allow the system to handle more complex workflows while keeping responsibilities clearly separated.

Network (or Swarm)

This is a decentralized architecture where all agents are equals and there is no single leader controlling the system. Agents communicate directly with one another, often through a shared conversation, and adjust their actions based on the existing conversation state.

Any agent can contribute at any time by proposing ideas, responding to others, or refining previous outputs. Through this ongoing interaction, agents can challenge, review, and improve each other’s work, allowing the system to gradually converge toward a higher-quality final result.

Custom Workflow

In a custom workflow architecture, you explicitly design the execution flow between agents. You decide how agents are connected and how tasks move through the system, using sequential steps, conditional branches, loops, or parallel execution. This approach does not rely on open-ended agent interactions or a central supervisor; instead, the behavior of the system is defined by the workflow itself, giving you full control over how it operates.

Popular Agent Frameworks (The “Toolkits”)

Frameworks provide the building blocks to implement these architectures.

• LlamaIndex: In LlamaIndex, an agent combines an LLM with memory and tools. You can create agent workflows using ready-made components (such as FunctionAgent or AgentWorkflow) or build fully custom, event-driven processes.
• LangGraph: LangGraph models systems as graphs. Each node represents a tool, a decision step, or an agent. Multi-agent systems are created by connecting these nodes and defining how messages and state move between them. For very common architectures, pre-built graphs exist (e.g., create_swarm, create_supervisor).
• AutoGen: AutoGen lets you create agents using the AssistantAgent class. It is excellent for conversational patterns like multi-agent debates or group chats.
• CrewAI: CrewAI is a lightweight Python framework focused on production use. It enables role-based agents with clear goals to collaborate using tools, memory, and task orchestration.
• OpenAI Agents SDK: This SDK (Software Development Kit) enables you to build agentic AI apps in a lightweight package. Its core primitives are agents, handoffs (agents delegating to other agents), and guardrails.
• Google ADK: Google’s Agent Development Kit is an enterprise-grade toolkit designed for modular, maintainable hierarchies and structured workflows.

Key Factors to Guide Your Choice

Choosing a multi-agent architecture or framework is not about picking the most popular option; it’s about matching the design to the problem you are trying to solve. Before diving into specific architectures and recommendations, it’s important to step back and think about a few core dimensions that shape how your system should behave.

Number of Agents

Start by identifying how many distinct roles or specializations your system really needs. Some problems can be handled by a small group of three to five agents, each with a clear responsibility. Others require a much larger setup, closer to a full organization, with many specialized agents working in parallel. The number of agents you plan to use will strongly influence both the architecture and the framework you choose.

Level of Decision Autonomy

Next, consider how much decision-making power each agent should have. In some systems, agents simply execute well-defined and assigned tasks. In others, agents independently determine the next action or which other agents to invoke. Higher autonomy can unlock more flexible behavior, but it also increases complexity and the risk of unpredictable outcomes.

Task Trajectory Complexity

Think about how the task is executed from start to finish. Is it a simple, sequential workflow with a clear path to the result, or does each request require complex branching decisions to reach a solution? This includes whether tasks follow a fixed order or can split into parallel paths. The more complex and variable the path to a solution, the more important the choice of architecture becomes.

Modularity & Reusability

Finally, consider how your system will evolve over time. Will you need to add new agents, replace existing ones, or reuse the same agent across multiple systems? A modular design makes experimentation, maintenance, and scaling much easier, especially as your application grows beyond a single use case.

Taken together, these questions form a practical lens for evaluating different multi-agent designs. With these criteria in mind, we can now look at relevant examples for each multi-agent architecture.

Multi-Agent Architectures: Scenario Examples

The easiest way to understand when to use each architecture is to see how they behave in real situations. Below are three concrete use cases, each illustrating a different multi-agent design pattern.

Hierarchical Architecture: Quarterly Competitive Report

You ask the system:

“Generate a Q3 competitive analysis report on ‘Competitor X’. Focus on product features, marketing campaigns, and customer sentiment.”

How it works:

• Supervisor (The “lead analyst agent”): The supervisor receives the request and breaks it into clear sections (product, marketing, sentiment).
• Specialized Agents: Each section is assigned to a dedicated agent (e.g., product analyst agent, marketing analyst agent, sentiment analyst agent).
• Parallel Execution: Agents work independently and do not need to coordinate with each other.
• Synthesis: The supervisor collects the outputs to assemble and review the final report.

This is a strong fit for a hierarchical architecture because the task is structured, predictable, and can be split into independent research efforts.

Network Architecture: Purchase Customer Support

A customer writes:

“My order #123 hasn’t arrived, and I’d also like to upgrade my subscription.”

How it works:

• Specialized Agents: The system includes a shipping agent to handle delivery issues and a billing agent to manage subscription and payment requests.
• Initial Agent: One of these agents receives the message first and determines whether it can fully handle the request or whether part of it should be routed to another agent.
• Autonomous Handoffs: After addressing the shipping issue, the shipping agent detects that the subscription request remains unresolved and transfers the conversation to the billing agent to complete the task.

This is perfect for a network architecture because the problem is unpredictable. Control naturally moves between specialists. This approach is flexible and adaptive, which is essential for conversations that cannot be fully planned in advance.

Custom Workflow Architecture: Technical Customer Support

A customer writes:

“Help! My dashboard isn’t loading”

How it works:

• Routing Step: A routing component classifies the request into technical and billing categories.
• Conditional Execution: Depending on the classification, the request is sent to either a technical support agent or a billing agent.
• Convergence Point: Regardless of the path taken, the response is sent to a quality-control agent.
• Final Polish: The quality-control agent rewrites the answer to match the company’s tone and standards before sending it to the customer.

This use case fits a custom workflow because it follows explicit logic rules and requires dynamic controlled routing while ensuring consistent quality through a final review step.

These examples highlight how different architectures excel under different constraints. In the next section, we’ll compare hierarchical, network, and custom workflow architectures across the key dimensions introduced earlier, number of agents, autonomy, task trajectory complexity, and modularity, to help you choose the right design for your own application.

The Great Multi-Agent Debate: Practical Guides & Recommendations

Now that we’ve explored concrete examples, let’s compare the three architectures across the four key dimensions introduced earlier.

Architecture vs. Technical Dimensions

Number of Agents

• Hierarchical (High): Best suited for large systems. You can organize dozens of agents into small teams managed by intermediate supervisors, keeping the overall system structured and understandable.
• Network (Low): Works best with small groups. As the number of agents grows, the system quickly becomes hard to follow, making handoffs, loops, and agent interactions difficult to track.
• Custom Workflow (Medium): Scalability depends on the complexity of the workflow graph. It handles mid-sized teams well, as long as routing logic and execution paths remain clear.

Level of Decision Autonomy

• Hierarchical (Low): Agents act as focused specialists. They execute tasks only when instructed and operate strictly within the scope defined by the supervisor.
• Network (High): Agents are independent peers. Each agent determines when its task is complete and triggers the next appropriate agent.
• Custom Workflow (Medium): Agents have freedom within clearly defined boundaries. For example, a technical agent determines the remediation steps, while a router controls whether that agent is invoked at all.

Task Trajectory Complexity

• Hierarchical (Medium): The path is centrally managed. A supervisor defines the plan and splits the work across agents, making it easy to manage multiple tasks in parallel. Individual tasks can be complex, but the overall flow stays clear, structured, and easy to debug.
• Network (Low): The path emerges naturally from agent conversations. There is no predefined plan, often moving one step at a time, which limits parallel execution. This makes the setup simple, but the sequence of actions can quickly become hard to follow or predict.
• Custom Workflow (High): The path is fully predefined. Each step, condition, and transition is explicitly designed. This requires more upfront work, but it gives you full visibility, precise control, potential parallelization and complete traceability of how decisions are made.

Modularity & Reusability

• Hierarchical (High): Individual agents can be upgraded or replaced without impacting the rest of the system. A new agent can be swapped in without changing the supervisor logic.
• Network (Medium): The system is flexible. New agents can be added easily, as long as existing agents are aware of their role.
• Custom Workflow (Medium): Modularity is tied to the workflow graph. Adding new behavior usually requires updating routing logic and defining new paths.

Framework Recommendations

Choosing a framework is about aligning its design philosophy with your architectural needs.

• LangGraph: Particularly well-suited for all architectures. Its stateful graph model gives you fine-grained control over execution paths.
• Google ADK: It excels at large hierarchical systems with strong modularity and clear management layers.
• AutoGen: A reference framework for network and debate-based systems, where agents collaborate through open-ended conversations.
• CrewAI: Well suited for role-based hierarchical teams. It offers a good balance between structure and flexibility.
• LlamaIndex: Can support all three architectures, but with varying levels of effort. It is easy to use for network-style systems, requires more setup for hierarchical designs, and involves the most complexity when implementing fully custom workflows.
• OpenAI Agents SDK: A strong choice for simple hierarchical systems.

Other elements, such as the overall enterprise technology landscape and the comfort level of the development team, will also influence the choice of framework.

Conclusion: Architecture First, Framework Second

Multi-agent systems are not about chasing the latest framework or recreating human organizations for the sake of it. They are about making complexity manageable. The biggest mistake teams make today is jumping straight to a toolkit without first deciding how agents should collaborate.

None of the three architectures described in this article are “better” in absolute terms; each one is optimized for a different trade-off between control, flexibility, and complexity. Frameworks like LangGraph, AutoGen, CrewAI, Google ADK, or the OpenAI Agents SDK simply operationalize these choices. Each framework comes with its own strengths and limitations, depending on the architecture you choose.

If there is one key takeaway from this article, it’s this: Architecture is the strategic decision; frameworks are tactical enablers.

This article is illustrated with a working implementation available on the Dataiku Public Gallery.

Multi-Agent Systems Demystified: How to Pick the Right Architecture and Framework for Your Task was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Future of AI Is About Empowering Agents With the Right Tools for the Job

It all started with a business problem so painful that it has haunted office workers for decades: responding to Requests for Proposal (RFPs) and Requests for Information (RFIs). These massive documents are a gauntlet of complex, repetitive, and interconnected questions.

A single question might require you to dig up info from three different internal documents, and the question on page 20 is probably a follow-up to one you answered back on page five. Working on this problem using an agentic RAG framework, I stumbled upon some hard-won lessons and insights that I believe can apply to a whole lot of RAG systems out there.

What Exactly Makes This Task Difficult?

Automating RFP and RFI completion means that your solution can handle these two main challenges:

• First, questions may require finding and combining information from multiple, separate knowledge sources to form a single answer. But just that would be playing on easy mode! What’s more, each retrieval often depends on previous parts of the answer, breaking any hope for parallel processing. This process of performing a series of intricate reasoning “hops” between documents to synthesize a complete response from scattered evidence is what’s known as a Multi-Hop challenge.
• Second, a later question might be a direct follow-up to a much earlier answer. This makes it necessary for the agent to maintain a conversational “thread” to handle the dependencies between questions, and breaks question answering parallelism. It is referred to as Multi-Turn questions (as this feed can be seen as a conversation between a user asking questions and an assistant answering, each speaking in turns)

Now, naive RAGs (1 retrieval + 1 generation) obviously cannot handle it, because their rigid, single-pass design lacks the necessary dynamic reasoning and memory capabilities. So, how can we do this?

Assembly Line vs. Master Chef: A Tale of Two AIs

When you build an advanced RAG system, you’re faced with a fundamental choice in its design philosophy, to define how it will reason and interact with data. This choice boils down to two competing paradigms: workflow vs. agentic.

1. Workflow: The Assembly Line

Think of this as the classic, industrial approach. The system is a rigid assembly line with a fixed, hard-coded sequence of steps, functioning as a predefined system. The AI is a highly-skilled worker on this line, performing its specific task perfectly, but it has no say in the overall process. The path is always the same, for instance: first, rewrite the query; second, retrieve the documents; third, re-rank them; fourth, generate the answer. No deviations allowed.

This is the “System 1” of AI: controllable, tunable, interpretable, but completely inflexible. In the present case, you would typically include:

• A question decomposer that breaks the complex initial query into a fixed sequence of simpler sub-questions.
• A query builder that rephrases each sub-question, often by integrating answers from previous turns to handle the multi-turn context.
• A retriever called sequentially, that execute in a predetermined order to fetch the necessary information for each sub-question.
• One or several evaluators, at various stages, to check that the model’s on track.
• A memory handler, so that you save past answers, to be used in later questions.
• A synthesizer, a final module that takes all the separately retrieved pieces of information and combines them into a single, coherent answer.

And these are just the required blocks, nothing fancy yet…

2. Agentic System: The Master Chef

This is where things get exciting. In the agentic paradigm, the AI is no longer a worker on the line; it’s the master chef running the kitchen.

The agent has complete autonomy. It looks at the user’s request, assesses the available information and tools, and dynamically creates its own recipe for solving the problem. It decides if it needs a tool, which tool to use from its toolkit, and in what order. It might start with a web search, then decide based on the results that it needs to query an internal database, all while “thinking” and adapting its plan on the fly.

This is the “System 2” of AI: deliberate, flexible, and powerful enough to tackle the messy, unpredictable problems of the real world. However, you must trust the AI ability to effectively handle those tools and reason efficiently. Expecting a master chef but getting a chaotic kitchen hand? That’s bad news for your restaurant…

So, which paradigm should you use? Luckily for you, I tested them myself, and here are the results.

Cages vs. Playgrounds: Choosing How Your AI Thinks

We have our two competing philosophies. Here’s what I did: I tested several instances of each kind, including the modular frameworks in the predefined setting, and different ReAct modules for the agentic paradigm. More specifically, I replicated the ComposeRAG architecture from this recent paper, an approach noted for its strong performance in business settings on this kind of Multi-Hop challenges, and then benchmarked it against several relevant datasets.

Then, I compared it with a ReAct agent equipped with a tool to access the same database. I won’t delve into anymore details, but rather stick to results and general conclusions that I think are safe to draw at this point.

The Assembly Line on Trial

As you’ve probably already guessed, I am not a big fan of this paradigm. The workflow approach is fundamentally rooted in a mistrust of the model’s own capabilities. It operates like a micromanager, forcing the AI down a strict, unchangeable path because it doesn’t trust it to figure out the best steps on its own. This philosophy leads to these critical flaws:

• It is inherently inefficient. This paradigm forces every problem, regardless of its complexity, through the same rigid, multi-step pipeline. This “one-size-fits-all” approach prevents the model from adapting its effort, often wasting significant computational resources on simple tasks, and lacking budget for the hardest ones. Sure, you can try to multiply the pipes, discriminating between cases, but then…
• It requires significant upfront engineering. Designing a robust, fixed pipeline involves meticulously planning a complex web of inter-module dependencies to prevent errors and optimize cooperation. Thus, complexity (hence time) grows exponentially with the number of blocks. Building this sophisticated support structure to guide the model is a time-consuming process that may become obsolete as models evolve.
• It cannot adapt easily through time. Like an assembly line built for one specific product, it lacks the flexibility to adapt to small changes in the goal or nature of the task over time. You have the right to mistrust AI’s ability to reason, but then you ought to do it yourself, and anticipate each and every corner case that could pop up. Otherwise, you may be forcing your system to put a square peg in a round hole. And if you have been engineering solutions for some time, you know that unexpected cases are bound to happen. Even worse, a single unexpected event can cause the entire pipeline to fail but take some time to debug, retracing all generations, from all modules…

The Agentic Approach Takes the Stage

This approach is the exact opposite. It’s built on trusting the model’s intelligence and giving it the autonomy to be a maverick. Give the agent its goals, the tools it can use, and let him handle the rest.

• It simplifies development and debugging. Instead of engineering a complex, rigid pipeline with numerous inter-dependencies, the developer’s role shifts to providing the AI with a clear set of tools. The agent itself handles the complex task of planning and orchestration, which can make the underlying code far simpler to write and maintain.
• It leverages the model’s superior decision-making. AI (with all due respect) is already smarter than you when it comes to tool handling, which often leads to more effective and efficient solutions than a human can predict and hard-code, as the model can adapt its strategy in real-time.
• It’s a future-proof bet. Building agentic systems aligns with the primary focus of major AI labs, as the latest and future models are being explicitly trained for autonomous tool use.

A Few Conclusions

Ultimately, this choice between the Micromanager and the Maverick is the very heart of what I call the Crutch Dilemma. A crutch is an indispensable tool for someone with a broken leg, providing the necessary support to overcome a temporary weakness. However, once the leg has healed and recovered its strength, continuing to rely on it becomes counterproductive. It actively hinders natural movement and slows the person down.

Similarly, many of the complex, engineered solutions we build to guide LLMs are essentially “cognitive crutches”. They are sophisticated systems designed to compensate for the specific, temporary weaknesses of a particular model generation.

The dilemma arises because these models are “healing” at an astonishing rate. The carefully designed crutch that helps today’s model may become a liability just a few months later. This creates a difficult choice: Is it worth investing significant effort to build a complex support structure for a problem that a simple model update might solve organically in the near future?

Back to our example: I was working on these frameworks based on OpenAI’s o4-mini and o3 when gpt-5 was released in August. So this was the perfect test to evaluate how a more powerful model would perform. Guess what: When placed into a workflow, the new model’s performance stagnated and sometimes even degraded, whereas the agentic framework allowed gpt-5 to flourish, showing massive performance gains on top of its already strong baseline.

So, when it comes to the future of AI, will you bet on control or autonomy?

The Elephant in the Room: It’s Always Been About the Data

We’ve spent a lot of time looking at the AI’s “brain,” but lurking behind every success and failure is the real kingmaker of any RAG system: the data itself!

The True Bottleneck

Let’s be clear: In the world of RAG, the bottleneck has almost always been the data. The most common reason these systems fail is not that the AI misunderstands the information it’s given, but that the retrieval system failed to provide the right information in the first place. The principle of “garbage in, garbage out” is paramount. If your data handling is flawed, even the most brilliant AI agent is doomed to fail. So was it worth discussing on reasoning frameworks if the bottleneck is elsewhere? Well, everything’s linked.

A New Paradigm: From Retrieval to Navigation

How your knowledge base is structured and accessed is the most critical question. The traditional approach is to take structured documents and flatten them into a collection of unstructured text chunks. This is a massive problem because it severs the inherent relationships between sections, destroying the document’s original, logical structure.

Think about how a human researcher works. They don’t search through a million disconnected sentences. They navigate hierarchically: skimming a table of contents, identifying relevant chapters, scanning section headings, and only then reading a specific passage. This navigational paradigm proposes giving AI agents a similar set of tools to traverse the hierarchical structure of a knowledge base.

This is where everything comes together. This advanced, navigational approach to data only works symbiotically with an agentic system. A workflow, simply cannot perform this kind of dynamic exploration natively. It can’t decide to “skim a table of contents” and then “read a specific section” based on what it finds, unless it relies on numerous, complex and brittle modules that will not fit well in this setting.

An agentic system, however, is perfectly suited for this, because it was created for this, based on an LLM trained to do so. Its ability to autonomously choose which tool to use and what to do next is exactly what’s needed to intelligently navigate a complex document structure, just like a human would.

The future of data handling doesn’t just prefer the agentic approach; it demands it.

Your Takeaway: Stop Building Pipelines, Start Empowering Agents

The future of advanced, useful AI lies not in forcing models into rigid pipelines, but in empowering them with the autonomy, and ad-hoc tools to navigate problems on their own terms.

So if you’re building a system to tackle complex Q&A, the recommendations are simple:

1. Structure your data for an AI system. Make your knowledge base easy for an agent to interact with, ideally through a well-documented API.
2. Give it the right tools. Provide the agent with simple, clear functions to access that data.

In the age of AI, the ones who truly understand the technology are the ones who will lead the race. And that means understanding that the best systems don’t constrain AI — they set it free.

I Trust My AI Agent, Do You Trust Yours? was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

It could be the first or the tenth time you’ve heard the word “agent” today but, either way, you’ve made it to this article. If you are looking to get more clarity on all this “agent talk,” then you are in the right place. We will start by demystifying the concept of agents so we can then understand where and how they can be used to generate quantifiable value.

Given that 80% of companies have at least one division using AI per McKinsey, it is likely that you have or have been asked about the questions listed above. You probably also noticed that there isn’t a real consensus around the topic and answers may vary based on people’s backgrounds, industry, or what new agent article they read that week.

Without further ado, let’s dive in.

What Is an Agent?

An agent is one of the possible ways of leveraging the raw power of generative AI (GenAI), specifically large language models (LLMs) commoditized by the likes of OpenAI, Anthropic, Google, etc.

Typical agent use cases generate value by automating tasks that have traditionally resisted automation such as solving support tickets, resolving a policy claim, building a marketing campaign, or even answering challenging questions about your data.

A GenAI application qualifies as an agent if it is LLM-powered, exhibits some degree of reasoning and, when triggered, can complete a defined task using a specific set of tools.

Let’s dive into the three core concepts of an agent: task, tools, and reasoning.

Any task has a beginning and an end, so does an agent’s execution. All agent executions will start with an input trigger such as a chat message, image, email, alert, or signal that will inform the agent of the task at hand. All agent executions will at some point end with a final answer signaling the end of the task.

Tools are what separates agents from LLMs. LLMs are great for reasoning, but their knowledge is limited to the data that they were trained on. Tools give LLMs the option to interact with the outside world elements like the internet, internal corporate data, or email servers. Tools can be categorized in three different buckets:

1. Context fetchers (Read) Ex: Internet search, vector database search, dataset query/lookup
2. Persisting actions (Write) Ex: Append to dataset, send email, run procedure
3. Orchestration (Delegate) Ex: LLM, subagent, Set of subagents, human

It is key to remember these categories as they will be needed for the agent identification framework in the following section.

And finally, reasoning, a crucial component of an agent. It gives the agent the ability to dynamically and iteratively choose the actions/tools that need to be taken to achieve the task at hand. If it was not for this reasoning, the task could not be fully automated and would require some level of human judgment or intervention to be completed.

All in all, if you have a GenAI application that has a clear task and that requires reasoning to efficiently use tools to solve a task … then you might just be looking at an agent!

Agents vs. Workflows

In the world of agent systems, there is another popular type of application commonly known as workflows. Workflows are any predefined acyclical series of steps (ex; Do A, do B, and if A + B >50, do C).

A GenAI workflow can be used for:

• Automating the application of sequential business rules.
• Applying business logic to the output of an LLM call
• Or even chaining a series of LLM calls.

Workflows, albeit very valuable GenAI applications, are not in the scope of this article but still deserve a mention as they are intimately connected to agents. As a matter of fact, agents can use workflows as part of their execution and workflows can run an agent as part of their execution.

Now that we are comfortable with agents, their three concepts, what they are, and what they aren’t, it is time to dive into the deep end and learn how to identify agent use cases in the wild!

Do Agents Apply to My Role? To My Organization?

To start off this exercise, you will need to first build up a list of as many processes/tasks that you, your team, the teams you work with, and the department and/or the company as a whole have. Don’t worry, you do not need to build an exhaustive list!

To help build the list, here are a few prompting questions that can help. Think of tasks that:

• Would take less time to validate than to do it from scratch.
• Have resisted being automated due to the unstructured nature of the data (mainly text).
• Are tasks repeated on a daily, weekly, or monthly basis and could be automated given a bit of common sense.
• Take a long time to complete given the space distribution of data owned by multiple people from various teams.
• Rely heavily on subject matter expert(s) in your organization that can act as process bottlenecks given their limited availability.

Technically speaking, agents, given the right set of tools, can be delegated any task that can be done by only interacting with a computer, so you can be ambitious on the process/tasks to consider in this list.

Once you have a healthy list of processes/tasks, it is time to identify the ones that could be delegated to an agent. For this, we want to single out processes that involve a finite task, reasoning for decision making, and that require interacting with external systems through tools (yes, these are the three core agent components we detailed in the previous section).

Let’s get started!

Counterintuitively, the first thing you will have to do is to completely forget that we are talking about agents and instead focus on something much more familiar … coworkers.

Meet Tim. Tim is a new intern who just joined your organization. Tim is smart and well-educated, but he does not know anything about your organization, its core business, its systems, or its processes. Tim also does not take initiatives when the context is too ambiguous. For each of the previously identified tasks, how would you explain the process/task to Tim?

Given the explanation you gave Tim, can he solve the task by following a predetermined set of steps (workflow) or will Tim require reasoning to choose the right company resources and tooling to accomplish the task? If the latter, the task you gave Tim is a good candidate use case to be delegated to an agent, but we are not done yet!

To confirm its agentic nature, the next step is to run this use case using the following Agent Identification Framework.

Agent Use Case Identification Framework

This Agent Identification Framework aims to qualify and define an agentic use case. After applying this framework, you will have clearly identified the following five characteristics that fully define an agent.

1. The Input/Trigger of the agent (what is the agent’s starting point).
2. The Objective of the agent (what should come out of the agent).
3. The Scope of the agent (what are/aren’t its responsibilities).
4. The exhaustive list of Tools the agent needs to use to achieve its goal.
5. The Playbook/Guidelines that the agent should follow.

This framework is a two-step process:

• Step I: Identify and document a set of representative examples of agent runs.
• Step II: We extract the five agent characteristics listed above.

Step I: Identify & Document Representative Examples

To start with the exercise, we need to go back to Tim and identify a set of representative end-to-end examples of the task (typically three to five) that he would need to accomplish. For each example, we want to identify the step-by-step process that Tim would go through to successfully complete the task.

Each example description should contain the following three elements: input, output, and process.

Input (Initial Instructions)

Imagine you’ll be on vacation for a week and you need to provide Tim with an email that has enough details to start on a given task without needing further assistance from you.

Output (Final expected state)

Describe the output state of the completed task including all the actions performed by Tim and the final output answer. Think of it again as an email that Tim would need to write back to you. It should contain enough information for you, coming back from your two-week holiday to validate that the task has been done, and done well.

Process (Step-by-Step Actions to Solve the Task)

List the set of Thought, Action, & Observations that Tim needs to go through to complete the task, from the provided input all the way to the expected output state. Use the following guide to properly document each Thought, Action, and Result.

Thought: Given the current state (everything done so far), what would Tim’s next best action be? Specify the reasoning behind it. For example:

• I need to check my unread emails to see if anyone has contacted me.
• I need to search the web as I need public information on the government’s website.

Action: Explicitly define the next action needed including any specifications

• Open inbox and filter for unread emails in the last 30 days.
• Call the Google search engine with the query “Top 5 enterprise agent building platforms.”

Result: Describe the outcome of the action. This is the new information that Tim has to decide what the best next step is.

The final action of any task is to “end task” and return a final output message.

Sales Analysis Agent Example

Business context and objective: Regional sales leaders want answers to their elaborate questions that should rely on data scattered across multiple internal information sources (SharePoint and Snowflake).

Example Case #1

Input: (Text) What are the top five revenue-generating customers in the Americas region over the last three years?

Output: (Text) The top five customers in the Americas over the last three years are X, Y, Z, A, & B, in that order.

Process

Step 1:

• Thought: The request requires me to leverage data at my disposal. I first need to list all the available tables and their schema descriptions to isolate the relevant tables.
• Action: List accessible tables that can be leveraged from the “sales data schema” database.
• Result: List of all sales tables and their schema’s metadata.

Step 2:

• Thought: From the available schemas, the table called product_sales in Snowflake I will need to leverage the customer_id, year, and amount columns to answer the question, given we are in 2025.
• Action: Generate and run SQL query with input “Filter the product_sales for the region Americas and transaction between 2023 and 2025, group data by customer_id, and sum the revenue. Keep only the top five customer_id.”
• Result: Resulting summary table listing the five customers.

Step 3:

• Thought: I know the answer to the user’s question, I am ready to answer.
• Action: End task.
• Result: Answer the question with the retrieved answer.

Example Case #2

Input: (Text) Hello, how are you?

Output: (Text) I am good, thank you, I am your AI Sales Assistant, is there anything I can help you with?

Process

Step 1:

• Thought: The user query does not require any internal knowledge. I can reply directly and propose to be helpful.
• Action: End task.
• Result: Answer with generic helpful message.

Once enough examples have been provided to give a representative sample of the types of tasks the agent might have to tackle, it is time to wrap up the exercise with Step II!

Step II: Extract the 5 Agent Characteristics

Lucky for us, the input/trigger, objective (output state), and the scope have already been documented for us through the examples above.

The input/trigger can be inferred from all the example inputs documented above. Here the agent is triggered by the regional sales leaders submitting a question about their sales data.

The objective will be the combination of persistent actions (write tools) and the agent’s response for all inputs. Here, the agent needs to leverage available sales data to answer all questions from the sales regional leader.

The scope of the agent is the set of tasks of the same nature as the ones documented above. In the case you identified edge-case tasks that were not part of the examples above, feel free to update Step I accordingly.

The remaining two elements to extract are the tools and agent’s playbook/guide.

To extract the tools, go through all the Action elements in the examples and classify each as:

1. No tools needed: All the information is already known, nothing to do.
2. Tool is needed: Add/update the tool to the list, including:
   > Tool name (short and explicit)
   > Tool description (when to use the tool)
   > Tool input parameters (its signature)
   > Tool output (content and format)
   > Tool category (Read, Write, or Delegate)
   Note: Make sure to consolidate tools together, as your agent probably used the same tool multiple times.
3. Tool needs clarification: An action is needed, but we don’t have enough information to define the five descriptors of a single tool.
   > This means that the action is too complex and should be split into different tools or delegated all together to a subagent.

In the examples above, the final tools would be:

You now have an exhaustive list of tools (and possibly subagents) waiting to be built!

To define the playbook/guidelines of the agent, also known as the agent prompt, you will have to go through all of the Thought sections of your examples. For each thought, take note of all the logic based on external business knowledge and not inferred from neither the previous tool call results nor from common knowledge (i.e., internal business process logic, special naming conventions, specific personality, acronyms, etc).

You now have all the context and guidance needed by the agent ahead of each task ready to be added to the agent prompt!

If you’ve made it this far, congrats! You now know how to find and narrow down potential agents, rank them by their business value, and run them through the Identification framework on your own.

Conclusion

We learned what an agent was and what differentiated it from its counterpart agentic system: workflows. We learned how to build a shortlist of potential agent use cases and how to leverage the Agent Identification Framework to concretely define the scope, roles, and responsibilities of these agents.

How Can You Identify an Agentic AI Use Case? was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Guide to Test-Time Compute Scaling to Unlock Smarter, More Accurate AI Without Breaking the Bank on Retraining

For decades, the playbook in AI has been simple: Bigger is better. Got a problem? Build a bigger model. Need more accuracy? Throw more data and a few million dollars at it, retrain through backpropagation, and there you are. This relentless scaling gave us incredible tools, but we’ve started to hit a massive, expensive, and frustrating “Scaling Wall.”

Training the next generation of trillion-parameter models now costs more than a Hollywood blockbuster (we’re talking >$100 million), requires more high-quality data than we can find on the whole internet, and is often plagued by instability.

Post-training helped for some time (you know, fine-tuning the model so that it aligns with our goals, our way of thinking, etc.), but eventually hit a wall as well (see figure below). So, researchers and tech firms thought, “What if, instead of building a bigger brain, we could just teach the brains we have to think harder?”

This is the promise of Test-Time Compute (TTC), the next frontier in AI scaling that blossomed in 2025. Get ready to understand how it works (and whether you should use it or not).

What on Earth Is Test-Time Compute?

Let’s break it down with a simple analogy: Imagine that your AI model is a brilliant student.

• Training Time is their entire education — years of learning, reading books, and taking exams. This is where the model learns its skills. It’s incredibly expensive and time-consuming.
• Test-Time (or “inference”) is … test time! When you give that student a single, tough question, and he/she must now use their knowledge to understand the question, its structure and underlying ins and outs, and answer it in a comprehensive way.

The old way was to just hope the student’s education was good enough to blurt out the right answer instantly.

A (hopefully) better way is giving that student a piece of scratch paper and telling them, “Hey, take your time. Think about this step by step. Write down your thoughts before you give me the final answer”. This is one possible technique based on TTC scaling, usually called Chain-of-Thoughts (CoT), but there are others. We’ll delve into the details in a minute.

Before this, let’s give TTC scaling a proper definition, breaking it down term-by-term:

• Test-Time: It’s the phase where training is done, and weights of the model are frozen.
• Compute: The amount of computing power needed, usually expressed in FLOPs (floating-point operations per second). Remember: LLMs are basically huge neural networks, composed of thousands of matrices undergoing basic operations (multiplication, addition, etc.). So, in the context of LLMs at test-time, compute is directly related to the number of matrix calculations you need. As each new generated token implies a pass in the model, longer generation (more tokens) means more compute.
• Scaling: As you’d imagine, tech firms do not want to spoil money on supplementary compute if it does not lead to increased performance. So, the goal is to find techniques that let us continue riding these nice scaling laws (see figure below).

So, TTC scaling is just this: using more compute at inference to boost accuracy.

So, does it work? Short answer is … yes, and quite dramatically!

The Verdict Is In: TTC Scaling Works

The concept has been discussed for several years, initially in the context of board games and AlphaZero algorithms. However, it gained significant traction in the LLM space following two key events in late 2024 and early 2025:

• OpenAI’s o1: In September 2024, OpenAI’s release of o1 marked a major breakthrough for the reasoning paradigm.
• Deepseek’s R1: In January 2025, the open-sourcing of the Deepseek R1 model enabled wider community experimentation with these techniques.

For years, poor performance on the ARC-AGI benchmark — a test of abstract reasoning using visual puzzles, specifically designed to test raw reasoning abilities — fueled the argument that LLMs couldn’t truly “think.” The consistent failure of even advanced models like GPT-4 suggested they were merely sophisticated mimics incapable of genuine generalization.

This long-held belief was shattered by OpenAI’s “o1” and then “o3” models (see figure above). They achieved a sudden, massive jump in performance on ARC-AGI, directly challenging the perceived limits of AI reasoning and reigniting the debate about their potential for true intelligence. And guess what, the key change was the use of TTC scaling!

These models also outperformed any previous ones in a series of very challenging benchmarks (competition-level maths, Ph.D.-level science questions, etc.). As a consequence, all major LLM providers propose reasoning models (we will see in a minute what they are).

Because they’ve proved so effective, we’ll now look at how these reasoning models have been trained.

Learning to Reason

Training an LLM to reason goes beyond simply showing it a massive library of text (which had basically been the training paradigm until then). The process is a deliberate, multi-stage effort to teach the model how to construct logical arguments and solve problems step by step.

At its core, the training relies on specialized datasets and sophisticated reward mechanisms.

• Curated Datasets With Explicit Reasoning: The model isn’t just trained on the final answers to problems; it’s fine-tuned on high-quality, step-by-step annotated data. These datasets contain problems (in mathematics, logic, or code) paired with detailed, human-verified reasoning chains. This teaches the model the structure of a logical argument, not just the correlation between a question and its answer. For example, the GSM8K dataset, a popular benchmark, includes grade-school math problems with fully worked-out solutions.
• Process Supervision in Reinforcement Learning: Standard Reinforcement Learning from Human Feedback (RLHF) might only reward a model for getting the final answer correct (outcome supervision). This is risky, as the model could learn flawed logic that happens to work for some problems. Reasoning models, however, are often trained using process supervision. In this method, a human reviewer evaluates and rewards each individual step in the model’s generated reasoning chain. This is far more labor-intensive but ensures the model learns a robust and reliable reasoning process, penalizing logical fallacies even if the final answer is accidentally correct.
• Synthetic Data Generation and Self-Correction: To scale the creation of reasoning data, techniques like Self-Taught Reasoner (STaR) are employed. Here, an LLM is used to generate its own reasoning chains for a set of problems. If the final answer is correct, the generated chain is added back into the training data as a positive example. This creates a powerful self-improvement loop.
• Architectural and Algorithmic Enhancements: The training process is designed to support more complex inference-time algorithms. While a standard LLM generates text token-by-token in a linear fashion, reasoning models are trained to be amenable to more structured exploration. This enables techniques like Tree of Thoughts (ToT), where the model can explore multiple parallel reasoning paths (branches of a tree), evaluate their potential, and backtrack from unpromising ones. This requires the model to be proficient at generating self-evaluations and scoring intermediate “thoughts.”

These are the main building blocks when training a powerful “native” thinker: At inference, the model will spontaneously produce CoT detailing its reasoning path before generating a final answer.

But keep in mind that reasoning models are just one way to leverage CoT, a specific technique based on TTC scaling. The main principle is increasing the number of generated tokens at inference: There are multiple approaches to this, and some can even be combined. We will see now a taxonomy of the main techniques.

The TTC Toolkit: How to Make Your AI “Think”

TTC isn’t a single method; it’s a whole galaxy of clever techniques. Most of them fall into three main families, but keep in mind that this is not exhaustive.

1. CoT: The “Show Your Work” Method

This is the most popular TTC technique and also the way it emerged historically. Instead of letting the model jump straight to an answer, you prompt it to generate a “chain of thought” — a series of reasoning steps that lead to the solution.

It can be done by giving it examples in your prompt (in-context learning) or by simply adding a magic phrase like “Let’s think step by step.”

This is at the core of what we call reasoning models, including OpenAI’s “o” series: models that have been specially trained on thousands of CoT, to replicate this behavior. To the point that, at inference, they resort automatically to this drafting: They “think” natively before answering, without the need to nudge them.

2. Sampling: The “Phone a Friend … or 50” Method

Why get one opinion when you can get many? Sampling techniques involve asking the model the same question multiple times to generate several different answers, which are then combined to find the best one.

• Self-Consistency: You generate, say, 10 answers and pick the one that appears most often. It’s a simple majority vote that dramatically boosts reliability.
• Best-of-N: You generate N answers and then use a second, smaller model (a “verifier” or “reward model” trained specifically for this task) to score each one and pick the winner.
• Beam Search: A most advanced way to navigate into the tree of thoughts (where branches are feeds of successive thoughts) while only keeping most promising leads.

3. Criticism & Feedback: The “Draft, Revise, Repeat” Method

This is my personal favorite. It turns the generation process into an iterative loop of refinement.

1. The LLM generates an initial draft of the answer.
2. The LLM (or another model) critiques that draft, pointing out flaws or areas for improvement.
3. The LLM takes the feedback and generates a new, better version.

You can loop this process until the answer is perfect, or you’ve hit your budget limit. A variation is using a powerful “teacher” LLM to guide a smaller, cheaper “student” model, correcting its reasoning only when it starts to go off track.

Now let’s talk about when to use these techniques.

To TTC or Not to TTC?

TTC techniques are more suited for certain tasks and are unlikely to help for others. Choosing the right technique is all about diagnosing the type of difficulty you’re facing, as each family aims at solving a different kind of failure. Here are basic heuristics:

CoT-based techniques, including the use of reasoning models, are adapted for tasks within the model’s known capabilities but with intricate and multiple steps. Here’s a simple rule of thumb I call The Human Expert Test.

Ask yourself: If I gave a human expert all the necessary information to perform a task, could they solve this task instantly, on the fly?

• YES? (e.g., live translation, answering a simple factual question, writing a basic code snippet). Then TTC probably won’t help much. If your LLM is failing, the problem is likely somewhere else — it probably doesn’t have the right information.
• NO? (e.g., summarizing a messy 50-page document, solving a multi-step math problem, planning a complex project). If the expert needs to pause, think, and organize their thoughts, that’s a perfect use case for TTC! Letting the LLM “think” will allow it to break down the problem and deliver a much more accurate result.

When the model is generally capable of a task but suffers from a lack of consistency, sometimes succeeding, other times hallucinating facts or getting stuck in a suboptimal reasoning path, Sampling techniques may help.

Finally, feedback can be used for novel tasks, likely falling outside the model’s training distribution. If it involves novel concepts, or requires adhering to very strict and unusual constraints, then checks and corrections can improve your results.

With all that said, a question remains: Should you DIY your own TTC techniques? The answer is unclear and depends greatly on your level of expertise, the task you are working on, the model you use, the budget you have, etc.

Keep in mind that the most powerful, mind-bending reasoning techniques are already being baked directly into the flagship models from the big leagues. They’re serving you five-star models, and they are getting better with every update.

So you could spend days or weeks of your time (and money) building a brilliant custom “thinking” engine for your AI, just in time for the next big model to be released, offering even better reasoning right out of the box and turning your masterpiece into a museum piece overnight. Furthermore, LLMs training processes are notoriously hard to predict, and the clever trick that boosts your accuracy today could become completely useless when the model’s mood shifts tomorrow.

Another point is that generation is not that often the real bottleneck in business use cases, not so much as data quality and availability. Most benchmarks build on very clean environments where models have access to every information they need, under the right format, in controlled settings, etc. This is not always (never?) the case in real life! So before any expensive experimentation, it is worth checking where the real bottleneck of your system lies.

Conclusion

So, did we go through all of this just to learn that the best move is to … wait for the next model? Was it even worth it?

Well, first, there are still a lot of cases where TTC scaling can be applied successfully, so it might be worth trying depending on factors we highlighted below.

But more importantly, TTC isn’t just a temporary hack; it’s part of the new playbook for cutting-edge AI. Understanding how it works is your ticket to keeping up with the next wave of tech breakthroughs. Even if you let the big providers handle the complex implementation, knowing the blueprint is how you’ll be ready for what comes next.

Finally, it is a massive leap forward, but it is not a silver bullet. Models with reasoning abilities still struggle with things like physical common sense, recognizing when a task is impossible, or adapting to new procedures on the fly. So, having a feeling of how all this works let you use these models accordingly, understanding their strengths and weaknesses.

In the age of AI, the ones who truly understand the technology are most probably the ones who will lead the race.

Sources

All three illustrations without indicated source are from A Visual Guide to Reasoning LLMs. Maarten Grootendorst, 2025.

Additional resources:

• What is test-time compute and how to scale it? Ksenia Se, Alyona Vert, 2024.
• Scaling test-time compute with open models. Edward Beeching, Lewis Tunstall, Sasha Rush, 2024.
• Learning to reason with LLMs. OpenAi team, 2024.
• A Visual Guide to Reasoning LLMs. Maarten Grootendorst, 2025.
• Why We Think. Lilian Weng, 2025.

AI Hit a Wall and the Solution Was to Teach It to Think Longer was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

An Overview of Modern Approaches for the Enterprise

Organizations accumulate vast amounts of key information, much of which is locked away in documents. These documents — whether they are reports, contracts, invoices, or emails — are typically designed for human consumption, making them difficult to process automatically. Fortunately, Document AI, the subfield of AI focused on documents, is making rapid and significant progress. In this post, we offer a glimpse into the possibilities offered by modern Document AI. More specifically, we:

• Clarify the scope of Document AI;
• Present its main approaches;
• Illustrate how key Document AI tasks can be concretely performed;
• Introduce a Dataiku sample project that implements most of the techniques discussed.

Please note that this post does not cover the creation of question-answering systems over multi-page documents since a previous article of Data From the Trenches already covers this topic.

What Is Document AI?

Document AI, or Document Understanding, refers to automatically analyzing scanned or natively digital documents in order to categorize them or extract information from them. Some of the key Document AI tasks are:

• Optical Character Recognition (OCR): The conversion of the text present in images into machine-encoded text;.
• Layout Analysis: The analysis of the structure of a document, notably the detection and classification of elements like paragraphs, headings, tables, and images.
• Document Classification: The categorization of documents into predefined classes (e.g., forms, invoices, emails) based on their content, such as invoices, forms, or emails.
• Visual Question Answering: A task where a model answers questions based on the visual content of a document.
• Key Information Extraction: The process of extracting specific structured data (like names, dates, or price) from unstructured documents.

Handling documents is challenging for three main reasons: they generally combine text and images (e.g., diagrams, charts); their structure is often implicit and only suggested through their layouts; and they come in a variety of formats (e.g., layout, font family, font size). To overcome these obstacles, Document AI takes advantage of methods from both computer vision and natural language processing, alternatively seeing documents as just images, just texts, or combinations of both.

Layout Analysis

In this section and the following ones, we illustrate with some examples how Document AI tasks can be performed with open source packages, open weight models, or proprietary API-based LLMs. We first start with layout analysis, which is an important prerequisite to receiving meaningful OCR results from documents with non-trivial layouts.

Current approaches for layout analysis rely on computer vision models fine-tuned for object detection specifically in documents. Typical classes are “title”, “text”, “page header”, “section header”, etc. For example, the unstructured Python library makes several of these models available and easy to use, along with other features to parse and process various document formats.

OCR

When it comes to OCR, a straightforward approach is to use a specialized deep learning model such as the recently published open weight transformer model GOT or the LSTM models embedded in the well-known Tesseract open source library. These models are lightweight, fast, and effective.

However, their performance depends on the practical circumstances of the use case. Whether they perform well for certain languages and for both handwritten and printed text depends on their training dataset. Tesseract is quite ineffective on handwritten documents for instance. Furthermore, image pre-processing steps may be required, such as:

• Isolating the various text boxes in the document with layout analysis.
• Correcting the text orientation with the Hough transform;
• Removing visual noise (i.e., details that are irrelevant for the task at hand);
• Adjusting the scale, brightness, and contrast of the document.

The recent emergence of powerful multimodal LLMs (M-LLMs) offers an alternative to these specialized OCR models. With M-LLMs like GPT-4o or an open-weight model like InternVL, we can simply include the target image in the prompt and ask the model to extract its text content. The level of performance varies with the context of the use case as with specialized models but it is generally very strong.

However, M-LLMs sometimes exhibit two unwanted behaviors. First, they can add their own comments, such as “The text in the image is…”, to the text extracted from the document. Second, they may correct some words deemed to be misspelled in the original document or even “hallucinate” some content. Both of these problems may require adjusting the prompt or post-processing the LLM output.

Document Classification

M-LLMs, e.g., Qwen2-VL or GPT-4o, are also strong options to classify documents in a predetermined list of categories (e.g., article, email, letter, form). In this case, the prompt would include the instruction to classify the document, the list of classes, the document page to classify, and potentially few-shot examples (i.e., some document pages and their ground-truth labels). Since we expect only one of the specified classes, a good practice is to use structured text generation techniques, such as function calling or structured outputs, to mitigate or cancel the risk of an invalid answer.

If a sizable training dataset is available, it is also possible to fine-tune a model. UDOP (Universal Document Processing), an open-weight model, is a strong candidate for Document AI tasks. UDOP uses an OCR engine (Tesseract by default) and processes text, layout, and images as inputs. It is a versatile model, capable of performing various tasks such as document classification, layout analysis, visual question answering, key information extraction, and document generation. It is usually better to use parameter-efficient fine-tuning (PEFT) techniques such as LoRA, especially if the training dataset is relatively small.

Visual Question Answering

Here, we restrict ourselves to questions whose answers come from a single page provided along with the question. Question-answering systems which cover whole documents generally require a multimodal Retrieval-Augmented Generation (RAG) approach which is already addressed in a previous blog post.

M-LLMs like Qwen2-VL or GPT-4o are the method of choice for visual question-answering tasks, given the general nature of the potential questions, the need to generate some text content and the need to take into account both visual and textual elements of the page. They may be fine-tuned if the domain at hand is very specific but recent M-LLMs generally yield good answers, even in a zero-shot setting.

Evaluating the quality of question-answering systems is especially challenging because equally valid answers can be phrased in very different ways and the best approach is generally to use an LLM-as-a-judge technique, i.e., to ask an LLM to assess the generated answers and compare them to ground-truth answers when available.

Key Information Extraction

In a key information extraction task, we are given a document page, e.g., the scan of a receipt, and we try to extract specific and predetermined pieces of information, e.g., the name and address of the company, the date of the receipt, the amount of the receipt, etc. M-LLMs (e.g., Qwen2-VL or GPT-4o again) are also strong candidates for this task, thanks to their text generation capabilities which are useful when the answer is quite open-ended like in this task. Fine-tuning can help but is often not necessary.

In this context, M-LLMs are typically prompted to generate a JSON object whose properties correspond exactly to the target fields. Even more so than for document classification, structured text generation techniques like function calling or structured outputs are essential to comply with the expected format.

Document AI in Dataiku

Dataiku users can start exploring the various techniques mentioned above thanks to a public and reusable sample project of the Dataiku project gallery. In particular, this project shows how to:

• Perform layout analysis and OCR with Unstructured, InternVL and GPT-4o;
• Fine-tune UDOP or leverage GPT-4o or Qwen2-VL for document classification;
• Answer questions based on documents with GPT-4o or Qwen2-VL and evaluate the corresponding answers;
• Extract specific fields from documents with GPT-4o or Qwen2-VL;
• Expose these capabilities in a simple web application.

Conclusion

Recent progress in Document AI enables the automation of an increasingly wide range of tasks. Use cases that would have been particularly challenging a few years ago become achievable. In this context, M-LLMs distinguish themselves with their versatility, user-friendliness, zero-shot or few-shot capabilities, and high performance and will likely play an increasingly important role in this domain. Still, some alternatives remain valid, especially for real-time use cases or in low-resource environments when the computational costs of M-LLMs are excessive.

Automating Document Processing With AI was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

Your Guide to Structured Text Generation

Large language models (LLMs) are like wild animals — powerful and versatile, but unpredictable and potentially dangerous. This makes deploying robust LLM applications challenging. In this blog post, we present the notion of structured text generation, which enables practitioners to “tame” LLMs by imposing formatting constraints on their outputs. More precisely, we will:

• List the main types of formatting constraints that structured text generation techniques can handle.
• Explain the benefits of structured text generation.
• Provide an overview of the structured text generation methods.
• Present the most commonly available features for open or proprietary models.
• Describe various pitfalls and the practical ways to avoid them.

A Spectrum of Possible Constraints

Structured text generation methods are available for four main categories of formatting constraints. The simplest one is restricting the LLM’s outputs to a predefined set of options. For example, when implementing an LLM-as-a-judge approach, we may want to generate a score from 1 to 5, in which case we would expect only five answers: “1”, “2”, “3”, “4”, and “5”.

More general constraints can be expressed through regular expressions. The most typical example is the generation of a JSON object adhering to a specific JSON schema. For example, if we perform sentiment analysis on online reviews, the expected LLM response may be a JSON object with two properties: “sentiment” which is a string with three potential values (“positive”, “negative”, and “neutral”) and “themes” which is a list of strings.

An even wider family of constraints is based on formal grammars, which are particularly interesting when we want to obtain syntactically correct computer code through an LLM.

Finally, formatting constraints can take the form of templates, which are dynamic, fill-in-the-blank texts whose placeholders are meant to be filled by an LLM.

Enforcing one of the constraints described above may bring various benefits:

• Guaranteed Validity: For certain tasks such as text classification or key information extraction, formatting constraints ensure that the answer is valid: All requested fields are present, they can only take allowed values and no unexpected fields have been added. For example, if we are extracting certain fields from a scanned invoice with a multimodal LLM, we may want to systematically get the company name, the total amount, and the date of the invoice. In this case, the answer would be invalid if one of these pieces of information were missing. Note that even with valid formatting, the values within the response can still be incorrect.
• Effortless Parsing: LLM outputs often contain multiple parts that need to be isolated and fed into a user interface or a downstream application. Being able to generate text following a data serialization format like JSON, XML, or YAML makes this process much easier and more reliable.
• Enhanced Reasoning: Adding structure requirements can also lead to more relevant responses. The simplest example is the chain-of-thought method, which involves prompting the LLM to provide a justification before its final answer. This not only makes its output easier to understand and check, but it also has been shown empirically that this approach improves the accuracy of the response.

Prompt Engineering and Fine-Tuning

LLM developers have a toolbox of techniques at their disposal to impose formatting constraints. The simplest approach relies on prompt engineering. This involves explicitly stating the desired format in the prompt or providing examples. This could be as simple as writing “Respond in JSON format” or as detailed as providing a full JSON schema.

While easy to implement, this method often results in too many formatting errors. Moreover, adding more instructions to the prompt increases its length, which leads to higher costs and longer response times.

For stricter adherence to formatting constraints, we can turn to supervised fine-tuning. This consists of further training a pre-trained LLM on a dataset of well-formatted examples. This method yields a high level of compliance, but without increasing the prompt size. It can even allow developers to use smaller, more efficient LLM models, further reducing costs and latency. However, fine-tuning requires significant upfront effort. It demands expertise, a base LLM, a well-prepared training dataset, and computing resources (or a fine-tuning feature offered by the provider of an API-based LLM). The training process itself can take hours, delaying the moment when the LLM is ready to be used.

A special and advanced case of fine-tuning is function calling. Some LLMs (including the most prominent proprietary LLMs like GPT-4o and several open-weight models) have been specifically trained to call functions, allowing developers to extend their capabilities. Imagine wanting the LLM to provide the current weather in a city. The LLM itself does not have access to real-time weather data, but you can define a function that does. By including a description of this function in the prompt, the LLM can be trained to recognize when to call this function and how to provide the necessary input in the correct JSON format.

While this feature is usually used to execute the function and feed the results back to the LLM, it can be cleverly repurposed for enforcing formatting constraints. Instead of a real function, developers can define a dummy function that never actually gets executed but has input parameters structured exactly in the desired format. When the LLM “calls” this function, it generates a JSON object that perfectly matches the constraints. This approach achieves high compliance without needing any training data, but it requires the LLM to have been specifically fine-tuned for function calling and formatting errors remain possible, even if much less likely.

Constrained Decoding

The most powerful structured text generation approach is constrained decoding, which ensures 100% compliance even for complex constraints. It operates by masking incompatible tokens during the text generation process. For example, let’s imagine we prompt an LLM to generate a score on a scale from one to five. In the standard approach, the prompt is converted into a sequence of tokens, which are just words or subwords. These tokens are ingested by the LLM, which returns a vector providing the probability for each potential token to be generated next. The “next token” can then be obtained by sampling this probability distribution. It will likely be “1”, “2”, “3”, “4”, or “5”, but this is not guaranteed.

With constrained decoding, we automatically build a mask vector which assigns a one to each token compatible with the constraint — in this example, “1”, “2”, “3”, “4”, and “5” — and a zero to all other tokens. If we multiply the probability vector and the mask vector term-wise and then normalize the resulting vector, we obtain a new probability distribution. By sampling this probability distribution instead of the original, we get a next token that will necessarily respect the constraint.

The example above is simplistic because the constraint is limited to a few potential choices and each of these choices corresponds to only a single token. Fortunately, the token matching approach is generalizable to all the categories of constraints listed above. For example, with a constraint based on a regular expression, we can build a deterministic finite automaton corresponding to the regular expression, adjust it so that it operates on tokens rather than characters and use it to keep track of the tokens already generated and determine the next tokens permitted by the constraint.

Overall, constrained decoding guarantees perfect compliance, does not require training data, and can even speed up text generation compared to unconstrained decoding. It introduces some initial overhead (for example to build the automaton in the case of a regular expression) but it is insignificant in most use cases. The disadvantages of constrained decoding are that it is only possible with a proprietary API-based LLM if the provider offers this feature. Constrained decoding also creates some pitfalls described below.

Structured Text Generation in Practice

In practice, the most popular structured text generation techniques available through open-source packages or features of proprietary LLM APIs are the following:

• Structured Output corresponds to constrained decoding with a specific JSON schema and 100% compliance. It is the most useful technique for concrete use cases.
• Function Calling also uses JSON schemas, but does not ensure perfect compliance.
• JSON Mode is a form of constrained decoding guaranteeing a valid JSON output (but without any JSON schema specified).

These features are very simple and convenient, but it is easy to overlook potential pitfalls. A major source of troubles with constrained decoding is the fact that computing the next token probabilities (before token masking) does not depend on the constraints. In a way, the LLM is “unaware” of the constraints.

In the first of the examples above, we ask an LLM who is the Hand of the King in the second season of Game of Thrones. Without constraints, the LLM is likely to write a paragraph that starts with “The Hand of the King” and includes the correct answer: “Tyrion”. However, if we restrict the potential choices to either “Tyrion” or “Theon”, the answer will incorrectly be “Theon”. This may seem strange because the unconstrained response suggests that the LLM “knows” the right answer, but this behavior is actually explainable given the way constrained decoding works.

Since “The” is a likely first token and “The” is compatible with the respect of the constraint (because “The” is a prefix of “Theon”), it will not be censored by the token mask and it will likely be selected again as the first token. But then, “Theon” becomes the only permitted answer compatible with the constraint. By enforcing the constraint, we unwittingly increased the odds of “Theon” simply because this particular name happens to have a very common English word as a prefix.

In the second example, adding a constraint to obtain a date may lead to a “1” being selected for the first token. This first token is compatible with both a DD/MM/YYYY or a MM/DD/YYYY format so it would not be masked with either of these two constraints and the LLM assigns a high probability to “1” because a likely answer starts with “14”. But with the MM/DD/YYYY format, only “0”, “1”, or “2” are allowed for the second token, so it is impossible to generate a correct answer at this point.

In the third example, we also unintentionally set the LLM on a track that leads to failure. Since tokens are generated one after the other and the LLM does not take into account the constraints when computing the next token probability, the probability to generate “Absolutely” does not depend on the fact that the constraint is “either ‘Absolutely not’ or ‘Definitely’” or “either ‘Absolutely’ or ‘Definitely not’”. Once the first token is generated, there is no choice anymore and having selected “Absolutely” for the first token leads to two opposite answers depending on the constraint.

The best remedy to this problem is to specify the constraint in the prompt, even when leveraging constrained decoding. This limits the gap between the next token probability distribution in the unconstrained vs. constrained case and increases the likelihood that the answer will reflect the internal knowledge of the LLM.

Some other pitfalls and the associated mitigating strategies are the following:

• Detrimental impact on quality: Some anecdotal evidence suggests that imposing constraints might sometimes decrease the quality of LLM outputs for tasks involving logical reasoning or mathematical problem-solving. If this happens and cannot be solved through prompt engineering or by adjusting the constraint, a good approach is to decompose the task into two parts. In the first step, generate the answer without any formatting constraints and in the second step, translate this answer into a properly formatted response.
• Hidden feedback: The formatting constraint may prevent the LLM from communicating important information. When left unconstrained, the LLM may explain the limitations of its answers or why it may not answer. For example, an LLM may abstain from providing an answer because of the cutoff date of its training corpus. All this useful information may be hidden when a formatting constraint is imposed. If some strange behavior is observed, it is useful to temporarily turn off the constraint to diagnose and solve the problem and then reactivate the constraint.
• Impact of the order of JSON key-value pairs: While the order of key-value pairs in a JSON object is irrelevant from a data perspective, LLMs can be sensitive to this order. This is because the generation process unfolds sequentially, and earlier parts of the output can influence later parts. Therefore, it is recommended to structure JSON objects so that the content corresponding to the reasoning process of the LLM (e.g., a justification for an answer) is generated before the content corresponding to the outcome of this reasoning process.
• Limited JSON schema support: The various constrained decoding implementations do not support all features of the JSON schema specification. For example, the Structured Outputs feature of the OpenAI API does not allow for optional fields. It is important to consult the relevant documentation to avoid surprises, especially if the JSON schema is complex.
• Underestimating fine-tuning: While constrained decoding already guarantees compliance, fine-tuning remains a valuable tool. With regard to the first pitfall mentioned above, fine-tuning can be a way to reduce the discrepancy between the unconstrained and constrained behaviors of the model, without lengthening the prompt. It can also help the LLM understand the semantic nuances of the target format. For example, in the case of a sentiment analysis task with three potential classes (“positive”, “neutral”, and “negative”), constrained decoding is enough to guarantee that only these responses will be generated but it cannot make the LLM grasp the somewhat subjective and arbitrary distinction between “neutral” and “positive”, or “neutral” and “negative”.

Conclusion

Structured text generation is no longer an option; it is a cornerstone of building robust and reliable LLM applications.

In particular, constrained decoding techniques guarantee 100% compliance. Powerful and simple implementations are available for both open-weight LLMs and proprietary API-based LLMs. Constrained decoding creates some risks, but developers can adequately manage them if they understand the basics of these techniques.

Dataiku empowers practitioners to tame the powerful, unpredictable nature of LLMs, with safe and precise control. Dataiku provides seamless access to both top LLM services and self-hosted models, along with tools like function calling, JSON mode, and soon, structured output. Whether leveraging open-source tools or the latest APIs, these features allow you to turn LLM potential into structured, impactful content ready for enterprise-grade applications.

Dive deeper into the essentials of structured text generation and learn firsthand how to control and optimize LLM outputs for real-world applications in our webinar, “How to Get 100% Well-Formatted LLM Responses.”

Taming LLM Outputs was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most interesting Generative AI trends is the development of agents powered by large language models (LLMs). The word “agent” denotes an automated system that can perceive its environment and take actions. More specifically, in the context of LLMs, as explained in a recent academic article entitled “AI Agents That Matter,” this word is used mainly when one or several of the following conditions are met:

• The surrounding environment is complex and the potential tasks are open ended.
• The system can be given directives in natural language.
• The system can act with limited or no supervision from a human user.
• The system can use external tools.
• The control flow of the system is dynamic.

The purpose of this blog post is to present some of the most popular open source Python frameworks used to implement LLM-powered agents. We will focus on the following frameworks and we will highlight their similarities as well as their differences (and summarize them in a table in the appendix):

• LangGraph: A general, low-level library of the LangChain ecosystem
• LlamaIndex: A framework specialized in the implementation of retrieval-augmented LLM pipelines
• AutoGen: A framework that models an LLM application as a conversation between multiple agents
• CrewAI: A multi-agent framework that emphasizes user-friendliness

Features Common to All Frameworks

Let’s start with the similarities between these frameworks. First, they all offer the possibility to implement one or several predefined execution logics. The execution logic defines the sequence of LLM completion queries, tool invocations, and user interactions that allows an agent to complete its task. ReAct is a popular example of agent execution logic. All frameworks above also allow developers to create customized execution logics.

Another key feature is the possibility to define tools that agents can use to interact with their environment. From the perspective of an agent, a tool is simply a function that is described in the prompt, can be called with certain input parameters and returns a text output. Typical tools are search engines, web browsers, code interpreters, or messaging services. All frameworks let the developers use off-the-shelf tools and easily create customized tools.

They also include at least some human-in-the-loop features. Agents are prone to errors, especially if they need to follow a relatively high number of steps to perform a task. This is especially worrisome if the agents use tools which not only return information but also trigger side effects (e.g., an administrative task which affects the account of a customer). In this context, it may be critical to involve a human user, either to ask clarifying questions on the requested task or to request a confirmation before taking a sensitive action.

On the technical side, all the frameworks leave users the flexibility to easily switch from one LLM API to another. They offer an integration with the OpenAI API and, by extension, they allow developers to connect to the increasingly common OpenAI-compatible APIs. Furthermore, it is relatively straightforward to define custom LLMs and there are also several other LLM integrations available.

Finally, an unfortunate commonality among the four frameworks is that they offer limited agent evaluation features, if any. Depending on the use case, we may want to only evaluate the final answer provided by an agent or its whole trajectory, including the intermediate steps (especially if tools with side effects are used), and we may have access to a ground truth or not. The four frameworks allow developers to fully record the agents’ trajectories but none of them include evaluation features that encompass these four potential scenarios.

Single Agent or Multi-Agent Systems?

CrewAI and AutoGen are presented as multi-agent frameworks while LlamaIndex focuses on single agent systems, and LangGraph enables both approaches. In a multi-agent system, several specialized agents collaborate to reach a common objective. The execution logic of the system is then framed as a “conversation” between the agents. The agents can, for example, “speak” one after the other or a “manager” agent can give the floor to the agents when relevant.

It makes sense to replace a single agent with multiple agents when it would be impossible or impractical to impart a single agent with all the capabilities required for the use case at hand. For instance, a prompt that includes the instructions and examples for all the target capabilities might be exceedingly complex for the LLM or simply too large for its context window. While managing a single agent is conceptually simpler, combining multiple agents is more modular: We can develop and test independent specialized agents and combine them to better scale with the complexity of a use case.

Single agent frameworks are actually also able to model multiple agents because an agent can be packaged as a tool that is then made available to another agent. In this way, the distinction between single agent frameworks and multi-agent frameworks lies more in the developer experience than in their expressiveness.

Distinctive Features of Agent Frameworks

We now focus on the specific aspects of each framework covered in this blog post.

LangGraph has recently become the preferred way of implementing agents in the popular LangChain ecosystem. LangChain includes some agent-related features but many of them are deprecated and planned to be removed in the near future. The promise of LangGraph is the possibility to implement arbitrary agent execution logic in a simple manner. This execution logic is declared as a graph which specifies how the state of the agent evolves over time. For example, the graph below corresponds to a tool-augmented agent that requests the validation of a human user for certain sensitive tools. The key features of LangGraph are:

• The control the behavior of the agent in a very fine-grained manner
• The seamless integration in the LangChain ecosystem (in particular, the very large number of available built-in tools)
• The creation of checkpoints that capture the past and present states of the agent for monitoring and error recovery purposes
• The evaluation of agents’ trajectories, to assess the relevance of individual actions (which does not, however, cover the case when a ground truth for the full trajectory is available)

LlamaIndex is a framework specialized in the creation of LLM applications that rely on external data sources. LlamaIndex gives the possibility to create agents among many options to better leverage these data sources. Please note that the LlamaIndex team is currently building llama-agents, a “powerful framework for building production multi-agent AI systems” but we do not cover it in this blog post because it is in an early stage. Some of the most distinctive agent features of LlamaIndex are:

• Several pre-configured agents that can directly be used (function calling agent, ReAct agent, and more advanced agent algorithms like Language Agent Tree Search or Chain-of-Abstraction)
• The compatibility with other LlamaIndex components and, in particular, the possibility to create tools based on LlamaIndex data query engines

AutoGen is a multi-agent framework developed by Microsoft. Its central high-level abstraction is the conversation, which corresponds to the execution logic of the multi-agent system. Noteworthy features of AutoGen include:

• A wide range of conversation patterns (e.g., group chat with multiple agents, nested chats, speaker selection, etc.) that gives flexibility to create agent workflows of various complexities
• Several options to involve one or several human users
• A built-in code execution component that offers several options (local Shell, Docker Shell, Jupyter Kernel) to let an agent execute automatically generated code

CrewAI is a developer-friendly multi-agent framework. It represents multi-agent systems as “crews” of specialized agents and its abstractions reflect notions applicable to human teams, such as tasks, processes, collaboration, management, planning, delegation, etc. CrewAI notably offers the following features:

• Definition of the agents and their interactions in a simple yet flexible manner
• Management of the memory of the agents with four options (short-term memory, long-term memory, entity memory, contextual memory)
• Compatibility with LangChain LLMs, LangChain tools, and LlamaIndex tools

Conclusion

Although the use of LLM-powered agents is new in real-life use cases, there are already solid options to start implementing agent applications. In this blog post, we presented four agent frameworks with distinct strengths. At this stage, our preliminary generic recommendations would be:

• Use LangGraph when the need to define the execution logic in a very fine-grained manner is paramount (for example, in the case of complex human-in-the-loop interactions).
• Use LlamaIndex for data-augmented LLM applications (e.g., question answering applications), especially when speed of development and maintainability trump flexibility.
• For tasks that are naturally decomposed in several, well delineated sub-tasks, use CrewAI or AutoGen depending on the desired balance between speed of development and precise control.

Whatever framework is chosen and given the LLMs currently available, agents will in any case remain brittle and will need to be carefully designed, tested, and monitored. As illustrated in the Dataiku LLM Starter Kit, Dataiku can support the development of agent applications through various features:

• LangChain integration which ensures all LLMs accessible through Dataiku can be used with LangGraph, LlamaIndex, and CrewAI while a custom LLM can be created for AutoGen
• Function calling support
• LLM evaluation (Early Adopter Program)
• Programmatic access to the various objects in Dataiku through the Python API, which facilitates the creation of custom tools
• Easy creation and operation of web user interfaces
• Experiment tracking with MLFlow.

Appendix: Overview of the Frameworks

A Tour of Popular Open Source Frameworks for LLM-Powered Agents was originally published in data from the trenches on Medium, where people are continuing the conversation by highlighting and responding to this story.