<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Royi Aizenstain on Medium]]></title>
        <description><![CDATA[Stories by Royi Aizenstain on Medium]]></description>
        <link>https://medium.com/@royi.aizen?source=rss-a010249ae54e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6Szc2g3E3Rabh4BZ1nbylQ@2x.jpeg</url>
            <title>Stories by Royi Aizenstain on Medium</title>
            <link>https://medium.com/@royi.aizen?source=rss-a010249ae54e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:27:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@royi.aizen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evaluating BeeAI Agents: Building a DeepEval + Ragas Integration]]></title>
            <link>https://medium.com/@royi.aizen/evaluating-beeai-agents-building-a-deepeval-ragas-integration-0b2e071fcc20?source=rss-a010249ae54e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b2e071fcc20</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm-as-a-judge]]></category>
            <category><![CDATA[evaluation]]></category>
            <dc:creator><![CDATA[Royi Aizenstain]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 14:34:19 GMT</pubDate>
            <atom:updated>2026-10-12T18:04:16.383Z</atom:updated>
            <content:encoded><![CDATA[<p>Testing whether an AI agent “works” is a different problem than testing whether a function returns the right value. An agent’s output is prose, its correctness is partly semantic, and its path to an answer (which tools it called, in what order, with what arguments) matters as much as the final text.</p><p><a href="https://github.com/i-am-bee/beeai-framework">BeeAI Framework</a> is an open-source toolkit for building multi-agent systems in Python and TypeScript: agents that reason, call tools, and collaborate, backed by a unified interface across 26+ LLM providers (OpenAI, Anthropic, Ollama, VertexAI, Bedrock, and more). We recently added an evaluation module to it (currently Python-only) that lets you score agents systematically instead of eyeballing transcripts. I want to walk through the actual design decisions behind it, not just what it does but why it’s shaped the way it is.</p><h3>Why not build our own evaluation framework?</h3><p>The first temptation with any framework is to invent your own metrics system. We didn’t, for a simple reason: <a href="https://docs.confident-ai.com/">DeepEval</a> and <a href="https://docs.ragas.io/">Ragas</a> already solve the hard parts, and solve them well.</p><p>DeepEval is a general-purpose LLM evaluation library, think “pytest for LLM outputs,” with metrics for answer relevancy, faithfulness, tool correctness, and an easy path to writing your own. Ragas focuses specifically on retrieval-augmented and agentic pipelines, with metrics built around the idea that an answer is only as good as the context it was grounded in. Both are async-native, both support LLM-as-judge scoring (having a model grade another model’s output against a rubric), and both already carry years of tuning on things like “how do you phrase a judge prompt so the score is stable across runs.” Reinventing that ourselves would have meant reinventing all of it, badly, from scratch.</p><p>So instead, we’ve added native LLM-as-judge (LLMaaJ) support to BeeAI: a way for these frameworks’ judge metrics to call <em>*any*</em> model BeeAI already supports (Ollama, OpenAI, VertexAI, Bedrock, whatever a user has configured), instead of requiring a separate, framework-specific API key and client setup. That’s a bridge, not a framework. So that’s what we built: two adapters.</p><h3>Two adapters, for maximal robustness</h3><p>It would be tidy if DeepEvalLLM and InstructorRagasLLM shared a common base class. They don&#39;t, and that&#39;s deliberate: the two frameworks define genuinely different LLM contracts, and papering over that difference would produce a leakier abstraction than just writing two small, honest adapters.</p><p>DeepEval’s DeepEvalBaseLLM expects generate()/a_generate() to return a <strong>string</strong>:</p><pre>class DeepEvalLLM(DeepEvalBaseLLM):<br>    async def a_generate(self, prompt: str, schema: TSchema | None = None) -&gt; str:<br>        response = await self._model.run(<br>            [UserMessage(prompt)],<br>            response_format=schema.model_json_schema(mode=&quot;serialization&quot;) if schema else None,<br>        )<br>        return response.get_text_content()</pre><p>Ragas’s InstructorBaseRagasLLM, on the other hand, expects a <strong>validated Pydantic object</strong> back. It&#39;s built around the <a href="https://github.com/instructor-ai/instructor">Instructor</a> pattern of structured extraction:</p><pre>class InstructorRagasLLM(InstructorBaseRagasLLM):<br>    async def agenerate(self, prompt: str, response_model: type[T]) -&gt; T:<br>        response = await self.model.run([UserMessage(prompt)], response_format=response_model)<br>        if not isinstance(response.output_structured, response_model):<br>            raise ChatModelError(<br>                &quot;The model failed to produce structured output matching the requested schema.&quot;,<br>                context={&quot;output&quot;: response.get_text_content()},<br>            )<br>        return response.output_structured</pre><p>Both adapters lean on the same underlying mechanism: BeeAI’s ChatModel.run(..., response_format=...), which already validates and repairs structured output internally. But they hand back fundamentally different shapes because that&#39;s what each framework actually needs. An early version of the Ragas adapter manually extracted JSON from the raw text with string slicing and a bit of regex. Once we noticed response.output_structured already did exactly this, more robustly, the manual extraction just disappeared: fewer lines, and one less thing that could silently break on a slightly-off model response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*635KHU35cDM2CjKr30flig.png" /><figcaption>One ChatModel, two adapters: DeepEval accepts text or an optional schema, Ragas always expects a validated Pydantic object.</figcaption></figure><h3>What “evaluating an agent” actually means</h3><p>Evaluating an agent means running it against a set of known test cases, questions where you already know a correct (or acceptable) answer, and then scoring what comes back. In its simplest form, that’s just a dataset: a list of (question, expected_answer) pairs and a way to check the agent&#39;s actual answer against the expected one.</p><p>But an agent isn’t a pure function. It reasons over multiple steps, decides which tools to call and in what order, and produces prose rather than a single deterministic value. So “did it get the right answer” is only one axis worth scoring. Did it call the tools it needed to, and skip the ones it didn’t? Did it cite the right supporting facts, or land on the right answer for the wrong reason? Two agents can produce the identical final text and still deserve very different scores once you look at how they got there.</p><p>That’s the actual shape of the problem BeeAI’s evaluation module is built to handle: define a dataset once, run an agent against it, and score the result from several angles at once (answer correctness, tool usage, factual grounding) rather than a single pass/fail check.</p><h3>A complete, runnable example</h3><p>Adapters and metrics are only useful if there’s something real to point them at, so alongside the adapters we built a full working example: an agent, a bundled dataset, and two ready-to-run experiment scripts (one for DeepEval, one for Ragas) that evaluate the same agent against the same questions. It’s meant to be the thing you actually run first, before you build your own dataset or agent:</p><pre>pip install &quot;beeai-framework[evaluation]&quot;<br>cd python/examples/evaluation<br>EVAL_CHAT_MODEL_NAME=openai:gpt-4o python deepeval/experiment.py</pre><p>That runs the full bundled dataset through the agent and prints a pass/fail summary with a score per metric. Swap `deepeval/experiment.py` for `ragas/experiment.py` (after a one-time `python ragas/dataset.py` to build its dataset file) to see the same questions scored through Ragas instead.</p><h3>Why HotpotQA for the bundled dataset</h3><p>The example dataset needed to exercise the interesting part of agent evaluation: multi-step reasoning across separate information sources, not single-lookup Q&amp;A. <a href="https://hotpotqa.github.io/">HotpotQA</a> is a well-known multi-hop QA benchmark built exactly for this. Each question requires combining facts from two or more separate Wikipedia articles.</p><p>Rather than write synthetic questions, the bundled example uses two real HotpotQA records, verified against the actual dataset, not just written in a similar style. But HotpotQA’s own schema carries more than an agent evaluation needs: a context field with every sentence from every candidate article, including irrelevant &quot;distractor&quot; paragraphs meant to test retrieval precision in the original benchmark&#39;s own use case (which isn&#39;t ours), plus a supporting_facts field that points into context by title + sent_id index rather than carrying the actual sentence text.</p><p>We simplified this down to five fields an agent and its metrics can use directly: question, expected_answer, supporting_sentences (the pointed-to sentences resolved into real text, not indices), supporting_titles, and expected_tool_calls (how many distinct Wikipedia lookups the question requires, which has no HotpotQA equivalent since it&#39;s specific to scoring <em>our</em> agent&#39;s tool usage). The distractor paragraphs and HotpotQA&#39;s own difficulty/type metadata got dropped; they&#39;re not wrong to have, just not needed for this example&#39;s scope.</p><h3>Walking through the example</h3><p>It helps to see this against one real question from the bundled dataset:</p><blockquote><em>Which magazine was started first, Arthur’s Magazine or First for Women?</em></blockquote><p>Neither answer is in the question, and no single Wikipedia article settles it either: you need the founding date of <em>Arthur’s Magazine</em> and the founding date of <em>First for Women</em>, then a comparison. That’s the multi-hop shape the whole example is built around: two independent lookups, combined into one answer.</p><p>The agent under evaluation is a `RequirementAgent` with three tools and a tightly-scoped role: `WikipediaTool` and `OpenMeteoTool` from the framework, plus a small example-only `CalculatorTool` defined alongside the example code itself.</p><pre>def create_agent(model_name: str | None = None) -&gt; RequirementAgent:<br>    llm = ChatModel.from_name(model_name, {&quot;allow_parallel_tool_calls&quot;: True})<br>    return RequirementAgent(<br>        llm=llm,<br>        tools=[WikipediaTool(), OpenMeteoTool(), CalculatorTool()],<br>        memory=UnconstrainedMemory(),<br>        role=(<br>            &quot;You are an expert Multi-hop Question Answering (QA) agent. &quot;<br>            &quot;Your primary role is to extract and combine information from the provided context &quot;<br>            &quot;to answer the user&#39;s question. Answer in json format only.&quot;<br>        ),<br>        instructions=[<br>            &quot;1. SOURCE ADHERENCE (NO HALLUCINATION): base your answer ONLY on tool output, &quot;<br>            &quot;never external knowledge.&quot;,<br>            &quot;2. MULTI-HOP: perform multi-step reasoning or use multiple tools if the question requires it.&quot;,<br>            &quot;3. FINAL FORMAT: respond with a single JSON object only: answer, tool_used, &quot;<br>            &quot;supporting_titles, supporting_sentences, reasoning_explanation.&quot;,<br>        ],<br>    )</pre><p>WikipediaTool is the one doing real work here; OpenMeteoTool and CalculatorTool exist so the agent has a couple of other tools it <em>shouldn&#39;t</em> reach for on this kind of question. That matters, because one of the metrics scores whether it called the <em>right</em> tools, not just any tools.</p><p>The dataset entry for this question carries everything needed to grade the answer once it comes back:</p><pre>{<br>  &quot;question&quot;: &quot;Which magazine was started first Arthur&#39;s Magazine or First for Women?&quot;,<br>  &quot;expected_answer&quot;: &quot;Arthur&#39;s Magazine&quot;,<br>  &quot;supporting_sentences&quot;: [<br>    &quot;Arthur&#39;s Magazine (1844–1846) was an American literary periodical published in Philadelphia...&quot;,<br>    &quot;First for Women is a woman&#39;s magazine published by Bauer Media Group in the USA.&quot;<br>  ],<br>  &quot;expected_tool_calls&quot;: 2,<br>  &quot;supporting_titles&quot;: [&quot;Arthur&#39;s Magazine&quot;, &quot;First for Women&quot;]<br>}</pre><p>Run the agent on this question, and a passing trace looks up both magazines on Wikipedia, compares their founding information, and returns &quot;Arthur&#39;s Magazine&quot; as the concise answer, with supporting_titles matching the two article titles it actually consulted. From there, the metrics take over: AnswerLLMJudgeMetric checks the answer is right (semantically, not just character-for-character), ToolUsageMetric checks it called Wikipedia twice as expected rather than guessing after one lookup, and FactsSimilarityMetric checks the cited sentences actually match what the dataset says justifies the answer. Three different questions about the same run, three different scores: that&#39;s the whole reason to run more than one metric in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3o1O--uJ36I6wgcrnVxZg.png" /><figcaption>One run, three checks: each metric reads a different part of the trace and compares it to the HotpotQA ground truth.</figcaption></figure><h3>Where the built-in metrics stop, and why we wrote three more</h3><p>DeepEval and Ragas ship strong general-purpose metrics: answer relevancy, faithfulness, context recall. What they don’t ship is anything specific to a given agent’s contract. Alongside the adapters, the bundled example under `examples/evaluation/` includes three example metrics that show the pattern, each filling a specific gap:</p><ul><li><strong>AnswerLLMJudgeMetric</strong>: semantic equivalence between the agent&#39;s answer and the expected one, judged by an LLM rather than exact string matching (paraphrasing shouldn&#39;t count as wrong).</li><li><strong>ToolUsageMetric</strong>: whether the agent called the expected tools the expected <em>number</em> of times, which matters specifically for multi-hop questions where under-calling a tool (stopping after one Wikipedia lookup when two were needed) is a distinct failure mode from calling the wrong tool.</li><li><strong>FactsSimilarityMetric</strong>: semantic similarity between the facts the agent actually cited and the facts it was expected to cite, independent of whether the final answer text happened to match.</li></ul><p>Each extends the relevant framework’s own metric base class and uses the adapter to do its LLM-judge scoring, so a custom metric is just as portable across providers as a built-in one. Writing your own metric doesn’t mean giving up the “use any BeeAI-supported model” property; it’s the same adapter, reused.</p><h3>What a bad score tells you</h3><p>The real value of running several metrics on the same trace is that each failure points somewhere different:</p><ul><li><strong>Low </strong><strong>AnswerLLMJudgeMetric</strong>: the most direct signal. The agent just got it wrong, time to look at the model, the prompt, or the question itself.</li><li><strong>Low </strong><strong>ToolUsageMetric, correct answer</strong>: the agent got lucky. It skipped a lookup it needed or called a tool it didn’t, worth fixing even though the final text looked fine.</li><li><strong>Low </strong><strong>FactsSimilarityMetric, correct answer</strong>: the one to take most seriously. The agent landed on the right text while citing weak or unrelated supporting facts, a quieter form of the same problem as hallucination: right conclusion, ungrounded reasoning.</li></ul><p>None of these show up if you only check whether the final answer string matches; that’s the whole case for scoring from more than one angle.</p><h3>What this actually buys you</h3><p>None of this is exotic engineering: it’s two adapters, a simplified dataset, and three metrics. But that’s kind of the point. The interesting problem in agent evaluation isn’t inventing new statistics, it’s making the <em>plumbing</em> boring enough that you can focus on what the numbers are actually telling you. pip install &quot;beeai-framework[evaluation]&quot;, wrap a ChatModel, and DeepEval or Ragas can judge your agent&#39;s answers using the same model config you already have. No separate API keys, no framework-specific client setup, no format mismatch to debug at 2am.</p><p>The adapters live at `beeai_framework.evaluation.adapters`, ready to import and reuse in your own metrics. The three example metrics above, along with the bundled multi-hop QA dataset and agent, live under `examples/evaluation/` in the repo — not in the installed package — if you want to see the whole thing wired together or use them as a starting point for your own.</p><p>This work was done under the guidance of <a href="https://medium.com/u/91ca279f8848">Antonpuz</a> (IBM Research Israel) and in collaboration with <a href="https://medium.com/u/9010cc4639f6">Omri Meroz</a> 💪🏻</p><p>Thanks <a href="https://medium.com/u/b433d5cc36bf">👨‍💻 Tomáš Dvořák (Tomas2D)</a> for the code review :)</p><p>Github: <a href="https://github.com/i-am-bee/beeai-framework/pull/1453">https://github.com/i-am-bee/beeai-framework/pull/1453</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b2e071fcc20" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>