<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shion Honda on Medium]]></title>
        <description><![CDATA[Stories by Shion Honda on Medium]]></description>
        <link>https://medium.com/@shionhonda?source=rss-8a15808834c4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*znqPdcP0sFBQv1lbv3UBoQ.png</url>
            <title>Stories by Shion Honda on Medium</title>
            <link>https://medium.com/@shionhonda?source=rss-8a15808834c4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:58:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shionhonda/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Measuring the Responses API on Alan’s Support AI agents]]></title>
            <link>https://medium.com/alan/measuring-the-responses-api-on-alans-support-ai-agents-b4ca04a4818a?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4ca04a4818a</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 15:10:43 GMT</pubDate>
            <atom:updated>2026-08-03T06:10:01.367Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2025, OpenAI introduced a new way to call its models: <a href="https://developers.openai.com/api/reference/resources/responses">the Responses API</a>. Their pitch is that switching from the legacy <a href="https://developers.openai.com/api/reference/chat-completions/overview">Chat Completions API</a> to the Responses API makes reasoning models work better. OpenAI’s own reported number is a <a href="https://developers.openai.com/cookbook/examples/gpt-5/gpt-5_prompting_guide#reusing-reasoning-context-with-the-responses-api">5% lift on Tau-bench retail</a>.</p><p>However, as we discussed in <a href="https://medium.com/alan/benchmarking-ai-agents-stop-trusting-headline-scores-start-measuring-trade-offs-0fdae3a418cf">our previous post</a>, we should not blindly trust headline scores. Benchmarks rarely look like production, and the size of the gain depends on the problem, the prompt, and the harness. That’s why we ran our own A/B test on two of our <a href="https://medium.com/alan/how-we-built-alans-ai-assistant-for-customer-support-2a82194df8be">Support AI agents</a> to compare.</p><p>The result: Responses API client delivered a <strong>+17% relative lift in automation rate</strong> at the aggregate level, which is statistically significant. This post explains what the API actually changes and what our test measured.</p><h3>The Responses API: what is it?</h3><p>When you call a reasoning model, the two APIs differ in whether you can carry the reasoning traces to the next API call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6Hf9AEq8qT7LlewH2wvvg.png" /><figcaption>Chat Completions API doesn’t carry reasoning traces to the next request. Responses API does.</figcaption></figure><p><strong>Chat Completions API (legacy)</strong> is stateless. Every call, you re-send the entire conversation. The model reads it, replies, and forgets. To keep context across turns, you feed the model’s own output back in on the next call.</p><p>This is a problem with reasoning models. Providers don’t return the chain-of-thought, so it never comes back to you to re-send. The model throws away its reasoning at the end of each turn and starts from scratch on the next one.</p><p><strong>Responses API</strong> is stateful. The provider keeps the context on its side, including reasoning, tool calls, and intermediate steps. You reference the previous response with previous_response_id and the model picks up where it left off.</p><p>If you prefer zero data retention as we do, you can use Responses API as a stateless one and ask it to return the reasoning as an encrypted blob*. You can pass it back on the next call, so you get reasoning continuity across turns without OpenAI persisting your data.</p><p><em>*Model providers often encrypt reasoning traces so users can’t reverse-engineer their reasoning mechanism.</em></p><p>The Responses API also ships built-in tools — web search, file search, a code interpreter, and MCP — which are useful even for non-reasoning models.</p><h3>The test</h3><p>We A/B tested the two APIs on two of our Support agents, both running GPT-5.1*. Let’s call them <strong>Agent X</strong> and <strong>Y</strong>. They differ in their system prompts and reasoning effort (X: minimal, Y: low). Each contact was randomly assigned to the Chat Completions or the Responses, and the primary metric was the automation rate: the share of contacts the agent resolved without escalation to a human.</p><p><em>*Why GPT-5.1 and not a newer model? We ran the test before GPT-5.5 was available. There’s also a less-known constraint worth flagging: from GPT-5.4 onward, the legacy Chat Completions API no longer supports reasoning together with tool calling (c.f., </em><a href="https://docs.litellm.ai/docs/providers/openai#openai-chat-completion-to-responses-api-bridge"><em>LiteLLM doc</em></a><em>). For modern reasoning agents that call tools, </em><strong><em>Responses API is effectively the only option</em></strong><em>.</em></p><h3>Results</h3><p>First, the automation rate. <strong>The aggregate relative lift is +17%</strong>, and it’s statistically significant.</p><p>The effect shows up on both agents, including Agent X at minimal reasoning effort, so carrying reasoning traces to the next turn is useful even when the reasoning effort is minimal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*k63fAXZqDWTwaHk2wkqEdQ.png" /><figcaption>The aggregate relative lift in the automation rate is significant.</figcaption></figure><p>How does carrying reasoning traces change the reasoning behavior? The share of LLM calls that did any reasoning dropped from <strong>94% to 88%</strong> when we switched Agent Y to Responses API. That drop is good news: because the model can reuse prior reasoning across turns, it skips re-reasoning on calls where it already has what it needs.</p><p>The average number of reasoning tokens per reasoning call stays flat (within the confidence interval). This means the Responses API doesn’t shrink the reasoning on a given call; it respects the reasoning effort you set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*Vq0zaOkuqJp0vF0XQ254ZQ.png" /><figcaption>The average number of reasoning tokens per reasoning call stays flat (within the confidence interval)</figcaption></figure><p>While OpenAI doesn’t give us raw LLM reasoning texts, we can ask it for human-readable reasoning summaries. It doesn’t always produce summaries, so we calculated the frequency for each reasoning effort setting. The result is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*Y3aQSFZ2b0Zx07wPe9ANzg.png" /></figure><p><strong>Summaries are only produced at low reasoning effort or above</strong>. At minimal, the model still reasons and still returns encrypted reasoning content, but we never received a summary. So don’t use the presence of a summary to detect whether the model reasoned.</p><h3>Concluding remarks</h3><p>Our 17% lift is much larger than OpenAI’s 5% on Tau-bench. We didn’t do a full analysis on those two experiments, but our best guess is that our use case benefits more from consistent reasoning than Tau-bench does. Tau-bench only checks the final database state, so the agent can iterate freely until it lands in the right place. Our Support agents are far more constrained: a handful of loops per turn, and a single nonsensical reply escalates the ticket and costs us the automation. Our system prompts are also much longer and more intricate than Tau-bench’s, so consistent reasoning likely helps the model navigate our prompts. For reference, the reported gains vary widely by setup: OpenAI also cites <a href="https://developers.openai.com/cookbook/examples/responses_api/reasoning_items">+3% on SWE-bench</a>, while MiniMax claims <a href="https://www.minimax.io/news/why-is-interleaved-thinking-important-for-m2">+36% on Tau-bench</a>.</p><p>[Edit on 2026–08–03] <a href="https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/">A more recent OpenAI experiment</a> provides further evidence that the agent harness can matter as much as the underlying model. On ARC-AGI-3, retaining reasoning across turns and replacing rolling truncation with compaction increased GPT-5.6 Sol’s score from 13.3% to 38.3% while reducing output tokens by roughly 6x. Although this is a very different task from customer support, the mechanism is consistent with our results: agents perform better when they can preserve what they have already learned across a long trajectory.</p><p>One caveat is that the API is OpenAI-specific. Other providers expose their own ways to preserve reasoning across turns (<a href="https://platform.claude.com/docs/en/build-with-claude/extended-thinking#preserving-thinking-blocks">Anthropic</a>, <a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/thought-signatures">Google</a>, <a href="https://docs.mistral.ai/studio-api/conversations/reasoning/native">Mistral</a>), but the shapes have diverged a lot since the Chat Completions days. Standardization efforts like <a href="https://www.openresponses.org/">Open Responses</a> exist but aren’t adopted by the other major providers yet, so there is a lock-in risk to keep an eye on.</p><p><strong>Takeaway</strong>: if you run a reasoning agent on Chat Completions today, you’re throwing away the model’s reasoning at every turn. Switching to the Responses API (or your provider’s equivalent) lets it carry that reasoning forward. For a constrained, multi-turn use case like ours, that was worth +17%. If your agent re-reasons from scratch each turn, that’s a clear thing to test.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4ca04a4818a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/measuring-the-responses-api-on-alans-support-ai-agents-b4ca04a4818a">Measuring the Responses API on Alan’s Support AI agents</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking AI Agents: Stop Trusting Headline Scores, Start Measuring Trade-offs]]></title>
            <link>https://medium.com/alan/benchmarking-ai-agents-stop-trusting-headline-scores-start-measuring-trade-offs-0fdae3a418cf?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fdae3a418cf</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Mon, 03 Nov 2025 14:46:57 GMT</pubDate>
            <atom:updated>2025-11-03T15:20:26.095Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/alan/benchmarking-ai-agents-the-challenge-of-real-world-evaluation-6aa1c2aa4b41">our previous post</a>, we discussed the challenge of evaluating AI agents and introduced Tau-bench as a promising approach for stateful agent evaluation. This post takes a deeper look at what happens when you actually try to use public benchmarks to make production decisions.</p><p>Imagine you’re choosing an AI model for your customer support team. You check the latest model releases: OpenAI showcases GPT-5’s performance on Tau-bench, while Anthropic claims Claude Sonnet 4.5 beats GPT-5 on the same benchmark. You might be tempted to use Claude Sonnet 4.5, but these numbers tell only part of the story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qk2_3pLAATQtztFQWzdlug.png" /><figcaption><em>Left: </em><a href="https://openai.com/index/introducing-gpt-5/"><em>Results reported by OpenAI</em></a><em>. Right: </em><a href="https://www.anthropic.com/news/claude-sonnet-4-5"><em>Results reported by Anthropic</em></a><em>.</em></figcaption></figure><p>Published benchmark scores often obscure critical details like optimized prompts, unspecified reasoning efforts, and different evaluation settings. When companies report results without documenting their configurations, direct comparison becomes impossible.</p><p>We reproduced Tau-bench ourselves to clarify the details. What we discovered is that the real value isn’t in chasing headline numbers. Instead, you need to map the score-latency-cost frontier for your specific use case. This post shares our methodology, results, and the lessons we learned about why we should be careful about using published benchmark scores.</p><h3>A Quick Primer on Tau-bench</h3><p>In <a href="https://medium.com/alan/benchmarking-ai-agents-the-challenge-of-real-world-evaluation-6aa1c2aa4b41">our previous post</a>, we introduced Tau-bench as one of the few public benchmarks that evaluates stateful tool use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSsRDTTuuWG9wi-Db_00Dg.png" /><figcaption><em>An overview of </em><a href="https://arxiv.org/abs/2406.12045"><em>Tau-bench</em></a><em>. It persists user states in a database, and the agent makes read/write operations to the database via tool calling.</em></figcaption></figure><p>Unlike benchmarks that only validate tool-call syntax, Tau-bench checks whether agents actually complete tasks correctly by verifying both the final answer and the resulting database state. For example, if a user requests a cancellation, the benchmark confirms the order status changed to “cancelled” in the database, not just that the agent called the right API.</p><p>Each task involves an AI agent, a database (implemented as local JSON files) with API tools, and a user simulator (which is also an AI). The agent must gather information through conversation, follow domain-specific policies (retail, airline, or telecom rules), and execute database operations via tool calls. The user simulator follows a scenario instruction but doesn’t see the agent’s tool calls. This mimics realistic information asymmetry where customers don’t know what’s happening behind the scenes.</p><p>The newer Tau2-bench extends this with a dual-control framework. Not only the agent but also the user now have distinct tools to observe and modify their own database. For instance, in telecom support, the user can check their phone’s airplane mode while the agent queries network settings. This tests coordination and communication under partial observability, making scenarios more realistic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQiA50M3DrG7wWs6ngq5Dg.png" /><figcaption><a href="https://arxiv.org/abs/2506.07982"><em>Tau2-bench</em></a><em> supports a dual-control framework, where both the agent and the user have distinct tools to observe and modify their own database.</em></figcaption></figure><p>In contrast to the <strong>pass@k</strong> metric (“at least one of k attempts succeeded”) which is widely adopted, Tau-bench introduces <strong>pass^k</strong> (“all k attempts succeeded”) as a reliability-focused metric. Mathematically, pass^k = p^k decays exponentially. A model with 90% success rate (pass@1) drops to just 57% consistency at k=8. This makes pass^k highly sensitive to worst-case reliability, capturing how often systems fail under repeated attempts rather than average performance.*</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*DQa-SBqWJ01lyGOKZDSVuQ.png" /><figcaption><em>Pass^k (solid line) and pass@k (dotted line) in the retail domain (</em><a href="https://arxiv.org/abs/2406.12045"><em>source</em></a><em>). The exponential decay of pass^k illustrates how reliability drops sharply with repeated trials, while pass@k remains optimistic. In this article, we focus on pass@1 (=pass¹) for simplicity.</em></figcaption></figure><p><em>*If you are interested in the math behind these metrics, </em><a href="https://hippocampus-garden.com/pass_k/"><em>here</em></a><em> is my blog post about this topic.</em></p><h3>The Challenge of Hidden Benchmark Details</h3><p>When you looked at the first figure, you might have thought: “Claude Sonnet 4.5 beats GPT-5, so let’s use Claude.” But look closer at these published results, and you’ll find critical details that make direct comparison impossible.</p><p>First, optimized prompts that aren’t disclosed. If you scroll to the bottom of <a href="https://www.anthropic.com/news/claude-sonnet-4-5">Anthropic’s release notes for Claude Sonnet 4.5</a>, you’ll find this admission:</p><blockquote>Scores were achieved using extended thinking with tool use and a prompt addendum to the Airline and Telecom Agent Policy instructing Claude to better target its known failure modes when using the vanilla prompt.</blockquote><p>Anthropic optimized the system prompt specifically for their models. This could be valid for showcasing capabilities, but it means their scores aren’t directly comparable to results using the standard prompt. Meanwhile, OpenAI’s release notes provide no details about their configuration at all.</p><p>Second, unspecified reasoning efforts that affect latency and cost. Reasoning models like GPT-5 “think” by generating extra reasoning tokens before responding. You can control this thinking budget, and higher budgets usually increase accuracy on complex tool sequences but add significant latency and cost. For a production system with latency limits, this trade-off is critical, but the published scores often hide it.</p><p>Third, different evaluation settings. Beyond prompts and reasoning efforts, organizations may use different user simulators or retry logic. Without standardized settings, benchmark scores become apples-to-oranges comparisons.</p><h3>Our Reproduction: Methodology and Results</h3><p>That’s why we reproduced Tau-bench ourselves. Our goal was to evaluate widely-available frontier models under realistic production constraints and see how they perform in practice.</p><p>Our approach:</p><ul><li>We stayed as close as possible to <a href="https://github.com/sierra-research/tau2-bench">Sierra Research’s original Tau2-bench configuration</a> to maintain comparability.</li><li>For GPT-5, we explicitly tested and reported different reasoning effort levels (low, minimal) in our charts.</li><li>The only major change we made was formatting the input to meet <a href="https://docs.claude.com/en/docs/build-with-claude/extended-thinking#extended-thinking-with-tool-use">the specific requirements of Claude models</a> with extended thinking and tool use.</li><li>We used <a href="https://github.com/sierra-research/tau2-bench/blob/5ba9e3e56db57c5e4114bf7f901291f09b2c5619/src/tau2/agent/llm_agent.py#L24-L31">the exact same system prompt</a> for all models. No model-specific optimizations.</li><li>We skipped slower configurations (like GPT-5 with high reasoning effort) that would violate customer support latency requirements.</li></ul><h3>Comparing Our Results to OpenAI and Anthropic’s Published Scores</h3><p>Here are our results for 3 domains (retail, airline, and telecom) and the weighted average score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*v1PXsp0ULrBRIub2CGl-Pg.png" /><figcaption><em>Results for 3 domains and the weighted average score.</em></figcaption></figure><p>Key findings:</p><ul><li>OpenAI scores are aligned with ours. Even with low reasoning effort, we reproduced results close to what OpenAI reported (presumably with higher reasoning effort). This suggests OpenAI’s settings weren’t dramatically different from ours.</li><li>Anthropic scores diverged significantly. We consistently got lower scores for Anthropic models compared to their published results. The most likely explanation is the prompt optimization they disclosed made a substantial difference (10+ percentage points).</li><li>Domain variance was notable. GPT-5 with reasoning significantly outperformed other models in telecom because: (1) telecom tasks are more complex and require longer tool-call sequences where reasoning helps, and (2) telecom has less user simulator errors that make tasks ill-defined — a key focus of Tau2-bench’s dual-control framework.</li></ul><h3>Validating Against Third-Party Evaluators</h3><p>What about scores reported by independent third-party evaluators? We compared our results against two trusted sources.</p><p>ArtificialAnalysis is one of the most respected benchmark evaluators. <a href="https://artificialanalysis.ai/methodology/intelligence-benchmarking#tau2-bench-telecom">ArtificialAnalysis clearly documents their custom settings</a>:</p><ul><li>They only use the telecom domain</li><li>They use Qwen3 235B A22B 2507 (non-reasoning) for the user agent instead of GPT-4.1 (Sierra Research’s default user agent) to have full control</li><li>They added a mechanism to escape stuck loops (when user and assistant repeat the same exchange)</li><li>They run 3 times and report the average score to reduce variance</li></ul><p>Below is the correlation between our scores and theirs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*TC0YENUhqfr0ZXOgvyUOTQ.png" /><figcaption><em>Correlation between our scores and the scores reported by ArtificialAnalysis. Our scores correlated highly with those reported by ArtificialAnalysis.</em></figcaption></figure><p>The high correlation suggests that despite different settings, the relative model rankings remain consistent. This validates that our methodology doesn’t introduce significant bias.</p><p>Sierra Research recently launched <a href="https://taubench.com/#leaderboard">a live leaderboard</a>. Our results for GPT-5 and GPT-4.1 align well with their verified results (as of Oct 27, 2025), further validating our approach.</p><h3>Mapping the Frontier of Score-Latency Trade-offs</h3><p>Benchmark scores tell you which model is most accurate, but production decisions require balancing three dimensions: score, latency, and cost. A model that scores 5 percentage points higher but takes 4 times longer might be the wrong choice when you want a prompt answer.</p><p>This is where the published benchmark scores fall short. They report accuracy without context about the trade-offs. Here’s what we found when we mapped the score-latency frontier.</p><p>Below is our scatter plot for score vs median latency, where latency means time-to-last-token (TTLT) for the final LLM call in each agentic loop. We excluded LLM calls for tool calling to avoid bias (models better at tool use have more tool-calling steps, which are generally faster than message generation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*Jdo81eIgAQPdz9JypztuWw.png" /><figcaption><em>Scatter plot for score vs median latency.</em></figcaption></figure><p>Key findings:</p><ul><li>GPT-5 (low) achieved the highest score but was roughly 4 times slower than GPT-4.1.</li><li>GPT-5 (minimal) was still roughly 2 times slower than GPT-4.1 despite being the fastest GPT-5 configuration.</li><li>The Pareto frontier included GPT-4.1, GPT-5 (minimal), and GPT-5 (low).</li><li>Cost scales with latency. Reasoning tokens aren’t free. GPT-5’s latency increase comes with proportional cost increases, making the total cost of ownership significantly higher.</li></ul><h3>Key Takeaways</h3><p>After reproducing Tau-bench and comparing our results against published scores from OpenAI, Anthropic, and third-party evaluators, we learned that benchmark numbers alone can’t guide production decisions. The gap between published scores and real-world performance depends on configuration details that are rarely disclosed. Here’s what matters when evaluating AI agents for production use:</p><p><strong>Published benchmark scores can’t answer production questions.</strong> Undisclosed configurations make direct comparisons impossible. Our Tau-bench reproduction showed Anthropic’s scores diverged 10+ percentage points from ours due to prompt optimization.</p><p><strong>Single-number rankings hide critical trade-offs.</strong> GPT-5’s highest scores came with 4 times latency overhead. We think all the benchmarks should report the latency as well as the score.</p><p><strong>Run benchmarks yourself with your actual constraints.</strong> Different use cases may favor different models. Even if you can’t afford to build your own benchmark, you can map the score-latency-cost frontier for your specific use case.</p><p>The paradox of public benchmarks is that they’re most useful when you go beyond their headline numbers. Rather than asking “which model achieves the highest score?”, the right question is “which model best fits my specific constraints?” That question can only be answered through your own evaluation — whether that’s building custom benchmarks or at minimum, reproducing existing ones with your actual configuration and measuring what matters for your use case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fdae3a418cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/benchmarking-ai-agents-stop-trusting-headline-scores-start-measuring-trade-offs-0fdae3a418cf">Benchmarking AI Agents: Stop Trusting Headline Scores, Start Measuring Trade-offs</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking AI Agents: The Challenge of Real-World Evaluation]]></title>
            <link>https://medium.com/alan/benchmarking-ai-agents-the-challenge-of-real-world-evaluation-6aa1c2aa4b41?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa1c2aa4b41</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Tue, 14 Oct 2025 14:57:56 GMT</pubDate>
            <atom:updated>2025-10-14T15:02:41.551Z</atom:updated>
            <content:encoded><![CDATA[<p>Last year, we <a href="https://medium.com/alan/benchmarking-large-language-models-1e1ab5b809ac">discussed different ways to benchmark large language models (LLMs)</a>. This is a follow-up to discuss how to benchmark AI agents — a fundamentally different challenge.</p><p>Modern agents are not just “one-shot” LLM calls. They hold context, call tools, read and write to internal databases, and must complete tasks reliably. This reality breaks many of the assumptions behind classic LLM benchmarks. This post explains why evaluating agents is harder than evaluating models, why some popular public benchmarks are insufficient for business workflows, and how a new generation of stateful benchmarks (i.e., ones that hold states in web or databases) like Tau-bench provide a solution blueprint for real-world agent evaluation.</p><h3>What Makes Agent Evaluation Hard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SvPUDokiOwpx_g1LVxIo5Q.png" /></figure><p>LLM benchmarking is typically simple: ask a generic question (for example, a math expression) and compare the single output to a ground truth. Agent evaluation is different because an agent involves a loop of multiple steps. It reads from the web or an internal database and may write back. The answer depends on the current state of the world.</p><p>Consider the customer support query: “I want to change my flight.” The correct answer depends on the user’s booking, fare rules, and cancellation policy. A realistic agent flow might look like this:</p><ol><li>Receive the user message.</li><li>Query get_flights(user_id) → returns a flight ID like “AF1234”.</li><li>Query get_flight_info(“AF1234”) → returns destination, times, cancellation rules.</li><li>Compose a final answer and possibly write back a change/cancellation.</li></ol><p>The key difference is that an agent’s behavior is conditional on a mutable state that lives outside the model. Any evaluation system for agents has to handle this state somehow.</p><h3>How Public Benchmarks Try to Solve it</h3><p>Many popular benchmarks such as <a href="https://www.swebench.com/">SWE-bench</a>, <a href="https://webarena.dev/">WebArena</a>, and <a href="https://arxiv.org/abs/2308.03688">AgentBench</a> run agents inside containers with public tools (OS, browser, code interpreter). These are valuable, but they have two key limitations for business workflows: they typically avoid <strong>the hardest part for businesses: interacting with a complex, internal database</strong>, and they assume agents run autonomously without user interaction. Without realistic state, write actions, and user simulation, you can test tool-use mechanics, but not whether the agent truly completes business tasks under realistic constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uIWtOcFxOUc4-cL1xFbtXA.png" /><figcaption><a href="https://webarena.dev/"><em>WebArena</em></a><em> benchmarks agents by having them act in simulated web apps, but without persistent state. This limits relevance for real business workflows.</em></figcaption></figure><h3>The Rise of Stateful Benchmarks</h3><p>There are a few benchmarks that try to move closer to reality by addressing these limitations:</p><ul><li><a href="https://arxiv.org/abs/2406.12045"><strong>Tau-bench</strong></a> and <a href="https://arxiv.org/abs/2506.07982">Tau2-bench</a>: These use a mocked database persisted as local JSON files to simulate customer support conversations. Agents can read and write through tools, and the evaluation checks both the final answer and <strong>the resulting database state</strong>. Critically, they also simulate user interactions with user agents, mimicking real-world back-and-forth conversations.</li><li><a href="https://arxiv.org/abs/2501.12851">ACE Bench</a>: This benchmark also simulates user interactions with agents. It mocks tools without a full database and mainly checks whether the agent calls the right tool with the right arguments.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSsRDTTuuWG9wi-Db_00Dg.png" /><figcaption><em>An overview of </em><a href="https://arxiv.org/abs/2406.12045"><em>Tau-bench</em></a><em>. It has a database to persist user information, and the agent makes read/write operations to the database via tool calling.</em></figcaption></figure><p>Among these, we think Tau-bench is the most relevant one for us because it evaluates “what they should solve” rather than “how they solve it”. In the real world, there are often different ways to achieve the same goal, and we want to evaluate the agent’s ability to finish real tasks, not to enforce a specific resolution path. This is particularly true for customer support tasks, which we are tackling at Alan.</p><p>Tau-bench’s approach — mocking a database with local JSON files and validating both conversational outputs and database state changes — offers a practical blueprint that can be adapted to internal business contexts. But implementing this approach raises several important questions.</p><h3>Adapting Stateful Benchmarks for Real Business Workflows</h3><p>Let’s say we want to build a “stateful” benchmark for customer support. We prepare a set of users, traits (language, tone, etc.), questions, ground truth, and available agent tools. Can we just do this by taking real (anonymized*) conversations and feeding them into the agent (i.e., offline replay)? That’s not so easy.</p><p><em>* Anonymization is also not easy because we need to keep the consistency of the data. For example, the name of the user and the pronouns used in the conversation should be consistent.</em></p><p>A naive approach would be to mock the tools. This works well for simple tools like get_flights(user_id). However, it doesn’t work for tools that have many possible combinations of parameters (e.g., search_flights(origin, destination, date)).</p><p>That’s when we consider using the real tools and database, but copying users (and all the related data) from production to the benchmark environment comes with challenges:</p><ul><li>Data models are complex and evolving. It is not easy to replicate them and keep the copy consistent with the original.</li><li>We need to make test runs idempotent. For example, if the agent writes to the database in one test run, it should not affect the database in the next test run.</li></ul><p>Building a controlled state library is the hardest part. However, we can get some hints from trusted public benchmarks. We believe the key is to abstract some parts of the data model and focus on high-quality, small datasets (~100 scenarios) that cover critical business workflows. In our next post, we’ll do a detailed walkthrough of how Tau-bench works and how different LLMs perform on it.</p><h3>Looking Ahead</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GidJw85t15z3WMjscMuMiA.png" /><figcaption><em>AI benchmarks positioned by business realism, from academic evaluations to stateful business workflows.</em></figcaption></figure><p>As agents evolve from single LLM calls into stateful systems that orchestrate multiple tools and database operations, our evaluation methods must evolve too. Public benchmarks have made progress, but they often sidestep the hardest challenge: ensuring agents can reliably complete real business workflows that depend on complex internal state. The path forward is to build internal evaluations that test what matters: database states and responses to the user.</p><p>If you’re tackling similar challenges or have insights on stateful agent evaluation, we’d love to hear from you! Reach out in the comments or connect with us directly. We’re actively building these systems and learning as we go.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa1c2aa4b41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/benchmarking-ai-agents-the-challenge-of-real-world-evaluation-6aa1c2aa4b41">Benchmarking AI Agents: The Challenge of Real-World Evaluation</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside Alan’s Claim Agent: How an AI Investigates Claims just like a Human, only Faster]]></title>
            <link>https://medium.com/alan/inside-alans-claim-agent-how-an-ai-investigates-claims-just-like-a-human-only-faster-11d62bd7efc9?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/11d62bd7efc9</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 11:14:35 GMT</pubDate>
            <atom:updated>2025-06-24T04:33:00.994Z</atom:updated>
            <content:encoded><![CDATA[<p>At Alan, we’ve always believed that exceptional customer care sets us apart in the insurance industry. Earlier this year, we shared <a href="https://medium.com/alan/how-we-built-alans-ai-assistant-for-customer-support-2a82194df8be">how we had built our AI assistant to automate customer contacts</a> while maintaining the same customer satisfaction scores as our human support team. Today, we’re excited to dive deeper into our most sophisticated AI agent yet: the <strong>Claim Agent</strong>. Our initial release has fully automated roughly 30% of claims-related support tickets it receives.</p><p>In this blog post, we’ll explore how the agent works, share real-world examples of it in action, and reveal what we learned from building it.</p><h3>Why We Needed an Agentic Approach</h3><p>Our AI-powered customer support journey began with specialized<a href="https://medium.com/alan/how-we-built-alans-ai-assistant-for-customer-support-2a82194df8be"> agents focused on specific domains</a>: the Coverage Agent for insurance coverage questions and the Generic Agent for broader FAQ-style queries. These agents worked well for straightforward, well-defined problems where required information could be anticipated and included in the system prompt. However, as we expanded our automation scope, we encountered a significant challenge: <strong>claims support </strong>— helping members understand their reimbursements, track claim status, and resolve issues with medical expenses.</p><p>Claims are one of the most complex areas of customer support, representing 20% of all tickets. Typical questions include: “Why was I partly reimbursed for this Osteopathy session?” Unlike coverage questions, which can usually be answered with a static coverage table, claims require dynamic investigation of:</p><ul><li>Multiple care events across different dates and beneficiaries</li><li>Various insurance documents and their processing states</li><li>Complex reimbursement calculations involving different health guarantees</li><li>Member-specific policy details and consumption tracking</li></ul><p>Our traditional approach of building comprehensive system prompts quickly hit a wall. Including all potential claims-related information created massive contexts cluttered with irrelevant data, slowing response times and reducing answer quality.</p><p>When human agents work on claims, you’ll notice they don’t try to anticipate every possible piece of information upfront. Instead, they first interpret the member’s question, reason about what additional information they need, retrieve those specific details from backend systems, and repeat this targeted approach until they have enough information to answer the question accurately.</p><p>Observing this natural workflow inspired us to build our most sophisticated agent yet — one capable of <strong>dynamically retrieving exactly the information needed for each specific scenario</strong>.</p><h3>Claim Agent as a ReAct Agent</h3><p>Unlike our previous static agents, the Claim Agent implements the <a href="https://www.promptingguide.ai/techniques/react"><strong>ReAct</strong> (Reason + Act) paradigm</a> — a powerful framework enabling AI agents to iteratively gather information and make decisions.</p><p>The Claim Agent starts with a lean system prompt containing:</p><ul><li><strong>Playbook</strong>: A collection of general claims knowledge and resolution strategies</li><li><strong>Member attributes</strong>: Contextual information like current care event or document details</li><li><strong>Tool descriptions</strong>: Available backend functions with usage examples and conditions for their use</li></ul><p>Here’s how the ReAct loop works in practice:</p><ol><li><strong>Think</strong>: The LLM analyzes the member’s question and available context to determine if more information is needed.</li><li><strong>Act</strong>: If additional data is required, the agent calls backend tools to fetch relevant information</li><li><strong>Observe</strong>: Tool results are returned and fed back into the LLM’s context.</li><li><strong>Repeat</strong>: The cycle continues until the agent has sufficient information to provide a comprehensive answer</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*68F9UGCNArhuks1t" /><figcaption><em>Claim Agent ReAct loop — yellow boxes are LLM calls</em></figcaption></figure><p>This approach avoids “brute-forcing” unnecessary data into the prompt while maintaining the ability to access relevant information when needed.</p><p>During the “Act” step the Claim Agent is able to leverage specific backend tools (which are deterministic and not LLM-based):</p><ul><li><em>get_care_event_details(care_event_id)</em>: Fetches care event information (e.g., reimbursed amount, status)</li><li><em>get_document_details(document_id)</em>: Fetches insurance document details (e.g., document type, processing status)</li><li><em>get_health_guarantee_details(health_guarantee_title)</em>: Fetches health guarantee conditions and remaining allowances</li><li><em>list_care_events(date, name, care_type)</em>: Searches for care events by date or other criteria</li></ul><p>What makes this approach powerful is that tools are called <strong>only when needed</strong>. Our data shows that tool calling happens in approximately 20% of conversations, keeping the system efficient while maintaining comprehensive coverage.</p><h3>Example Stories: Claim Agent in Action</h3><p>To better understand how this works in practice, let’s examine three real scenarios that showcase both the agent’s capabilities and its limitations.</p><p><strong>Example 1: Benefit Consumption Query</strong></p><p>In this example, a member asked, “How many alternative medicine sessions do I have left?”</p><p>The agent processed the question in the following way:</p><p><strong>Loop 1</strong></p><ol><li><strong>Think</strong>: “The member is asking about benefit consumption. I need to check their health guarantee details.”</li><li><strong>Act</strong>: Call <em>get_health_guarantee_details(“psychology”)</em></li><li><strong>Observe</strong>: Tool returns health guarantee details and remaining allowances</li></ol><p><strong>Loop 2</strong></p><ol><li><strong>Think</strong>: “I now have sufficient information to answer. I will now exit the loop”</li></ol><p><strong>Final Answer (Outside of ReAct loop)</strong></p><p>“You have used 4 of your 5 psychology sessions. You have 1 session remaining this year.”</p><p>The “Think”, “Act”, and “Observe” steps are completed with a single LLM call. In this example, we made 3 LLM calls total.</p><p><strong>Example 2: Finding the right care event from an ambiguous question</strong></p><p>In this example, a member asked about their reimbursement and then asked about one of their beneficiaries (“What about my daughter’s?”).</p><p>The agent processed the question in the following way.</p><p><strong>Loop 1</strong></p><ol><li><strong>Think</strong>: “The member is asking about a specific beneficiary’s reimbursement, but I don’t know which care event they’re referring to.”</li><li><strong>Act</strong>: Call <em>list_care_events(beneficiary=&lt;beneficiary_name&gt;)</em></li><li><strong>Observe</strong>: Multiple care events returned for beneficiary across different dates</li></ol><p><strong>Loop 2</strong></p><ol><li><strong>Think</strong>: “Too many results. I need more specific information.”</li></ol><p><strong>Answer to member</strong></p><p>“I see several care events for &lt;beneficiary_name&gt;. Could you specify the date?”</p><p>Then the member replied: “It was last Tuesday.”</p><p><strong>Loop 1</strong></p><ol><li><strong>Think</strong>: “I need to find the care event for &lt;beneficiary_name&gt; on 2025–06–17.”</li><li><strong>Act</strong>: Call <em>list_care_events(beneficiary=&lt;beneficiary_name&gt;, date=”2025–06–17&quot;)</em></li><li><strong>Observe</strong>: Single care event found</li></ol><p><strong>Loop 2</strong></p><ol><li><strong>Think</strong>: “Now I’ll check the details of this care event.”</li><li><strong>Act</strong>: Call <em>get_care_event_details(care_event_id=&lt;care_event_id&gt;)</em></li><li><strong>Observe</strong>: Tool returns care event details</li></ol><p><strong>Loop 3</strong></p><ol><li><strong>Think</strong>: “I now have sufficient information to answer.”</li></ol><p><strong>Final Answer</strong></p><p>“Your daughter’s care event was reimbursed on 2025–06–17.”</p><p>This example demonstrates the agent’s ability to handle ambiguous queries through clarifying questions.</p><p><strong>Example 3: When tool calls go wrong</strong></p><p>We occasionally encountered errors when the agent provided incorrect parameters. One example occurred when the agent called get_care_event_details with an ID <em>care_event_id_missing_in_the_system_prompt</em>, which isn’t valid at all!</p><p>Our error handling strategy returns descriptive error messages as strings rather than exceptions, allowing the LLM to learn from mistakes and self-correct. However, tool calling isn’t 100% reliable, and we must always prepare for mistakes. When the agent can’t retrieve the necessary information after multiple attempts, it <strong>gracefully escalates to human agents</strong> rather than providing potentially incorrect answers, ensuring members always receive accurate support.</p><h3>Lessons Learned</h3><p>Building the Claim Agent taught us valuable lessons about agentic AI systems that go far beyond our previous experience with static prompts. Here we share some of the key insights.</p><p><strong>Design foolproof tool interfaces. </strong>As we’ve seen, LLMs make mistakes, so we need to design tool interfaces that are robust against these errors:</p><ol><li><strong>Simplify tool interfaces</strong>: Remove redundant parameters. If tools are mostly used together, combine them into a single tool.</li><li><strong>Clearly specify parameters</strong>: Provide explicit format requirements and examples in tool descriptions (e.g., UUID format, YYYY-MM-DD date format)</li><li><strong>Implement robust error handling</strong>: Ensure tools return clear error messages rather than exceptions, enabling the agent to learn from mistakes and self-correct.</li><li><strong>Limit the number of loops</strong>: Set a strict limit on the number of ReAct loops to avoid infinite looping.</li></ol><p><strong>Don’t hesitate to be prescriptive in the system prompt.</strong> Sometimes providing tools isn’t enough for the agent to use them correctly. We encountered cases where the agent wouldn’t call any tool despite our efforts in the tool descriptions. In such cases, adding explicit instructions to the system prompt became necessary, even if this introduced duplication which we’d prefer to avoid.</p><p>We also implemented a crucial <strong>reflection step</strong>. After the agent completes its tool calls, we prompt it to “step back and reflect on what the member actually wanted” before generating the final answer. This helps the agent distinguish between directly answering the member’s question and merely presenting information gathered from tool calls.</p><p>Other key insights include:</p><ul><li><strong>System prompt vs. tool calling</strong>: We need to balance the amount of information included in the system prompt with the frequency of tool calls. We follow this rule of thumb: include information in the system prompt if it’s needed more than 80% of the time.</li><li><strong>Model choice</strong>: We observed that LLMs from 2025 are significantly better at following tool specifications and parameter formats than ones from 2024. Specifically, GPT-4o from May and August 2024 often struggles with choosing the right parameters. As LLM providers optimize their models for agentic use, we should always look for the latest models when implementing tool calling.</li><li><strong>Privacy concerns</strong>: Before executing tools (which are read-only in our case), we need to verify that the user has permission to access the data. Tool outputs go into the LLM prompt, which means the data could be exposed to the user. If the user isn’t allowed to access the data, we shouldn’t call the tool in the first place.</li></ul><p>These insights resonate a lot with <a href="https://www.oreilly.com/library/view/ai-engineering/9781098166298/"><em>AI Engineering</em></a>, a recent book by Chip Huyen (especially <a href="https://huyenchip.com/2025/01/07/agents.html">chapter 6 section 2</a>). We recommend it to anyone interested in building AI agents.</p><h3>Conclusion</h3><p>The journey from static prompts to dynamic, tool-calling agents marks a fundamental evolution in how we approach AI-powered customer support. The Claim Agent investigates, gathers evidence, and provides comprehensive solutions — just like human agents but much faster.</p><p>But this is just the beginning. Our next steps include:</p><ul><li><strong>Adding more tools</strong>: We’ll add more tools to the Claim Agent to handle increasingly complex cases</li><li><strong>Exploring advanced agentic architectures</strong>: We started with ReAct, one of the simplest agentic architectures. We’ll explore more advanced patterns such as the <a href="https://langchain-ai.github.io/langgraph/concepts/multi_agent/#multi-agent-architectures">supervisor pattern</a></li><li><strong>Enhancing testing and evaluation</strong>: Our in-house testing and evaluation framework doesn’t support tool calling yet. To make this work, we need to provide a way to mock tool calls, as tests and evaluations run against static datasets extracted from past data</li></ul><p><em>Interested in building the future of AI-powered customer care? </em><a href="https://jobs.ashbyhq.com/alan/1371db73-6795-4f8e-a94a-f560d0c6e063"><em>We’re hiring!</em></a><em> Join us as we continue to push the boundaries of what’s possible with agentic AI systems.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11d62bd7efc9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/inside-alans-claim-agent-how-an-ai-investigates-claims-just-like-a-human-only-faster-11d62bd7efc9">Inside Alan’s Claim Agent: How an AI Investigates Claims just like a Human, only Faster</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepSeek R1: Demystifying LLM’s Reasoning Capabilities]]></title>
            <link>https://medium.com/alan/deepseek-r1-demystifying-llms-reasoning-capabilities-f6332154349b?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6332154349b</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 17:33:37 GMT</pubDate>
            <atom:updated>2025-03-03T20:06:59.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hu-23pyOkD1uLbmE" /></figure><p>At Alan, we’re constantly exploring the cutting edge of AI technology. In a recent internal seminar, we covered DeepSeek’s latest model: <strong>R1</strong>. Unlike many high-profile models, R1 is an open-weight successor to OpenAI’s o1, offering comparable performance while making its weights public and free to use for further training. This transparency fuels innovation by enabling researchers and engineers to experiment and build upon its capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*abcHsW1cfy1WnOT4" /><figcaption><em>DeepSeek R1 achieves performance on par with OpenAI o1</em></figcaption></figure><p>In this post, we’ll share our learnings from <a href="https://arxiv.org/abs/2501.12948">their technical report</a>.</p><h3>Reinforcement Learning: Back to Fundamentals</h3><p>One of the most intriguing aspects of R1 is that its post-training is based on simple <strong>reinforcement learning</strong>. Historically, techniques like <strong>reinforcement learning from human feedback</strong> (<strong>RLHF</strong>) or <strong>AI feedback</strong> (<strong>RLAIF</strong>) have been common. However, these methods are expensive and complex due to the inherent complexity of human values. They require datasets with <strong>chain-of-thought</strong> (<strong>CoT</strong>) for <strong>supervised fine-tuning</strong> and human preferences for <strong>preference optimization</strong>. Additionally, they demand extra computational resources for training the reward model, which is also an LLM.</p><p>DeepSeek’s approach with R1 is refreshingly different. In domains such as math and programming, it’s possible to provide an exact reward at the end of each completion by simply verifying an answer, eliminating the need for annotations and reward models. By directly leveraging the exact reward signal from task completions, the DeepSeek team demonstrated that pure reinforcement learning could effectively instill reasoning capabilities without supervised fine-tuning. With this approach, the researchers created <strong>R1-zero</strong> from <strong>DeepSeek V3</strong>, a base model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eFdq2pn4H9Ye127m" /><figcaption><em>Emergence of reasoning skills during reinforcement learning</em></figcaption></figure><p>During training, they observed intriguing phenomena:</p><ul><li>The model learned to “think harder” as it encountered more complex problems, gradually extending its responses to 10,000 tokens (as long as a typical machine learning paper)</li><li>An “aha” moment occurred where the model backtracked in its chain of thought</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*xaYssHOBNJkd89U3" /><figcaption><em>One reason R1 became popular so quickly is that we can examine its full chain of thought, unlike with OpenAI models</em></figcaption></figure><p>While R1-Zero was impressive, it had a few issues such as language mixing, which is understandable because the model was only rewarded for the final answer and format. This motivated the team to explore a more refined approach — first trying supervised fine-tuning and then applying reinforcement learning. This led to their flagship model, <strong>R1</strong>. To reduce inference costs, the team also created smaller models by distilling R1 down to Llama-8B and Qwen-32B. They found that direct reinforcement learning on these smaller models didn’t work well.</p><h3>Technical Insights: What Worked, What Didn’t, and What’s Unknown</h3><p>The success of R1 is based on two of DeepSeek’s previous works:</p><ul><li><strong>DeepSeek V3</strong>: This massive model features 671 billion parameters, designed as a <strong>mixture of experts</strong> (<strong>MoE</strong>). Despite its size, only 37 billion parameters are active at any time, allowing training with limited computational resources. The team employed many advanced techniques to overcome hardware challenges (e.g., writing low-level code to maximize efficiency). If you’re interested in more details, please refer to <a href="https://arxiv.org/abs/2412.19437">the technical report</a>.</li><li><strong>Group Relative Optimization Policy</strong> (<strong>GRPO</strong>): GRPO was introduced as a more efficient alternative to traditional reinforcement learning methods (<a href="https://arxiv.org/abs/2402.03300">paper</a>). While conventional methods could also enable R1’s development, GRPO helped reduce computational demands by eliminating the need for a value model.</li></ul><p>The R1 paper also discusses approaches that proved ineffective:</p><ul><li><strong>Process reward model</strong>: The idea was to reward intermediate steps during the model’s reasoning process (CoT). While promising in theory, it proved challenging to implement. The model struggled with defining the fine-grained steps necessary for precise reasoning, and there was a risk of “reward hacking,” where the model might exploit the reward system without genuinely solving the problem.</li><li><strong>Monte Carlo tree search</strong> (<strong>MCTS</strong>): This technique, famous for its use in AlphaGo, involves systematically exploring the solution space. However, applying it to language models proved difficult. The search space was too vast, and training a reliable value model to guide the search was problematic, making it an inefficient approach for R1’s objectives.</li></ul><p>Despite the paper’s technical depth, several key aspects remain undisclosed:</p><ul><li><strong>Test-time scaling curve</strong>: They did not reveal the specifics of how R1 scales during inference time. <a href="https://openai.com/index/learning-to-reason-with-llms/">The curve OpenAI shared last September</a> didn’t disclose the compute amount, so we had hoped to see this information in the R1 paper.</li><li><strong>Training dataset</strong>: As is often the case with recent technical reports, it doesn’t give details about the training dataset. Nevertheless, we assume the dataset is similar to the one used to train <a href="https://arxiv.org/abs/2402.03300">DeepSeekMath</a> (GSM8k and MATH).</li><li><strong>Training setup</strong>: The paper does not fully explain the unique training setup that enabled their results. Given the simplicity of the core idea, there must be intricacies in the training setup that prevent others from replicating the same approach. However, DeepSeek has <a href="https://github.com/deepseek-ai/open-infra-index?tab=readme-ov-file#202502-open-source-week">recently begun open-sourcing some of their code</a>.</li></ul><h3>Community and Industry Impact</h3><p>R1 is already triggering significant activity in both research and industry. Several projects are under way to replicate the results and make the training methodology fully public: <a href="https://github.com/hkust-nlp/simpleRL-reason">simpleRL-reason</a>, <a href="https://github.com/Jiayi-Pan/TinyZero">TinyZero</a>, <a href="https://github.com/huggingface/open-r1">open-r1</a>, and more. Another interesting project is <a href="https://arxiv.org/abs/2501.19393">s1: simple test-time scaling</a>. It not only leverages R1’s chain of thought but also introduces innovative techniques (like “budget forcing”) to control response length.</p><p>We’ve tested R1 ourselves! We conducted an offline test on simulated customer support conversations and compared it against o1-mini. R1 performed better, especially in complex situations where the prompt required the model to handle conflicting conditions and make multiple decisions to reach resolution. While some challenges remain, such as high latency and API compatibility issues, our initial results were promising.</p><h3>Conclusion</h3><p>The main takeaway from R1 is that reinforcement learning can be highly effective with the right setup. As long as the base model has sufficient knowledge, we can strengthen its reasoning capability through reinforcement learning with automatic rewards.</p><p>Looking toward real-world applications, if we reward agents when they successfully complete tasks like purchasing products, resolving support tickets, implementing feature requests, or closing sales, we could create effective web agents, customer support agents, software engineer agents, or sales agents. However, the bottleneck is on the human side, as we need to be able to verify outputs automatically.</p><p>We’re excited to see a new contender in the AI leaderboard and look forward to the insights the research community will gain from this development. What an exciting way to start the year!</p><p><em>We are hiring software engineers. If you are interested, please submit a job application </em><a href="https://jobs.ashbyhq.com/alan?departmentId=e2d57d52-bd4d-433d-881a-9a0adbf7baa0"><em>here</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6332154349b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/deepseek-r1-demystifying-llms-reasoning-capabilities-f6332154349b">DeepSeek R1: Demystifying LLM’s Reasoning Capabilities</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built Alan’s AI Assistant for Customer Support]]></title>
            <link>https://medium.com/alan/how-we-built-alans-ai-assistant-for-customer-support-2a82194df8be?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a82194df8be</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 13:29:58 GMT</pubDate>
            <atom:updated>2025-01-08T13:37:49.360Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*Gjg9ciDhqwYDGczc" /></figure><p>At Alan, exceptional customer care isn’t just a service — it’s a core part of what sets us apart in the insurance industry.</p><p>Our commitment to providing outstanding support raised an important question: how could we introduce AI assistance while maintaining the high standards our members expect? We approached this challenge with the understanding that any automated solution would need to match or exceed the quality of our human support team.</p><p>That’s why we wanted to design our AI assistant to be transparent, human-centric, and seamlessly integrated with our support team. Every interaction needed to start with a clear understanding that you can connect with a human agent at any time, and our AI needed to be trained to recognize when a human touch is needed.</p><p>To meet these expectations, we launched our Automated Resolution project in 2023 with the aim of delivering exceptional support while reducing wait times and ensuring smooth transitions to human agents when necessary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aN1zZ_d8_Z85VRkFx-H1zQ.png" /><figcaption><em>How our customer support experience looks like</em></figcaption></figure><p>By 2024, we reached a significant milestone: automating close to 20% of customer contacts through our mobile app while maintaining the same customer satisfaction (CSAT) scores as our human Support without AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WlF00ItsVfEdGAMo" /></figure><p>In this blog post, we’ll share how we did it and the lessons we learned along the way.</p><h3>Coverage Agent: Starting small</h3><p>We began with a focused approach by automating answers to coverage-related questions — queries about what is and isn’t covered by a member’s health insurance plan. This structured domain allowed us to minimize risks while learning from real-world interactions. Coverage questions are frequent and backed by a well-defined coverage table, making them an ideal starting point.</p><p>The <strong>Coverage Agent</strong> operates in two steps:</p><ol><li><strong>Classification</strong> determines the relevant section of the coverage table for the question, ensuring precise and context-specific answers.</li><li><strong>Generation</strong> creates responses using the selected section of the coverage table.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*WNqb1d2VzXAtVzQy" /><figcaption><em>Coverage Agent overview. The question is used twice: for classifying the health guarantee and for generating an answer.</em></figcaption></figure><p>We also ensured that users could easily escalate to a human agent if dissatisfied with the chatbot’s answers. After every answer, users were prompted to indicate whether the answer was helpful, with dissatisfied users seamlessly connected to a human agent.</p><h3>Generic Agent: Expanding scope</h3><p>Having established a successful model with the Coverage Agent, we sought to expand its scope to address a wider variety of queries by introducing the <strong>Generic Agent</strong>. This agent uses the <strong>RAG</strong> (<strong>retrieval-augmented generation</strong>) framework and draws from FAQ articles in our Help Center to answer a broader range of queries.</p><p>Here is how it works. It has a more complex structure than Coverage Agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/0*c9JpsyrHXKVDeBrD" /><figcaption><em>Generic Agent overview. The question is used three times: for retrieving similar documents, for selecting relevant documents, and for generating an answer.</em></figcaption></figure><ul><li><strong>Document embeddings</strong>: Articles from our Help Center are converted into vector embeddings and stored in a vector database. This is done every day with a scheduled job.</li><li><strong>Question embedding</strong>: When a member asks a question, it is converted into a vector in the same way. This ensures that questions and articles can be compared within the same vector space.</li><li><strong>Retrieval</strong>: Similar articles are retrieved from the database based on vector similarity.</li><li><strong>Classification</strong>: The retrieved results may contain similar but irrelevant articles. To refine results, a classification step is introduced to filter out irrelevant articles. This step also uses an LLM to analyze retrieved articles and return a “true” or “false” value indicating relevance to the member’s question.</li><li><strong>Generation</strong>: The remaining articles serve as the basis for generating answers.</li></ul><h3>Agent Selector: Dispatching questions to the right agent</h3><p>At this point, we had two separate agents working on different app sections (Coverage Agent on the coverage section and Generic Agent on the Help Center). This meant that we couldn’t answer coverage questions coming from the Help Center and vice versa.</p><p>To make full use of the agents, we developed “<strong>Agent Selector</strong>,” which analyzes the member’s question and routes it to the correct agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r3PRx2SFXpX8A6yj" /></figure><p>While being a simple routing mechanism, this is our first step towards an agentic system, allowing us to get more automation by adding more specialized agents to the system. This will be even more important when we have more agents and more entry points to our AI Assistant.</p><h3>Monitoring and reviewing</h3><p>As the system grew in complexity, maintaining quality and gathering insights for continuous improvement became crucial. That’s why we implemented monitoring and review processes:</p><ul><li><strong>Monitoring dashboard</strong>: Provides a centralized overview of key performance indicators (KPIs) such as automation rate, answer acceptance rate, agent classification accuracy, and technical metrics.</li><li><strong>Review tool</strong>: We built a review tool on <a href="https://alan.com/fr-fr/blog/tout-alan/a/plongee-dans-les-outils-internes-crees-pour-notre-service-client">Marmotton</a>, our in-house helper tool for customer service. It encompasses two types of reviews. <strong>Escalation Reviews</strong> aim to detect incorrect answers and ensure that members receive correct information within the service level agreement (SLA). <strong>Monitoring Reviews</strong> are more comprehensive and for gathering feedback to improve the AI Assistant. This feedback covers aspects such as necessary resolution actions (e.g., update user information), missing information, required documents, and general impressions. Hence the reviewers need to know how the chatbot works internally.</li></ul><h3>Workflow Agents: Navigating complex decision trees</h3><p>To further expand our automation scope, we started to build another agent based on a new data source: <strong>workflows</strong>. Workflows are decision trees with prepared answers designed to help our customer service agents write personalized answers based on the member’s specific situation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*hT4Zf8bdlYp4I7b5HTUv1A.png" /><figcaption><em>An example of workflows</em></figcaption></figure><p>Among many existing workflows, we first picked the “portability” workflow to build a “<strong>Portability Agent</strong>” because of this workflow’s high volume. The Portability Agent required a different approach from Coverage and Generic agents.</p><ul><li><strong>Simplification</strong>: The original workflow designed for our customer service agents was too complex, so we manually cut unnecessary details to make it more understandable for LLMs.</li><li><strong>Serialization</strong>: As the workflow has a tree structure, we converted it into a JSON string with depth-first search to include it in the agent’s prompt.</li><li><strong>Redirections</strong>: we invented a special syntax with which the LLM can specify to which app section members should be redirected. This syntax triggers the backend systems to present the member with an appropriate call-to-action button, directing them to the relevant section of the app. For example, our chatbot can explain how to add a beneficiary in the app, with an actual button directing to the screen where we can add a beneficiary</li><li><strong>Member attributes</strong>: To provide tailored answers based on individual situations, each agent accesses a list of relevant member attributes, including their current policy status, policy details, and so on</li><li><strong>Write actions</strong> (experimental): Some agents can perform write actions such as updating a member’s policy state. We describe a set of available functions that the agent can call here.</li></ul><p>After refining the workflow and prompt, we achieved notable success, which encouraged us to scale this approach to develop numerous Workflow Agents. Today, new workflow agents can be deployed with minimal code modifications, and Product Specialists can independently launch them with ease. Their process involves refining a workflow, clicking on the “serialization button” to integrate it into the agent’s prompt, and defining relevant redirections and member attributes. Now everyone on our team is a prompt engineer!</p><h3>End-to-end testing: Ensuring quality at scale</h3><p>In the early days, the process of prompt changes was simple — test a user story manually and if it works, promote it to production. However, with several workflow agents sharing the same template, it became hard to anticipate the potential side effects. We realized the need for more rigorous testing before deployment, which led to the introduction of end-to-end testing.</p><p>Each test corresponds to a single user story (description of member problem) and we simulate a conversation between the member AI and our AI Assistant. Each test consists of a first member message and a set of assertions:</p><ul><li>Did we call the expected specialized agent?</li><li>Did we escalate the contact or not?</li><li>Did we send the right redirection button?</li><li>Did we solve the member problem (<a href="https://langfuse.com/docs/scores/model-based-evals"><strong>LLM-as-a-judge</strong></a>)?</li></ul><p>We run this test suite every day and also ad-hoc when trying to improve a prompt. With this, we can be more confident with our new version of a prompt before shipping it to the production environment.</p><h3>What we learned</h3><p>Through our iterations, we gained valuable insights that shaped our approach and strengthened our system. Below are some of the key lessons learned:</p><ul><li>With the right UX, <strong>we can automate 20% of contacts</strong>. Note that it is essential to prioritize maintaining a positive member experience. We could easily get to 100% by removing the path to the human Support team, but that would be a horrible experience for a member. We performed A/B testing which confirmed that our use of AI is not compromising the member experience.</li><li><strong>Close collaboration with product specialists is crucial</strong>. Their domain expertise is essential for the entire development process: problem scoping, building an agent, and reviewing the output. To move faster, we invested engineers’ resources in tooling for them to build and improve agents autonomously, while we don’t hesitate to do synced collaboration when needed.</li><li><strong>Automatic evaluations are necessary</strong> as systems mature. Rigorous testing in controlled environments becomes increasingly important to ensure a high-quality user experience for a growing user base. Beyond end-to-end testing, which works as a guardrail, we are now developing more sophisticated offline evaluation, the results of which should correlate with the actual automation rate. This will impel us to improve our system.</li></ul><h3>Future directions</h3><p>Our journey hasn’t ended! In 2025, we are aiming to automate 40% of contacts (not limited to contacts from mobile applications). To this end, we will focus on:</p><ul><li><strong>Offline evaluation</strong>: As we just discussed above, we are going beyond end-to-end testing. With more sophisticated offline evaluation, we will be able to optimize our system more confidently.</li><li><strong>Agentic system</strong>: We built our system with many scaffolds because in the early phase, LLMs were not reliable enough (we were using GPT-3.5 back then). This is working well but comes with some limitations. For example, our AI Assistant is not good at handling ambiguous questions because we dispatch to a specialized agent based on the very first message. Now that we have more powerful LLMs, we believe we can eliminate those initial constraints and let LLMs make the resolution plan.</li><li><strong>Full deployment and globalization</strong>: We are gradually rolling out our AI Assistant across all platforms. It is only available on the French mobile app at the moment. In 2025, we will build the web version and also start globalization to further increase the impact.</li></ul><p>Do you want to join us? We’re hiring! Please submit a job application <a href="https://jobs.ashbyhq.com/alan?departmentId=e2d57d52-bd4d-433d-881a-9a0adbf7baa0">here</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a82194df8be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/how-we-built-alans-ai-assistant-for-customer-support-2a82194df8be">How We Built Alan’s AI Assistant for Customer Support</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Llama 3 Takeaways for LLM Practitioners]]></title>
            <link>https://medium.com/alan/llama-3-takeaways-for-llm-practitioners-209c50e3892f?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/209c50e3892f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 06:40:27 GMT</pubDate>
            <atom:updated>2024-09-24T08:13:33.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4R-oNjq9r-u6f_8D" /></figure><p>Meta’s release of <a href="https://www.llama.com/"><strong>Llama 3</strong></a> in July 2024 has significantly impacted the landscape of large language models (LLMs). Unlike other open LLMs, Meta not only shared the model weights but also published <a href="https://ai.meta.com/research/publications/the-llama-3-herd-of-models/">a comprehensive paper</a> detailing their training recipe. This generosity is rare and provides a lot of valuable insights. Even if you’re a user who doesn’t train LLMs from scratch, it still offers useful lessons.</p><p>At its core, Llama 3 employs a straightforward neural net architecture, but its strength lies in meticulous data curation and iterative training. This underscores the effectiveness of <a href="https://arxiv.org/abs/1706.03762">the good old Transformer architecture</a> and highlights the importance of training data.</p><p>In this post, I’ll share key takeaways from the Llama 3 paper, highlighting three practical aspects: data curation, post-training, and evaluation.</p><h3>Data Curation Insights</h3><p>Data curation, which involves gathering, organizing, and refining data to ensure its quality and relevance for training machine learning models, is a cornerstone of Llama 3’s development. The process is divided into two phases: pre-training and post-training. In the pre-training phase, Meta’s team significantly reworked the data curation process to collect high-quality data from the web.</p><ul><li>A custom HTML parser was crafted to extract quality text from web documents</li><li>They experimented with model-based classifiers to select high-quality tokens.</li><li>They built domain-specific pipelines to harvest data from code and math-focused web pages.</li><li><a href="https://saturncloud.io/glossary/learning-rate-annealing/">Annealing</a>, a technique that involves a gradual reduction in learning rate, was applied alongside upsampling of quality code and mathematical data, which boosted model performance.</li></ul><p>In the post-training phase, Meta’s team tackled the challenge of data quality by primarily relying on synthetic data.</p><ul><li>Unlike <a href="https://arxiv.org/abs/2203.02155">OpenAI’s initial reliance on human annotators</a> for ChatGPT, Llama 3’s post-training largely depended on synthetic data. They generated over 2.7 million synthetic examples for <strong>supervised fine-tuning</strong> (<strong>SFT</strong>), challenging<a href="https://arxiv.org/abs/2305.11206"> the notion that minimal data suffices for alignment</a>. This is particularly important to those who don’t have enough training data or can’t use their data for privacy reasons</li><li>Like <a href="https://arxiv.org/abs/2307.09288">Llama 2</a>, Llama 3 was post-trained using <a href="https://arxiv.org/abs/2305.18290"><strong>Direct Preference Optimization</strong></a> (<strong>DPO</strong>), a technique that aligns models with preferred responses without needing to develop reward models. Nevertheless, reward models were trained to filter out low-quality synthetic samples through rejection sampling.</li><li>In the preference annotation step, annotators were asked to edit the AI-generated response in addition to selecting the preferred response. This resulted in tuples of three responses: “edited,” “chosen,” and “rejected,” in order of preference.</li><li>Machine-translated data was generally avoided to prevent biases (c.f., translationese), except when augmenting quantitative reasoning dialogues in simple language where the benefits outweighed the risks</li></ul><h3>Iterative Post-training</h3><p>Llama 3’s development was not about achieving perfection on the first try. Instead, they embraced an iterative, multi-stage approach, refining components progressively. The team was open to ad-hoc solutions for specific challenges, undergoing six rounds of reward modeling, SFT, and DPO. They crafted many model variants from experiments using various data and hyperparameters and averaged their weights at the end. This training pipeline is fairly complex compared to the industry standard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KH_rNyKH66p74NGc" /><figcaption>Llama 3’s post-training pipeline</figcaption></figure><h3>Rigorous Evaluation</h3><p>With a refined model through iterative post-training, the next crucial step was to rigorously evaluate Llama 3’s capabilities and limitations. Beyond standard benchmarks, they explored the model’s sensitivity to input variations (see the image below), which is crucial for practitioners as minor differences in prompts can lead to unexpected results. They also tackled data contamination issues, ensuring the model hadn’t been inadvertently trained on benchmark data, which can skew performance evaluations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*lBUfIBMdif-rDaRT" /><figcaption>Robustness to different label variants in the MMLU benchmark</figcaption></figure><h3>Concluding Remarks</h3><p>While this post focused on practical takeaways from Llama 3, the paper is 92 pages long, and there are many topics that we didn’t explore, such as infrastructure management, model safety evaluation, inference acceleration, extending LLM capabilities to vision and audio, and team organization. If you are interested in these topics, I highly recommend <a href="https://ai.meta.com/research/publications/the-llama-3-herd-of-models/">the original paper</a>.</p><p>At Alan, we are committed to enhancing our customer support experience by developing a more effective feedback loop and rigorous evaluation process. By leveraging the insights from Llama 3, we aim to improve the efficiency and reliability of our chatbot. I hope this post helps you enhance your applications as well!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=209c50e3892f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/llama-3-takeaways-for-llm-practitioners-209c50e3892f">Llama 3 Takeaways for LLM Practitioners</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking Large Language Models]]></title>
            <link>https://medium.com/alan/benchmarking-large-language-models-1e1ab5b809ac?source=rss-8a15808834c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e1ab5b809ac</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Shion Honda]]></dc:creator>
            <pubDate>Thu, 18 Apr 2024 07:36:18 GMT</pubDate>
            <atom:updated>2025-02-23T08:14:52.414Z</atom:updated>
            <content:encoded><![CDATA[<p>Judging the quality of <strong>large language models</strong> (<strong>LLMs</strong>) is an unsolved challenge in AI. This is why there are so many LLM benchmarks — such as MMLU, GSM8K, and Chatbot Arena, just to name a few. So what exactly does each benchmark evaluate? If you’d like to find out, this blog post is for you.</p><p><em>This post is based on what I recently presented at ML Sync, an internal seminar on machine learning that the Data and Engineering community hosts bi-weekly. If you find this post interesting and you’d like to join our team, please submit a job application </em><a href="https://jobs.ashbyhq.com/alan?departmentId=e2d57d52-bd4d-433d-881a-9a0adbf7baa0"><em>here</em></a><em>!</em></p><h3>The Multifaceted Nature of LLMs</h3><p>LLMs are versatile. They can solve various tasks spanning different domains and in different languages. This multifaceted nature necessitates a comprehensive approach to benchmarking, one that evaluates models from different angles. To get a high-level view on them, this paper by Chiang et al. categorizes the benchmarks into two main dimensions: static vs. live and ground truth vs. preference.</p><p>Static benchmarks are collections of pre-defined questions, while live benchmarks assess models in a dynamic and interactive environment. Ground truth benchmarks assess the accuracy of the LLM’s responses against a set standard, while preference benchmarks let humans choose the best response from multiple outputs based on certain criteria. Each of these approaches has its own pros and cons, which we will discuss later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SRCfemZNxcveGvAr.png" /><figcaption>Categorized LLM benchmarks (<a href="https://arxiv.org/abs/2403.04132">source</a>)</figcaption></figure><p>We also have a benchmark specifically designed to evaluate LLM’s ability to recall very specific content given in the prompt, which is called Needle in a Haystack. In the following part of this post, we will take a closer look at each category of benchmarks.</p><h3>Ground Truth Benchmarks</h3><p>Ground truth benchmarks are fairly straightforward — think school exams. Here are some notable examples:</p><ul><li><a href="https://arxiv.org/abs/2009.03300"><strong>MMLU</strong></a> (Massive Multi-task Language Understanding) is a collection of multiple-choice questions from 57 diverse subjects that evaluates an LLM’s knowledge and problem-solving skills</li><li><a href="https://arxiv.org/abs/1905.07830"><strong>HellaSwag</strong></a> consists of questions that require some common sense reasoning, which are considered to be easy for humans but not for current LLMs</li><li><a href="https://arxiv.org/abs/2110.14168"><strong>GSM8K</strong></a> (grade-school math 8.5k) evaluates the ability to solve math problems</li><li><a href="https://arxiv.org/abs/2107.03374"><strong>HumanEval</strong></a> evaluates Python coding abilities by prompting LLMs to write code from docstrings</li><li><a href="https://arxiv.org/abs/2206.04615"><strong>BigBench</strong></a> is a diverse, multi-lingual benchmark that consists of over 200 tasks</li></ul><p>However, these benchmarks have their downsides:</p><ul><li>Because it’s based on ground truth, we cannot evaluate how useful they are in real-world (i.e., interactive and open-ended) settings</li><li>Because it’s static, we cannot be sure that the test questions were not included in the training corpus. For example, <a href="https://arxiv.org/abs/2309.08632">You can achieve a perfect score by pretraining on the test set</a>. <a href="https://lmsys.org/blog/2023-11-14-llm-decontaminator/">Pretraining on a paraphrased test set also leads to unfairly good scores.</a></li></ul><p>Also, we should be careful about the community’s tendency to consider MMLU as a representative benchmark. It evaluates an LLM’s knowledge and problem-solving skills, but in practice, we don’t really expect LLMs to be knowledgeable because we provide them with necessary information in the prompt.</p><h3>Preference Benchmarks</h3><p>To fill the gap between the benchmarks based on ground truth and real-world applications, LMSYS proposed two benchmarks based on human preferences: <a href="https://arxiv.org/abs/2306.05685"><strong>MT-bench</strong></a> and <a href="https://arxiv.org/abs/2403.04132"><strong>Chatbot Arena</strong></a>.</p><p>MT-bench is a collection of 80 challenging multi-turn questions. Then humans (or even GPT-4 bots) compare pairs of responses generated by LLMs and vote. But MT-bench is still susceptible to data contamination because it relies on a static set of questions.</p><p>Chatbot Arena goes further to let you ask any question! It provides an <a href="https://chat.lmsys.org/">open platform</a> where you can send any prompt and compare outputs from two different LLMs. From an LLM’s point of view, this is an arena where they fight against each other on their ability to chat, and you are the judge!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DlZxLDNWSvfxTrx-" /><figcaption>Chatbot Arena</figcaption></figure><p>LMSYS calculates “<strong>Elo ratings</strong>” (a comparative ranking system, <a href="https://en.wikipedia.org/wiki/Elo_rating_system">more info here</a>) for each LLM based on the vote results and creates a <a href="https://chat.lmsys.org/?leaderboard">leaderboard</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ucexfXAr5zSfazoa" /><figcaption>Chatbot Arena Leaderboard (S<em>creenshot taken on April 11th, 2024.)</em></figcaption></figure><p><em>Note: They are not actually Elo ratings. They are calculated by a statistical model called </em><strong><em>Bradely-Terry</em></strong><em> because it is more sample-efficient. If you are interested in the difference between the two methods, here is </em><a href="https://hippocampus-garden.com/elo_vs_bt/"><em>my blog post</em></a><em> on this topic!</em></p><p>The way Chatbot Arena evaluates LLMs is very different from other benchmarks, but the calculated scores correlate very well. In the figure below, you see Arena Elo correlates well with MT-bench scores and MMLU scores. Here, the correlation with MT-bench is stronger than MMLU because MT-bench is more similar to Chatbot Arena as both are preference benchmarks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T8jgwe_daQ-1ZjXd" /><figcaption>Arena Elo correlates with MT-bench scores and MMLU scores</figcaption></figure><p>Chatbot Arena is one of the most trusted benchmarks in the community, but it still has some limitations:</p><ul><li>It’s not clear what is actually evaluated due to its open-endedness</li><li>Users tend to be LLM hobbyists</li><li>Most of the conversations are single-turn</li><li>77% of the conversations are done in English</li><li>Users might not care about safety or toxicity</li></ul><p>Generally speaking, it’s important to keep in mind that users of Chatbot Arena have different intentions from users of your application.</p><h3>Needle in a Haystack</h3><p>As a bonus of this post, I’d like to introduce an interesting task called <strong>Needle in a Haystack</strong> (<strong>NIAH</strong>). NIAH <a href="https://twitter.com/GregKamradt/status/1722386725635580292">was invented to evaluate capabilities in efficiently locating specific information within extensive texts</a>, simulating the real world use cases such as <strong>RAG</strong> (<a href="https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-rag.html">retrieval-augmented generation</a>).</p><p>By artificially inserting “needle” — a key piece of information — within a “haystack” of text, NIAH assesses an LLM’s ability to answer questions based on that needle. Through this pressure test, we can reveal that, for example, an LLM is not good at recalling sentences located in the middle part of the prompt.</p><p>It is worth noting that recent LLMs are smart enough to realize that the “needle” was artificially inserted and refuse to answer the question, saying “it seems that this sentence may have been a comment left accidentally in the document”. <a href="https://www.anthropic.com/news/claude-2-1-prompting">Anthropic recently noticed this problem and managed to fix it</a> by a tweak in the prompt. As the saying goes, the devil is in the details — especially with prompts!</p><h3>Conclusion</h3><p>In this post, we’ve discussed why we need to care about how we evaluate LLMs and took a closer look at what each benchmark evaluates. As there is no one-size-fits-all benchmark, we should not blindly trust reported performance figures. Instead, we should be aware of the nuanced differences of benchmarks and ideally test these models against the specific scenarios. This ensures a more accurate and relevant understanding of how an LLM will perform in real-world applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e1ab5b809ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/alan/benchmarking-large-language-models-1e1ab5b809ac">Benchmarking Large Language Models</a> was originally published in <a href="https://medium.com/alan">Alan Product and Technical Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>