<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ben Davis on Medium]]></title>
        <description><![CDATA[Stories by Ben Davis on Medium]]></description>
        <link>https://medium.com/@bdavis_38378?source=rss-5006b04c1127------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xaL7DUkRrrxMG9ip7t8VMA.png</url>
            <title>Stories by Ben Davis on Medium</title>
            <link>https://medium.com/@bdavis_38378?source=rss-5006b04c1127------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:23:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bdavis_38378/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ask Better Questions]]></title>
            <link>https://medium.com/99p-labs/ask-better-questions-b1c93ee1a1fc?source=rss-5006b04c1127------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1c93ee1a1fc</guid>
            <category><![CDATA[knowledge]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[community]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ben Davis]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 16:47:47 GMT</pubDate>
            <atom:updated>2026-10-08T16:47:47.868Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A hypothesis, tested and proven wrong</em></p><blockquote>Success represents the 1% of your work which results from the 99% that is called failure. - Soichiro Honda</blockquote><p>This is a tale from the 99%.</p><p>Start with a decent sized corpus of unstructured data chunked and stored in a vector database. Think of all the documentation that gets generated for a product development project. The hypothesis is that all the evidence that can support an answer to a question can be retrieved from the database, provided you ask the right questions.</p><p>Reframing the vector database content as evidence is intentional. Any inferred answers need to be backed up by evidence.</p><p>In the traditional approach to graph building, you define an ontology, and start connecting facts by relationships. To me that means elevating evidence to understanding and locking it in. When the LLM finds relationships in the graph, it will assert them as facts.</p><p>Instead of attacking the graph building problem head-on, I wanted to build a domain graph that would define the shape and characteristics of the corpus, without drawing specific conclusions. It would serve as a map to guide the LLM in generating hypothesis questions while keeping the expansion bounded. The process loop is expand-retrieve-evaluate-converge-answer.</p><h4>Build the ruler</h4><p>Step one is to generate a reference set of questions with answers and attributions. The answer key was built backwards from the evidence: read passages in the corpus, then write a question they answer, and record those passages as the correct result. Some questions were written to deliberately avoid the vocabulary of their own evidence, so that search had to do real work to find it. Actually there are two questions per answer. One well-stated, and one degraded. Running both questions shows the difference in retrieval quality between a good and bad prompt. That difference is what we were targeting for improvement.</p><h4>The gap</h4><p>Across three collections built from slides, spreadsheets, and business prose documents, a vague question found about a third less of the correct source material than a well-stated one asking for the same thing.</p><h4>Expand every query</h4><p>We expanded every query and recovered somewhere between a seventh and a third of what a badly-worded question costs. It never came close to closing it, and the best single result failed to replicate.</p><p>Then we ran it again with every graph-derived field emptied, so the model wrote its follow-up questions with nothing but the original question and the first round of search results. It matched the full pipeline exactly. Whatever recovery we were seeing came from asking a model to write follow-up questions, not from the graph.</p><p>Some of the questions turned on a technical concept, which a graph might know something about. Others turned on an arbitrary package code, which the graph couldn’t possibly know anything about. That second group was there as a sanity check, a case where a gain would mean we had made a measurement error.</p><p>Expansion was useless on precise identifier queries and harmful on vague ones. Asked to elaborate a vague question about a parts package, it generated questions about the parts’ function. Reasonable-sounding, but aimed at the wrong thing. Those results crowded out the few correct rows the original query had managed to find.</p><p>What was unintended is what the control exposed: not that expansion failed on those questions, but that the baseline search had never been able to answer them at all. Zero results containing the identifier. The experiment was asking “does expansion add value here?” and the answer came back “there is nothing here to add value to.”</p><h4>The bottom line</h4><p>The graph didn’t improve weak questions. But the control test, designed to show where the graph couldn’t help with identifiers, revealed that ordinary search couldn’t retrieve them either. That failure was invisible. The query just returned other things. It would have stayed hidden indefinitely, because nobody checks a search that confidently returns twenty plausible results. The discovery came from building a deliberate negative control and digging into why.</p><p>The query expansion was measured and characterized and ultimately left alone. Everything we tried to close the gap either did no better than a far simpler version of itself, or was worse than doing nothing.</p><p>The gap is still there. We know more about it than when we started — how big it is, that it holds steady across very different kinds of document, and that it is caused by what the question carries rather than by what the search can reach. The experiment failed at its stated goal but redeemed itself by finding a problem we didn’t know we had.</p><p>PS. yes, we did fix the problem with a full-text index and a payload filter.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1c93ee1a1fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/ask-better-questions-b1c93ee1a1fc">Ask Better Questions</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling AI Agents: An Event-Driven A2A Architecture Experiment]]></title>
            <link>https://medium.com/99p-labs/scaling-ai-agents-an-event-driven-a2a-architecture-experiment-ee951d359593?source=rss-5006b04c1127------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee951d359593</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[a2a-protocol]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ben Davis]]></dc:creator>
            <pubDate>Thu, 31 Jul 2025 23:22:58 GMT</pubDate>
            <atom:updated>2025-07-31T23:22:58.718Z</atom:updated>
            <content:encoded><![CDATA[<p>In April this year, Google introduced the Agent2Agent Protocol, or A2A (https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability). An Agent is a system that leverages AI to complete tasks on behalf of a user and the A2A protocol proposes a standard way for Agents to communicate with each other.</p><p>The concept of an AI agent reminds me of micro-services and a previous project where we had the micro-services communicating with each other through a Kafka broker. Is this a feasible architecture to use with agents? I’ve been experimenting with Agents learning how to best make use of A2A protocol and MCP. This blog will focus on just A2A and creating an experiment to test out this idea.</p><p>Here is a simple two-agent example:</p><figure><img alt="two-agent diagram" src="https://cdn-images-1.medium.com/max/982/1*pJQXLVdE3YCkoWCD7rJfrg.png" /></figure><p>Agent-1 starts it off by creating a Task object with the initial message. A Task is the basic unit of work for an Agent. The specification is not explicit about where the first message should go so I made it the first Message in the History. Agent-2 receives the request and interacts with an LLM to produce some result. That result gets added to the Task as an Artifact. Agent-1 then receives the completed task.</p><h4>Point-to-point vs Event driven</h4><p>In the above architecture, as you add more agents for performance and functionality, it can get very complicated quickly. The same thing happened when Micro-services were introduced as an alternative to Monoliths. If you follow the A2A spec, each agent hosts it’s own server and must handle all communications, security, and everything that can go wrong. Building on previous work with Kafka and data pipelines, I wanted to see how this would work if we put an event broker between the agents.</p><p>The idea of an event-driven A2A interaction is not an original one. Just google A2A and event brokers and you will see proposals for Kafka, SQS, Flink, MQTT, and other message queuing systems. This all makes sense on paper, but I’m a learn-by-doing kind of person a so let’s do an experiment.</p><figure><img alt="agent with broker diagram" src="https://cdn-images-1.medium.com/max/942/1*ieoHnc1OARrNV8SuDVPxKg.png" /></figure><p>Agent-1 and Agent-2 are completely decoupled and have no server function at all. Here you can see that scaling up Agent-2 is a matter of adding more of them.</p><figure><img alt="Pub/Sub agents with broker and partition details" src="https://cdn-images-1.medium.com/max/1024/1*jHcYiMzEeqP8_KdislFjIQ.png" /></figure><h4>The experiment: Find Waldo</h4><p>The idea was to find a simple problem that can be split up into many identical tasks to be run concurrently. Finding Waldo fits the bill because the game board is large and filled with a lot going on. Agent-1 starts by splitting up the board into smaller tiles, creating Tasks and publishing the Tasks to a topic. Agent-2 pulls a Task from the topic, asks an LLM to find Waldo in the tile. Agent-2 then adds the answer as an Artifact to the current Task and publishes it to another output topic. Agent-1 is watching the output topic and displays which tiles contain Waldo.</p><p>The input topic has nine partitions and the entire job consists of 408 tasks. For the experiment, we run the job repeatedly, each time increasing the number of agents by one until we reach the maximum number of partitions. For comparison, I ran the experiment against a local Ollama server and OpenAI. The broker is a Kafka-compatible platform called Redpanda and the Pub/Sub Agents are written in Go. The local model is LLaVA and the OpenAI model is ChatGPT4_1Nano.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEAX-MR9WOcZ6HO9y6KSjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVEN4Z9zMExZ6tWhsnSUJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E19U1qzAZy3Fcs4z3BjaDA.png" /></figure><p>The results are kind of what you’d expect. The local Ollama server peaks at at 50% after 3 agents and gets worse as you add more. The gain for OpenAI keeps going up as you add agents, reaching 350%. This is not meant to be a real performance test, but a demo to see this architecture in action. If you have access to a lot of GPUs, it’s not that difficult to maximize throughput for the type of use-case that can be broken up into discrete units of work.</p><h4>Conclusion</h4><p>Substituting an event broker for HTTP is not without it’s own difficulties and it mostly makes sense if your priority concern is throughput. And of course, not all workloads can be executed like this. In any case, I do think it’s important to make the Agent transport-agnostic. Separate the message protocol from the transport so you can more easily adapt to other environments. This experiment, for example, is using Protobufs (not JSON) over the Kafka (not HTTP) wire protocol.</p><p>The A2A specification says that the Agent must communicate with JSON over HTTP. It also talks about authentication, authorization, and discovery among other things. There are many different ways to discover, deliver and protect data and if you’re running in Kubernetes, for example, a lot of this gets delegated to the container. Defining A2A is currently a moving target. What seems practical to me at this stage is to focus on the data format and to develop the practice of using a common Agent language, providing a small measure of stability and compatibility with A2A as it evolves.</p><h4>References:</h4><p>A2A announcement: <a href="https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/">https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/</a></p><p>A2A Protocol Buffers: <a href="https://github.com/a2aproject/A2A/tree/main/specification/grpc">https://github.com/a2aproject/A2A/tree/main/specification/grpc</a></p><p>A2A protocol spec: <a href="https://a2aproject.github.io/A2A/latest/specification/">https://a2aproject.github.io/A2A/latest/specification/</a></p><p>A2A articles:</p><ul><li><a href="https://www.confluent.io/blog/google-agent2agent-protocol-needs-kafka">Why Google&#39;s Agent2Agent Protocol Needs Apache Kafka</a></li><li><a href="https://www.kai-waehner.de/blog/2025/05/26/agentic-ai-with-the-agent2agent-protocol-a2a-and-mcp-using-apache-kafka-as-event-broker/">Agentic AI with the Agent2Agent Protocol (A2A) and MCP using Apache Kafka as Event Broker</a></li><li><a href="https://blog.radixia.ai/edaa-event-driven-agentic-architecture/">EDAA: Event Driven Agentic Architecture</a></li></ul><p>Tools and libraries:</p><p>Redpanda: <a href="https://www.redpanda.com/">https://www.redpanda.com/</a></p><p>Kafka client lib: <a href="https://github.com/twmb/franz-go">https://github.com/twmb/franz-go</a></p><p>Ollama client lib: <a href="https://github.com/ollama/ollama">https://github.com/ollama/ollama</a></p><p>OpenAI client lib: <a href="https://github.com/openai/openai-go">https://github.com/openai/openai-go</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee951d359593" width="1" height="1" alt=""><hr><p><a href="https://medium.com/99p-labs/scaling-ai-agents-an-event-driven-a2a-architecture-experiment-ee951d359593">Scaling AI Agents: An Event-Driven A2A Architecture Experiment</a> was originally published in <a href="https://medium.com/99p-labs">99P Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>